Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
Hieu Xuan Le, Benjamin Goh, Quy Anh Tang · 2026
Read Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models by Hieu Xuan Le, Benjamin Goh, Quy Anh Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25176.txt.
Introduction
Le, Goh, and Tang test whether small language models can judge prompt attacks fast enough for live use. Big judge models catch jailbreaks well but run too slow for real chat traffic. The team guides light models through set steps: split the intent, check safety signals, weigh harm, and reflect. On a set mixing real benign chats with machine-made attacks, a small Gemini model judges well at low delay. It now guards public service chatbots in Singapore. Mixing several judges adds only small gains.
Catalog record
| CREATOR | Hieu Xuan Le, Benjamin Goh, Quy Anh Tang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46985 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models Hieu Xuan Le, Benjamin Goh, Quy Anh Tang Abstract Prompt attacks, including jailbreaks and prompt injections, pose a critical security risk to Large Language Model (LLM) systems. In production, guardrails must mitigate these attacks under strict low-latency constraints, resulting in a deployment gap in which lightweight classifiers and rule-based systems struggle to generalize under distribution shift, while high-capacity LLM-based judges remain too slow or costly for live enforcement. In this work, we examine whether lightweight, general-purpose LLMs can reliably serve as security judges under real-world production constraints. Through careful prompt and output design, lightweight LLMs are guided through a structured reasoning process involving explicit intent decomposition, safety-signal verification, harm assessment, and self-reflection. We evaluate our method on a curated dataset combining benign queries from real-world chatbots with adversarial prompts generated via automated red teaming (ART), covering diverse and evolving patterns. Our results show that general-purpose LLMs, such as gemini-2.0-flash-lite-001 , can serve as effective low-latency judges for live guardrails. This configuration is currently deployed in production as a centralized guardrail service for public service chatbots in Singapore. We additionally evaluate a Mixture-of-Models (MoM) setting to assess whether aggregating multiple LLM judges improves prompt-attack detection performance relative to single-model judges, with only modest gains observed. 1 Introduction Large Language Models (LLMs) have evolved from static text generation to powering autonomous agentic systems that reason, plan, and interact with external tools ( Yao et al. 2022 ; Wang et al. 2024 ; Xi et al. 2025 ; Plaat et al. 2025 ) . This transformation has driven widespread adoption, with agentic applications emerging in diverse domains such as web navigation ( Zhou et al. 2023 ; He et al. 2024 ) and automated software engineering ( Jimenez et al. 2023 ; Yang et al. 2024 ) . However, LLMs have internal vulnerabilities that make them susceptible to adversarial exploitation. First, competing objectives arise during training, where goals for helpfulness and instruction-following conflict with safety goals to avoid harm ( Bai et al. 2022 ; Ouyang et al. 2022 ; Wei et al. 2023 ) . For instance, prefix injection prompts the model to start with a harmless "Sure, here’s how to…”, which it follows due to helpfulness priors, then continues into prohibited content rather than refusing unnaturally ( Wei et al. 2023 ) . Second, mismatched generalization arises because safety training does not cover unusual inputs that broad pretraining handles well; examples include obfuscation techniques such as Base64 encoding ( Wei et al. 2023 ) , and payload splitting ( Kang et al. 2024 ) . Third, instruction/data confusion results from concatenated inputs - including system prompts provided by developers, user messages from end-users, and tool outputs - as a unified token stream without adequate privilege separation, allowing malicious content from untrusted sources to be interpreted as authoritative instructions and override intended behavior ( Greshake et al. 2023 ; Liu et al. 2024a ; Wallace et al. 2024 ) . In agentic systems, these vulnerabilities become far more severe. Whereas a successful attack on a standalone LLM typically results only in the generation of inappropriate or harmful text, the integration of tools granting external access elevates the consequences dramatically, enabling direct real-world harms such as unauthorized actions or data leakage ( Plaat et al. 2025 ) . A prominent example is indirect prompt injection, where adversaries hide malicious directives (e.g., reveal private business information) in retrieved external data (e.g., webpages or documents), forcing the agent to process them without direct user involvement ( Greshake et al. 2023 ; Liu et al. 2023a ; Liu et al. 2024b ) . Consequently, prompt attacks - including jailbreaking and various injection techniques - rank as the leading security threat for LLM applications ( OWASP Foundation 2025 ) . Because foundation models lack the inherent capability to reliably filter these adversarial inputs, organizations often deploy external detection layers - or guardrails - to sanitize inputs before execution. This necessity creates a dilemma for engineering teams: the strict trade-off between accurate detection and the low latency required for live applications. To maintain a seamless user experience, the added latency from these guardrails must be minimal to avoid disrupting the primary workflow. Conventional defenses struggle to meet this need. Lightweight encoder-based classifiers, such as those built on DeBERTa ( He et al. 2020 ) , are sufficiently fast but often lack the reasoning capabilities to detect nuanced "grey-area" attacks ( Hackett et al. 2025 ; Li and Liu 2024 ; Kumar et al. 2025 ) . Conversely, LLM-based approaches offer superior performance in identifying subtle manipulations but are typically slow and resource-intensive for live traffic ( Kumar et al. 2025 ; Wang et al. 2025 ) . Our Work and Contributions Motivated by this trade-off, our work examines prompt-attack detection from the perspective of practical deployment. We focus on highly reliable detection approaches that can operate under live latency constraints and on evaluation settings that reflect real-world usage and evolving attack patterns. Our key contributions are summarized below. • LLM-as-a-Judge for Real-Time Prompt Attack Detection. We propose an LLM-as-a-Judge approach that enforces a structured reasoning process rather than unconstrained judgement. The judge is required to explicitly deconstruct user intent, verify safety signals, assess potential harm, and perform self-reflection before issuing a final verdict. This prompt-level design enables general-purpose, lightweight LLMs to function reliably as prompt-attack detectors, making them well suited for real-world guardrail deployment. • Evaluation on Mixed Real-World and Red-Teamed Prompt Data. We curate a private dataset that combines benign prompts collected from public service chatbots in Singapore with complex, diverse, and evolving attack patterns generated via automated red teaming. The dataset reflects realistic user behavior and the latest prompt-attack techniques encountered in production systems, providing a practical evaluation setting for prompt-attack detection. • Mixture-of-Models for Robust Prompt-Attack Detection. We study a Mixture-of-Models (MoM) in the context of prompt-attack detection, evaluating whether aggregating judgments from multiple LLM judges improves robustness compared to single-model judging. 2 Related Work 2.1 Prompt Attack Benchmarks To evaluate robustness, researchers frequently utilize community-driven repositories like deepset/prompt-injections ( Schwenzow 2023 ) , jackhhao/jailbreak-classification ( Hao 2023 ) , and rubend18/ChatGPT-Jailbreak-Prompts ( Darío 2023 ) . While standard for training binary classifiers, these collections are not challenging enough to accurately grade modern guardrails. They typically rely on outdated, repetitive attack templates - such as "Do Anything Now" (DAN) ( Liu et al. 2023b ) , roleplay scenarios, or "Ignore previous instructions" - which many defense models are already well-trained to recognize. Furthermore, the benign samples in these datasets are often too simple and obvious (e.g., short greetings or factoid questions). This lack of complexity allows defense models relying on shallow heuristics - like flagging specific keywords (e.g., "ignore", "override") - to achieve inflated accuracy scores. Consequently, these models fail to generalize to real-world scenarios: they often miss novel, complex attacks like code-mixed payloads, while raising false positives on legitimate, complex instructions simply because they contain "suspicious" vocabulary. Furthermore, large-scale gamified datasets like HackAPrompt ( Schulhoff et al. 2023 ) and TensorTrust ( Toyer et al. 2023 ) introduce a distinct bias. Derived from "Capture-the-Flag" (CTF) competitions, these prompts are optimized for puzzle-solving objectives - such as forcing