The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities
Ron Litvak · 2026
Read The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities by Ron Litvak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25056.txt.
Introduction
Litvak shows that a mail agent's system prompt can make or break its phishing defense. He tests 11 models under 10 prompt plans. One model's bypass rate swings from under 1 percent to 97 percent with setup alone. Tuning prompts on strong clues lifts recall, but it also builds a brittle shield. A sender-domain match works until attackers register matching domains and flip the clue. Most wins for attackers follow that flipped clue. He proposes a score that weighs false alarms, and argues tools with outside facts are needed.
Catalog record
| CREATOR | Ron Litvak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92070 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25056
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities Ron Litvak Abstract Large language models are increasingly deployed as autonomous email agents that make implicit security decisions about incoming messages. Yet no systematic study has examined how the configuration of these agents shapes their security posture. We present PhishNChips, a large-scale benchmark of 220,000 evaluations spanning 11 models and 10 system prompt strategies. Our central finding is that, within this benchmark, prompt-model interaction strongly shapes security outcomes. A single model’s phishing bypass rate varies from under 1% to 97% across prompt configurations, while the false-positive cost of a given prompt varies by an order of magnitude across models. We develop signal-based prompt strategies achieving up to 93.7% recall at 3.8% false positive rate on the benchmark distribution, with gains that generalize cross-model without modification. However, these gains are both distribution-sensitive and adversarially brittle: domain-matching strategies perform well on a benchmark whose legitimate corpus is dominated by sender–URL-matched emails, yet fail sharply under signal inversion , where an adversary satisfies the privileged heuristic by construction. In our case, domain-matching defenses lose up to half their recall when attackers register matching infrastructure. Response-trace analysis of verbose bypass responses shows that 98% reason in ways consistent with the inverted signal – the models execute the instruction faithfully, but the instruction’s assumption is false. A counter-intuitive corollary emerges: narrowing instructions to specific signals can degrade already-capable models by up to 19 percentage points, replacing multi-signal reasoning with exploitable single-signal dependence. We characterize the resulting tension between detection, usability, and adversarial robustness as a navigable tradeoff rather than a hard impossibility, introduce Safetility – a deployability-aware metric that surfaces how few configurations remain attractive once false-positive costs are penalized – and argue that closing the adversarial gap likely requires tool augmentation that complements prompt-based reasoning with external ground truth. Keywords: LLM security, phishing detection, prompt engineering, email agents, adversarial robustness, system prompt, benchmark 1 Introduction Large language model-powered email assistants are no longer research prototypes. They are being integrated into enterprise productivity suites, customer service pipelines, and personal workflow automation, where they read, summarize, draft replies to, and in some configurations autonomously act upon incoming email. When such a system encounters a phishing message, it makes – implicitly or explicitly – a binary security decision. The consequence of error is not an incorrect summary or an awkward draft reply; it is credential theft, session hijacking, or full account compromise. The stakes of this decision are asymmetric and immediate, yet the factors that determine how an LLM-based agent makes it remain poorly understood. The limitations of human-centric defense reinforce this urgency: 11 found in a controlled study of 12,511 employees that neither lecture-based nor interactive anti-phishing training produced statistically significant reductions in phishing click rates ( p=0.450 , all effect sizes below 0.01) – strengthening the case for automated, technology-driven defenses. Yet no prior work has systematically examined how the configuration of LLM-based email agents – specifically, the system prompt that defines their persona and decision criteria – affects their ability to detect phishing. The conventional framing of this problem asks an incomplete question. Enterprises evaluating LLM email agents naturally ask: which model should we choose? The implicit assumption is that security performance is primarily a function of model architecture, training data, or parameter count – that some models are inherently better at detecting phishing than others. Model differences do exist – but in our benchmark they are often overshadowed by deployment configuration. We show that the same model, given different system prompt configurations, varies from near-perfect phishing detection to near-total blindness. GPT-4o-mini’s phishing bypass rate swings from under 1% under a security-first persona to 97% under an efficiency-focused persona, and its recall reaches 93.7% under an optimized signal-based strategy – a range attributable entirely to the words in the system prompt. This within-model variance exceeds the between-model variance observed at many fixed prompt configurations in our evaluation. The system prompt is not merely a deployment detail; in this setting, it is a first-order security variable. This finding has a natural consequence. If prompt design is a major lever controlling security behavior, then prompts should be engineered for security with the same rigor applied to model selection or fine-tuning. We pursue this logic directly, developing a family of signal-based prompt strategies that direct models to attend to specific, empirically discriminative features of phishing emails – most notably, sender–URL domain consistency. Through iterative optimization, these strategies push multiple models to high recall at low false positive rates within the benchmark, and critically, the resulting gains generalize cross-model without modification. A strategy optimized on one model transfers to architecturally distinct models with minimal performance degradation, suggesting that the signal-based framing aligns with features that are robustly detectable across model families. At the same time, these optimized gains must be interpreted relative to the evaluation distribution. Our legitimate corpus is dominated by sender–URL-matched emails, which makes sender–URL consistency unusually clean as a decision signal inside the benchmark. Later we show that legitimate cross-domain email can sharply raise false positive rates under the same strategies. We treat this not as a footnote but as part of the central lesson: prompt optimization can look near-optimal when it aligns with benchmark regularities while concealing structural brittleness. This is where the story would end if attackers were static. They are not. We show that the very optimization that produces high recall creates a specific, documentable attack surface. Our signal-based strategies rely heavily on sender–URL domain inconsistency as a phishing indicator – a signal that is both highly discriminative against commodity phishing and trivially controllable by an adversary. When we introduce infrastructure phishing , in which the attacker registers a domain that matches the URL in their phishing email, optimized strategies lose up to half their recall. To understand why, we collect and analyze verbose model responses on infrastructure phishing samples. The result is unambiguous: the vast majority of successful bypasses contain explicit reasoning that cites sender–URL domain consistency as evidence of legitimacy. The models are not failing to reason. They are reasoning correctly from a signal the attacker controls. The paper thus presents a three-act argument. In Act I, we show that system prompt configuration can rival or exceed model choice as a security variable in this benchmark, and that principled prompt engineering can achieve strong detection performance. In Act II, we demonstrate that this optimization is structurally brittle – it creates the very vulnerability an adaptive adversary exploits. In Act III, we show that an infrastructure-aware prompt can partially recover robustness, but the cost of that recovery varies by an order of magnitude across models – revealing that model-specific pre-deployment dispositions determine how efficiently any prompt instruction is executed. The conclusion is not that optimization is misguided, but that single-signal optimization against a fixed threat model produces defenses that are legible to the adversary and defeatable by construction. We characterize this tension as a navigable three-way tradeoff between commodity recall, false positive rate, and infrastructure robustness: across all 220,000 evaluations, no tested configuration maximizes all three, but the tradeoff space can be navigated through prompt-model co-optimization. A counter-intuitive corollary threads through all three acts: