Security–Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense
Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang · 2026
Read Security–Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense by Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30783.txt.
Introduction
Mitchell Hermon and four colleagues expose a cost of defending language models against prompt injection. Defences work by suppressing untrusted text. That is fine until the task must preserve the text, as in translation or document editing. To attack-success metrics, a model that ignores an injection looks identical to one that processes it as data. Security scores cannot tell the two apart.
Their benchmark, SecFid, constructs tasks where an injection may be executed, processed as data, or ignored, giving different outputs. Across forty-eight configurations, no model or defence achieved both. The highest-fidelity model kept ninety-six point five percent fidelity but only about forty-eight percent security. The most secure defences inverted the split: above ninety-nine percent security, fidelity near seventy-one to seventy-four percent. The authors argue the trade-off belongs to each deployment's costs, not to the defence.
Catalog record
| CREATOR | Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30783
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Security–Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang Abstract We identify a security–fidelity tradeoff in defending LLMs against indirect prompt injection: defenses resist injected instructions largely by suppressing untrusted text, which corrupts tasks that must preserve it, such as translation and document editing. Attack-success metrics cannot see this, because a model that ignores an injection and one that faithfully processes it as data score identically. We introduce SecFid , a benchmark built so that executing an injection, processing it as data, and ignoring it produce distinguishable outputs. This makes fidelity measurable, and exposes a frontier: across 1,168 1{,}168 examples and 48 48 configurations, no model or defense achieves both objectives. The highest-fidelity model reaches 96.5 % 96.5\% fidelity at 47.8 % 47.8\% security, while the most secure defenses invert this, at 99.3 % 99.3\% security but only 71.0 % 71.0\% – 73.9 % 73.9\% fidelity. Even defenses with identical security differ in how they earn it: some repair hijacks into faithful processing, others simply suppress benign content. A decision-theoretic analysis shows why no fixed choice can be right everywhere: the correct behavior is not a property of the defense but of the deployment, set by its relative cost of a hijack versus a dropped span. Security alone therefore measures only half of robustness, and reporting it without fidelity hides the price at which it was bought. 1 Introduction Prompt injection exploits a failure of instruction–data separation, in which an LLM treats untrusted text as a command rather than as content to process. The risk is intrinsic to context-augmented systems, where trusted task instructions share a prompt with webpages, emails, documents, search results, and API responses. These systems must use untrusted text without letting it control the model, which makes prompt injection a central security problem for LLM applications ( 27 ; 14 ; 26 ) . Defenses now span input filtering, injection classifiers, sanitization, instruction hierarchies, architectural separation, and safety fine-tuning ( 9 ; 24 ; 18 ; 34 ; 7 ) . All are evaluated on whether the model executed the injected instruction ( 27 ; 23 ; 41 ; 11 ) . This captures whether untrusted text was followed as a command, but it says nothing about whether the text was preserved as data when the task required it. A model can therefore score as secure in two different ways. It can treat the injected span as task data, or it can ignore it. Both behaviors lower attack success, but only the first keeps the content the task needs. Figure 1: Task-dependent interpretation of instruction-like text in untrusted inputs. The same product review contains an instruction-like sentence (Review B) that could be an indirect prompt injection. For translation , the sentence should be translated faithfully. For summarization , the sentence must not control the summary and may be summarized as content or omitted as irrelevant, depending on the task objective. The ideal behavior depends on the task. The distinction matters because instruction-like content is ubiquitous in data. Emails contain requests; documentation contains commands and setup steps; legal text quotes obligations; and transcripts contain imperative speech. Whether a model should process such content or drop it depends on the task. A translation system must render an instruction-like sentence faithfully, while a summarizer must not let the same sentence steer its output (Figure 1 ). When execution risk dominates a deployment, suppressing suspicious text could be acceptable, but if a task requires faithful preservation or transformation of its input, that same suppression becomes a fidelity failure, and one that attack-success metrics cannot detect. We close this gap with SecFid , an evaluation that makes execution, processing, and suppression separately observable, and we use it to map how current models and defenses navigate the resulting security–fidelity tradeoff. We find no configuration that achieves both objectives. Models and defenses instead spread along a frontier, and defenses with matched hijack rates differ sharply in whether non-execution reflects repair or suppression. Our contributions are as follows. • We show that non-execution is ambiguous and cannot, on its own, distinguish a model that separates instruction from data from one that suppresses the data, so attack-success metrics conflate the two. • We introduce SecFid , which makes that distinction measurable . 1,168 core instances and an agentic extension, constructed so that executing, processing, and ignoring a probe yield distinguishable outputs, decomposing behavior into security and fidelity. • We map an empirical security–fidelity tradeoff across 48 configurations, and reveal that defenses with similar hijack rates can reach security with diverging levels of fidelity. • We show the tradeoff is deployment-dependent , with the optimal operating point set by the costs of hijacking versus suppression, so no fixed robustness policy is optimal everywhere. 2 Threat Model We study indirect prompt injection in context-augmented LLM applications. A model M receives a trusted task instruction s and an untrusted data payload d , such as a retrieved passage, document, webpage, API response, or other externally supplied content. An attacker embeds adversarial text z into the data, producing d_{z} , and the model produces output y=M(s,d_{z}) . We refer to z as an adversarial probe , or simply a probe : inserted adversarial text constructed to make the model’s behavior observable. A probe is not part of the trusted task instruction. It appears within untrusted data, where the model should treat it as data for the task rather than as a command. The attacker controls z but cannot modify s , the model weights, or the inference pipeline. The attacker’s goal is instruction hijacking : causing the model to treat z as a control signal rather than data, so that y reflects the probe’s directive instead of the trusted task instruction. We evaluate fixed, non-adaptive probes: the attacker chooses z before observing the model output and does not adapt it to a particular model response, defense decision, or evaluator. The probes are measurement instruments rather than a complete taxonomy of malicious goals; they are constructed so that executing, processing, and ignoring the injected span produce distinguishable outputs. The defender wants the model to follow the trusted task instruction s reliably even when d contains adversarial text. This requires both security and fidelity : the model should not execute the probe as an instruction, but it also should not suppress instruction-like content that the task requires as data. 3 The SecFid Benchmark Prompt injection can be viewed as a failure of instruction-data separation, in which the model must follow trusted task instructions while treating external content as data to be processed rather than commands to be followed ( 14 ; 44 ) . Existing prompt-injection benchmarks measure only one side of this requirement. They ask whether the model executed an instruction injected within untrusted data, but they cannot tell whether an injection that escaped execution was actually treated as data. This matters because a model can avoid executing an injected instruction in two qualitatively different ways, either by processing the injected text as part of the data or by suppressing the text entirely. Both behaviors reduce attack success, and both can preserve apparent utility when the surrounding task output still looks