Defending Against Prompt Injection with DataFilter
Yizhu Wang, Sizhe Chen, Raghad Alkhudair, Basel Alomair, David Wagner, UC Berkeley, KACST · 2025
Read Defending Against Prompt Injection with DataFilter by Yizhu Wang, Sizhe Chen, Raghad Alkhudair, Basel Alomair, David Wagner, UC Berkeley, KACST (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19207.txt.
Introduction
Yizhu Wang and four co-authors present DataFilter, a defense against prompt injection attacks on language model agents. When an agent reads untrusted outside data, an attacker can hide instructions in that data and hijack the agent away from the user's task toward harmful actions.
DataFilter cleans the data before it reaches the main model, removing injected instructions at test time without needing model weights, heavy utility loss, or a rebuilt system. It is trained on simulated injections to recognize malicious text. A model-agnostic filter like this could be dropped in front of many existing agents to raise their security.
Catalog record
| CREATOR | Yizhu Wang, Sizhe Chen, Raghad Alkhudair, Basel Alomair, David Wagner, UC Berkeley, KACST |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Defending Against Prompt Injection with DataFilter Yizhu Wang, Sizhe Chen, Raghad Alkhudair, Basel Alomair, David Wagner, UC Berkeley, KACST Abstract When large language model (LLM) agents are increasingly deployed to automate tasks and interact with untrusted external data, prompt injection emerges as a significant security threat. By injecting malicious instructions into the data that LLMs access, an attacker can arbitrarily override the original user task and redirect the agent toward unintended, potentially harmful actions. Existing defenses either require access to model weights (fine-tuning), incur substantial utility loss (detection-based), or demand non-trivial system redesign (system-level). Motivated by this, we propose DataFilter , a test-time model-agnostic defense that removes malicious instructions from the data before it reaches the backend LLM. DataFilter is trained with supervised fine-tuning on simulated injections and leverages both the user’s instruction and the data to selectively strip adversarial content while preserving benign information. Across multiple benchmarks, DataFilter consistently reduces the prompt injection attack success rates to near zero while maintaining the LLMs’ utility. DataFilter delivers strong security, high utility, and plug-and-play deployment, making it a strong practical defense to secure black-box commercial LLMs against prompt injection. Our DataFilter model is released here for immediate use, with the code to reproduce our results here . I Introduction AI agents [ 1 , 2 ] have automated diverse tasks like web-navigation and tool-calling. In agents, the Large Language Model (LLM) interacts with the external environment (websites, documents, emails, etc.), where the data is untrusted and may contain a prompt injection attack [ 3 , 4 ] . By injecting a prompt into the data that LLMs access, an attacker can arbitrarily override the original user task and redirect the agent system towards unintended and potentially harmful actions. Successful prompt injection attacks against industry products [ 5 , 6 , 7 ] have been realized to cause actual harms like data leakage and malware execution. Thus, prompt injection risks hold back a broader adoption of AI agents and have been listed as the top-1 threat to LLM applications [ 8 ] . Against prompt injections, defenders have tried to secure the system outside the model (system-level defenses) or secure the model itself (model-level defenses). System-level defenses [ 9 , 10 , 11 ] offer an attractive guaranteed security by design and can be used with any existing model. However, they require non-trivial work from the agent developer to design their system in a way tailored around prompt injection robustness. Currently, system-level defenses can only be applied to a very limited set of tasks, thus rendering significant utility drop [ 12 ] . Model-level defenses [ 13 , 14 , 15 ] offer a different set of tradeoffs. They provide an exciting general defense and could protect all agents without requiring any special effort from the agent developer. However, modifying the well-trained model for security requires a delicate manipulation of the post-training pipeline to preserve the model utility. Perhaps for this reason, no major model provider currently provides secure models [ 13 ] despite consistent trials [ 16 , 17 ] . Thus, this approach might be promising in the long term, but is not an option today, especially for practically securing a production-level LLM. We propose a new defense, DataFilter , that combines some of the best aspects of system-level and model-level defenses. Specifically, we filter all queries to the LLM to remove all injected prompts (see Figure 1 ), so the LLM can operate on benign data. Like model-level defenses, our approach is easy to deploy and general. That is, an off-the-shelf DataFilter is ready for immediate protection on any agent systems with no required efforts from the agent developer. Like system-level defenses, it can be used with any model and does not require cooperation or support from the model provider. We also show that DataFilter maintains the utility of the underlying model when providing significant security. We believe it could be a practical defense in the short and medium term, despite its potential vulnerabilities against the most sophisticated attacks [ 12 , 18 ] as all existing defenses. Fig. 1 : DataFilter takes both the trusted instruction and untrusted data as input, removes potential prompt injections, and outputs the sanitized data. The backend LLM then executes the original instruction using the sanitized data. We train a small DataFilter model to filter the input. The key technical challenge is how to filter out parts of the input that might be involved in a prompt injection attack, without filtering out benign data. Prompt injection attacks can be diverse and hard to recognize, but they are all commanded by imperative sentences. Roughly speaking, we need to filter the data input out of imperative sentences, which are easy to recognize, and thus feasible to identify and delete by a reliable filter. In practice, however, some imperative sentences are not prompt injections and need to be preserved. Handling this challenge requires a non-trivial design of DataFilter’s training process. With that design, our DataFilter is much more sophisticated about locating and removing imperative sentences only if they could be a prompt injection. Empirically, we find that DataFilter is effective in deleting prompt injections that are not seen in its training. It reduces attack success rates (ASRs) from over 40% to about 2% (average over multiple benchmarks), across a range of different attack methods. Utility is reduced by about 1% (average over multiple benchmarks). Our experiments show that DataFilter provides a better security-utility tradeoff than all tested prior defenses that can directly secure any existing LLMs, see Figure 2 . In our experiments, PromptArmor [ 19 ] and sandwich prompting [ 20 ] are the two best prior model-agnostic schemes, and DataFilter is better than PromptArmor on both security and utility (average ASR 2.2% vs 5.9%, average utility drop of 1.0% vs 4.1%) and much more secure than sandwich prompting (average ASR 2.2% vs 22.8%). Therefore, we draw the community’s attention to this simple and effective mechanism for defending against prompt injection attacks. Fig. 2 : DataFilter achieves a better tradeoff between security (Attack Success Rate, ASR, \downarrow ) and utility ( \uparrow ) than any prior defense. The star indicates the best one could hope for (zero ASR without utility drop). DataFilter approaches this ideal more closely than other defenses. The ASR scores are averaged across three benchmarks: SEP [ 21 ] , InjecAgent [ 22 ] , and AgentDojo [ 23 ] . The ASR for a benchmark is calculated by the maximum ASR of various attacks (SEP, InjecAgent, and AgentDojo are tested with 6, 2, and 4 attack methods, respectively). The utility scores are averages across two benchmarks: AlpacaEval2 [ 24 ] and AgentDojo [ 23 ] . SEP and AlpacaEval2 are for instruction following; InjecAgent and AgentDojo are for agentic tool-calling. II Problem Statement II-A Prompt Injection Attack We consider a LLM-integrated application or agent. We assume it queries the LLM by providing a prompt and associated data. A LLM input in a LLM System Prompt: Summarize the strengths and weaknesses of this job candidate based on its CV. Data: Education: A University… Experience: B Company… The prompt is trusted: it is designed by the system to prompt the LLM to execute