How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
2026
Read How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15714.txt.
Introduction
Dziemian and co-authors report results from a public red-team contest on agent safety. Hidden instructions in emails, files, or code can steer AI agents without user knowledge. The worst attacks also hide all traces from the final reply. The contest drew 464 people and 272,000 attack attempts. Some 8,648 attacks succeeded across 41 scenarios and 13 frontier models. Success rates ran from 0.5 to 8.5 percent by model. Shared attack tricks crossed 21 behaviors and many model families.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36245 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15714
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition Abstract LLM based agents are increasingly deployed in high stakes settings where they process external data sources such as emails, documents, and code repositories. This creates exposure to indirect prompt injection attacks, where adversarial instructions embedded in external content manipulate agent behavior without user awareness. A critical but underexplored dimension of this threat is concealment: since users tend to observe only an agent’s final response, an attack can conceal its existence by presenting no clue of compromise in the final user facing response while successfully executing harmful actions. This leaves users unaware of the manipulation and likely to accept harmful outcomes as legitimate. We present findings from a large scale public red teaming competition evaluating this dual objective across three agent settings: tool calling, coding, and computer use. The competition attracted 464 participants who submitted 272,000 attack attempts against 13 frontier models, yielding 8,648 successful attacks across 41 scenarios. All models proved vulnerable, with attack success rates ranging from 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro). We identify universal attack strategies that transfer across 21 of 41 behaviors and multiple model families, suggesting fundamental weaknesses in instruction following architectures. Capability and robustness showed weak correlation, with Gemini 2.5 Pro exhibiting both high capability and high vulnerability. To address benchmark saturation and obsoleteness, we will endeavor to deliver quarterly updates through continued red teaming competitions. We open source the competition environment for use in evaluations, along with 95 successful attacks against Qwen that did not transfer to any closed source model. We share model-specific attack data with respective frontier labs, and the full dataset with the UK AISI and US CAISI to support robustness research. Mateusz Dziemian ∗ , Maxwell Lin ∗ , Xiaohan Fu ∗ , Micha Nowak ∗ , Nick Winter, Eliot Jones, Andy Zou † , Matt Fredrikson ‡ , Zico Kolter ‡ Gray Swan AI Lama Ahmad OpenAI Klaudia Krawiecka Meta Kamalika Chaudhuri Meta Riccardo Patana Anthropic Sahana Chennabasappa Meta Neil Perry US CAISI Xander Davies UK AISI Troy Peterson OpenAI Lauren Deason Meta Xiangyu Qi OpenAI Benjamin L. Edelman US CAISI Javier Rando Anthropic Tanner Emek Anthropic Zifan Wang Meta Ivan Evtimov Meta Zihan Wang Meta Jim Gust Meta Spencer Whitman Meta Maia Hamin US CAISI Eric Winsor UK AISI Kat He Meta Arman Zharmagambetov Meta ∗ Lead contributors. ‡ Senior Authors. † Co-affiliated with Carnegie Mellon University and Center for AI Safety. 1 Introduction LLM-based agents have rapidly advanced from simple chatbots to autonomous systems capable of completing complex, multi-step tasks across extended time horizons, with recent evaluations demonstrating that frontier models can now solve software engineering tasks requiring hours of sustained work ( kwa2025measuringaiabilitycomplete ) . Improvements in multimodal capabilities further enable agents to process and act on visual interfaces ( anthropic2024computeruse ) , spreadsheets ( anthropic2025excel ) , and audio inputs ( openai2024gpt4o ) . These capabilities are driving rapid enterprise deployment, with 88% of organizations now using AI in at least one business function and 62% experimenting with AI agents ( mckinsey2025stateofai ) . Agents already assist with software development ( cursor2024 ; githubcopilot ) and are increasingly deployed in high-stakes domains including healthcare and financial services. However, this expanded autonomy introduces significant security vulnerabilities ( lupinacci2025darkside ; li2025commercial ) . Particularly concerning is indirect prompt injection , attacks where adversarial instructions embedded in external data sources (emails, documents, websites, code) manipulate an agent to accomplish attacker-specified goals undesired by the user which could lead to financial loss or personal data leakage ( greshake2023indirect ) . Gemini 2.5 Pro Nova 1 Premier SecAlign 70B DeepSeek V3.1 Kimi K2 Nova 2 Lite Qwen3 VL 235B Grok 4 GPT-5.1 GPT-5 Claude Haiku 4.5 Claude Sonnet 4.5 Claude Opus 4.5 0 2 4 6 8 10 8.5 1670/19630 5.8 1091/18924 5.5 618/11238 5.4 808/15026 4.8 589/12299 4.7 672/14244 4.2 781/18733 2.9 366/12526 2.5 202/8096 2.0 261/13143 1.3 177/13696 1.0 136/13603 0.5 61/11969 Model ASR (%) Model Family Google Amazon Meta DeepSeek MoonshotAI Qwen xAI OpenAI Anthropic Figure 1 : ASR by model across the competition (wave 1 and wave 2) in decreasing order. ASR is computed by successful attacks / total attempts. A critical yet underemphasized aspect of this threat is concealment . A compromised model may not faithfully disclose that it has been manipulated to execute attacker-specified actions in its final response, and may even fabricate plausible explanations for actions that are in fact irrelevant or malicious. Although most agentic interfaces expose a tool execution history that could, in principle, help users identify such manipulation, the length and complexity of these logs make effective human inspection prohibitively difficult in practice. Monitoring the chain-of-thought (CoT) has also been proposed as a mitigation ( korbak2025cot_monitorability ) ; however, the verbosity and complexity of reasoning traces similarly render this approach difficult. Consequently, an attack can remain practically concealed as long as no explicit clues appear in the final response, even though the attacker’s objectives have already been successfully achieved. In this work, we focus on investigating the robustness of major models against indirect prompt injection attacks that satisfy this additional concealment requirement. We curate 41 unique agentic scenarios covering text-based tool use, coding agent, and computer use simulating various real world use cases. In each scenario, we consider an attacker who controls a fixed portion of an external data (such as a tool response, a file, or a webpage) involved as the last segment of a multi-turn agentic transcript. The target model ingests the transcript which carries the attack string and operates one turn with potentially multiple tool call rounds. The attack needs to fulfill the following two objectives: 1. force the model to achieve the target harmful goal in this assistant operation turn; 2. pass scenario-specific criteria on the model’s final response, such as concealment. To collect the most up to date and highest quality attacks, we hosted a large scale competition named Indirect Prompt Injection Arena collaborating with frontier labs and government AI institutes including the UK AISI 1 1 1 https://www.aisi.gov.uk and US CAISI 2 2 2 https://www.nist.gov/caisi . This three-week competition had a $40k total prize pool and attracted 464 total unique participants, who contributed 240K+ total attack attempts, 8K+ of which were successful. We observe an overall ASR ranging 0.5% to 8.5% across 13 participating models spanning major proprietary and open-weight ones (see Figure 1 ). Static benchmarks become obsolete quickly. Frontier models have saturated on capability benchmarks such as MMLU(-Pro), GSM8K, and HumanEval with >90\% correctness within less than a year of their release. Security benchmarks face an even worse situation: defense and attack techniques are both evolving quickly. The latest adaptive attacks can easily bypass defense mechanisms that were reporting 0% ASR ( nasr2025attacker ) , while existing static benchmarks are still carrying out of date attacks on old generations of models which are becoming irrelevant. Noting the unique challenge here, we will endeavor to host this competition in a recurrent format, with new scenarios designed each time and the latest models tested each time, to ensure the currency and quality of this benchmark over time. Contributions. • Concealment-aware prompt injections. We for