DoomArena: A Framework for Testing AI Agents Against Evolving Security Threats

Leo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, Gabriel Huang, Abhay PuriServiceNow Research, Avinandan Bose, Maryam FazelUniversity of Washington, Seattle, Quentin CappartPolytechnique Montréal, Jason Stanley, Alexandre Lacoste, Alexandre Drouin, Krishnamurthy (Dj) DvijothamServiceNow Resear · 2025

Read DoomArena: A Framework for Testing AI Agents Against Evolving Security Threats by Leo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, Gabriel Huang, Abhay PuriServiceNow Research, Avinandan Bose, Maryam FazelUniversity of Washington, Seattle, Quentin CappartPolytechnique Montréal, Jason Stanley, Alexandre Lacoste, Alexandre Drouin, Krishnamurthy (Dj) DvijothamServiceNow Resear (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14064.txt.

Introduction

Leo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, and a multi-institution team present DoomArena in this two thousand twenty-five paper. DoomArena is a framework for testing AI agents against evolving security threats. It plugs into existing agent environments such as BrowserGym for web agents and a tau-bench setup for tool-calling agents. Threat models are configurable: users specify which components can be attacked and what the attacker targets.

The framework separates attack development from environment details, so one attack can run across many settings. Tests on leading web and tool-calling agents produced surprises. No agent dominated across all threat models, combined attacks often reinforced each other, and guardrail-model defenses failed where stronger model-based defenses held up better. The authors use these results to probe trade-offs between vulnerability and task performance.

Catalog record

CREATORLeo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, Gabriel Huang, Abhay PuriServiceNow Research, Avinandan Bose, Maryam FazelUniversity of Washington, Seattle, Quentin CappartPolytechnique Montréal, Jason Stanley, Alexandre Lacoste, Alexandre Drouin, Krishnamurthy (Dj) DvijothamServiceNow Resear
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DoomArena: A Framework for Testing AI Agents Against Evolving Security Threats Leo Boisvert, Mihir Bansal, Chandra Kiran Reddy Evuru, Gabriel Huang, Abhay PuriServiceNow Research, Avinandan Bose, Maryam FazelUniversity of Washington, Seattle, Quentin CappartPolytechnique Montréal, Jason Stanley, Alexandre Lacoste, Alexandre Drouin, Krishnamurthy (Dj) DvijothamServiceNow ResearchCorrespondence to: [email protected] , [email protected] Abstract We present DoomArena, a security evaluation framework for AI agents. DoomArena is designed on three principles: 1) It is a plug-in framework and integrates easily into realistic agentic frameworks like BrowserGym (for web agents), τ \tau -bench (for tool calling agents) and OSWorld (for computer-use agents); 2) It is configurable and allows for detailed threat modeling, allowing configuration of specific components of the agentic framework being attackable, and specifying targets for the attacker; and 3) It is modular and decouples the development of attacks from details of the environment in which the agent is deployed, allowing for the same attacks to be applied across multiple environments. We illustrate several advantages of our framework, including the ability to adapt to new threat models and environments easily, the ability to easily combine several previously published attacks to enable comprehensive and fine-grained security testing, and the ability to analyze tradeoffs between various vulnerabilities and performance. We apply DoomArena to state-of-the-art (SOTA) web and tool-calling agents and find a number of surprising results: 1) SOTA agents have varying levels of vulnerability to different threat models (malicious user vs malicious environment), and there is no Pareto dominant agent across all threat models; 2) When multiple attacks are applied to an agent, they often combine constructively; 3) Guardrail model-based defenses seem to fail, while defenses based on powerful SOTA LLMs work better. DoomArena is available at https://github.com/ServiceNow/DoomArena † † footnotetext: {\dagger} denotes equal contribution and joint primary authorship; ‡ denotes affiliation with Mila-Quebec. 1 Introduction The rise of AI agents brings up exciting possibilities to automate valuable but repetitive tasks in the enterprise ( Drouin et al., 2024 ; Xu et al., 2024 ) , in scientific applications ( Gottweis et al., 2025 ) , and in knowledge work ( OpenAI, 2025 ) . However, the existence of autonomous agents also poses several security risks, including leakage of sensitive data ( Zharmagambetov et al., 2025 ) , privileged access, the proliferation of unauthorized financial transactions, etc. Several works demonstrating such risks from poisoning attacks ( Chen et al., 2024 ) , malicious pop-ups ( Zhang et al., 2024a ) , and prompt injections ( Altimetrik, 2024 ) have recently appeared, underscoring the critical need for research into the security of AI agents. Testing systematically for these risks in a manner that is informed by the deployment context of the agent while allowing for realistic threat modeling remains an open challenge. In this paper, we present DoomArena, a modular, plug-in, and configurable framework for security testing for AI agents. DoomArena is not a benchmark in itself, but facilitates the construction of realistic security benchmarks by providing various common components required for their construction. The ability to support multiple agentic frameworks and environments in a ( plug-in ) manner adding security testing capabilities to any agentic framework, the ability to develop generic adversarial attacks that apply across multiple agents and environments ( modular ), the ability to configure security testing by tagging specific components in the agent-user-environment loop as untrusted or potentially malicious, thus constraining potential adversarial attacks to arise only from plausible attack surfaces ( configurable ). DoomArena facilitates the injection of inference-time attacks in any of these components. Although it does not focus on training-time attacks, it can be used to evaluate them by inserting relevant triggers in the environment. We demonstrate the advantages of DoomArena in several ways: 1) We implement several well-known attacks and show how they can be easily combined via attack configurations in our framework, supporting security evaluations in the face of an evolving landscape of risks. 2) We show how DoomArena facilitates fine-grained security analysis, leading to a refined understanding of which agents are more or less susceptible to which attacks and under what conditions. 3) We show how these capabilities enable DoomArena to be used as laboratory for AI agent security research , and also use it to analyze the security of state-of-the-art agents on benchmarks for web agents (BrowserGym ( de Chezelles et al., 2025 ) ) and tool-calling agents ( \tau -Bench ( Yao et al., 2024 ) ), while further demonstrating its extensibility to other domains like computer-use agents (OSWorld ( Xie et al., 2024 ) ), uncovering interesting trends on the vulnerabilities of frontier LLM-based agents in different settings. 2 Related Work Several recent works document various attacks against AI agents. These include exploiting untrusted elements in the environment to inject prompts into agents ( Liao et al., 2024 ) , injecting visual injections into Vision-Language Model-based agents ( Wu et al., 2025 ) , using pop-ups to misdirect AI agents interacting with browsers and computers ( Zhang et al., 2024a ) , and executing jailbreak attacks that bypass safety guardrails in browser agents ( Perez & Ribeiro, 2022 ; Xu et al., 2023 ; Wei et al., 2023 ; Gong et al., 2023 ) . Recent research has revealed concerns about the gaps between the safety refusal capabilities of standalone LLMs and their agent implementations ( Kumar et al., 2024 ; Chiang et al., 2025 ) . For example, Kumar et al. (2024) found that while backbone LLMs often refuse to follow harmful instructions, their corresponding agents frequently execute these same instructions when deployed in browser environments. AI agents are vulnerable when user inputs are embedded into system prompts ( Chiang et al., 2025 ) , enabling attackers to exploit novel vulnerabilities in agentic AI systems like confidential data leaks, privilege escalation, etc. While prior work highlights these risks, deploying agents requires a systematic testing framework tailored to real-world threats . DoomArena provides this by enabling researchers to assess risks in a deployment-specific context. We organize prior work on safety/security benchmarks for AI agents into three categories: Static benchmarks: Static benchmarks ( Kumar et al., 2024 ; Andriushchenko et al., 2024 ; Mazeika et al., 2024 ; Zeng et al., 2024 ) use curated (human-generated/manual) malicious prompts to assess AI agent risks across harm categories like fraud, cybersecurity, hate speech, etc. AgentHarmBench ( Andriushchenko et al., 2024 ) , for instance, includes 110 malicious tasks spanning 11 harm categories; while useful for broad safety evaluations, many risks only emerge in interactive settings where agents process inputs from users and the environment. Stateful safety/security benchmarks: Unlike static evaluations, AI agents operate statefully, interacting with users and environments over multiple steps. SafeArena ( Tur et al., 2025 ) assesses the safety of autonomous web agents across 250 safe and 250 harmful tasks spanning four websites and five harm categories, revealing that models like GPT-4o ( OpenAI, 2024 ) and Qwen-2-VL ( Yang et al., 2024 ) complete a significant percentage of harmful tasks. Similarly, BrowserART ( Kumar et al., 2024 ) red-teams browser agents with 100 diverse browser-related harmful behaviors, showing that agents often fail safety standards despite backbone LLM refusing such behaviors. ST-WebAgentBench ( Levy et al., 2024 ) evaluates web agents’ safety and trustworthiness across six

Read the complete text