OpenAgentSafety : A Comprehensive Framework for Evaluating Real-World AI Agent Safety

Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap · 2025

Read OpenAgentSafety : A Comprehensive Framework for Evaluating Real-World AI Agent Safety by Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06134.txt.

Introduction

Sanidhya Vijayvargiya and co-authors present OpenAgentSafety, a test frame for the safety of AI agents in real settings, in a 2025 paper. Agents now book, code, and message on real tools, which opens room for unsafe acts. Past test sets leaned on faked worlds or narrow tasks.

The frame runs agents on real browsers, code runners, file stores, shells, and chat apps. It holds more than three hundred fifty multi-step tasks across eight risk types, with both kind and hostile user aims. Rule checks pair with model judges to catch plain and sly unsafe acts. Five top models failed often: unsafe acts showed in about half to near three quarters of the risky tasks.

Catalog record

CREATORSanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenAgentSafety : A Comprehensive Framework for Evaluating Real-World AI Agent Safety Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap Abstract Recent advances in AI agents capable of solving complex, everyday tasks—from scheduling to customer service—have enabled deployment in real-world settings, but their possibilities for unsafe behavior demands rigorous evaluation. While prior benchmarks have attempted to assess agent safety, most fall short by relying on simulated environments, narrow task domains, or unrealistic tool abstractions. We introduce OpenAgentSafety , a comprehensive and modular framework for evaluating agent behavior across eight critical risk categories. Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms; and supports over 350 multi-turn, multi-user tasks spanning both benign and adversarial user intents. OpenAgentSafety is designed for extensibility, allowing researchers to add tools, tasks, websites, and adversarial strategies with minimal effort. It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors. Empirical analysis of five prominent LLMs in agentic scenarios reveals unsafe behavior in 51.2% of safety-vulnerable tasks with Claude-Sonnet-3.7, to 72.7% with o3-mini, highlighting critical safety vulnerabilities and the need for stronger safeguards before real-world deployment. 1 Introduction The development of AI agents has surged, fueled by LLM breakthroughs that enable their deployment for software engineering [ 32 ] , web browsing [ 45 ] , and customer service tasks [ 14 ] among others. The rapid pace of their development has far outpaced progress on ensuring their safety. Agents are increasingly granted access to powerful tools that enable them to perform complex, multi-step tasks autonomously. Driven by competitive pressure and a large economic incentive to deploy, many agentic systems have been released without a thorough investigation into their failure modes or societal impacts [ 14 , 23 ] . The gap between capability advancement and safety assurance continues to widen, and creates vulnerability to both catastrophic failures and subtle but pervasive harms that could prove difficult to reverse once embedded in societal systems [ 43 ] . To address these risks, a number of benchmarks have recently been proposed ( Table 1 ). While valuable, existing benchmarks are often limited in scope as many rely on toy environments or simulated tool APIs, focus on narrow domains like browsing or coding, or omit multi-turn, multi-user interactions. These limitations make it difficult to evaluate how agents behave in complex, realistic situations. As agents become increasingly capable, it is critical to build benchmarks that accurately reflect real-world challenges, including diverse tools, varied user behavior, and long-horizon tasks. Table 1: Comparison of agent safety benchmarks based on their support for (i) real-world tool usage, (ii) diverse user intentions, and (iii) multi-turn user interactions. Only our benchmark, OpenAgentSafety , supports all three. In the “Diverse intents“ column, denotes support for benign user goals (e.g., a user unintentionally exposing a private key), and indicates support for malicious goals (e.g., a user attempting to coerce the agent into generating ransomware). Benchmark Real-world tools Diverse intents User interaction SALAD-Bench [ 17 ] ✗ ✗ h4rm3l [ 7 ] ✗ ✗ SafeBench [ 38 ] ✗ ✗ Agent-SafetyBench [ 43 ] ✗ ✗ SG-Bench [ 20 ] ✗ ✗ SafeAgentBench [ 37 ] ✗ ✗ ChemSafetyBench [ 44 ] ✗ ✗ LM-Emulated Sandbox [ 26 ] ✗ ✗ AdvWeb [ 35 ] ✓ ✗ Refusal-Trained LLMs [ 13 ] ✓ ✗ RedCode [ 9 ] ✓ ✗ From Interaction to Impact [ 42 ] ✓ ✗ PrivacyLens [ 28 ] ✓ ✗ Dissecting Adversarial [ 34 ] ✓ ✗ Infrastructure for AI [ 4 ] ✗ ✓ R-Judge [ 39 ] ✗ ✓ Trembling House [ 19 ] ✗ & ✗ AgentHarm [ 2 ] ✗ & ✗ WildTeaming [ 11 ] ✗ & ✗ SafetyPrompts [ 25 ] ✗ & ✓ ST-WebAgentBench [ 15 ] ✓ ✓ Frontier Models [ 18 ] ✓ ✓ SafeArena [ 31 ] ✓ & ✗ Haicosystem [ 46 ] ✗ & ✓ OpenAgentSafety (Ours) ✓ & ✓ We introduce OpenAgentSafety ( OA-Safety , § 2 ), a comprehensive and open-source simulation framework for evaluating the safety of AI agents in realistic, high-risk scenarios, supporting: • Real-world, comprehensive tool suite: Agents interact with actual file systems, command lines, code execution environments, and web interfaces. • Diverse user intentions: Tasks simulate user behavior from benign ambiguity to manipulation. • Multi-turn, multi-agent dynamics: Scenarios include extended interactions involving users and secondary actors (NPCs) such as colleagues and customers with conflicting goals. Figure 1: LLM-based agent unsafe behavior rates in the OpenAgentSafety framework while navigating conflicting user and NPC instructions. OA-Safety is built on a robust and modular infrastructure that supports sandboxed agent execution with real tools, including file systems, bash terminals, code execution environments, web browsers, and messaging platforms. It includes over 350 executable tasks, each simulating multi-turn interactions with users exhibiting benign, ambiguous, or adversarial intent, where adversarial users may appear cooperative but subtly aim to induce harmful agent behavior. The framework is designed for extensibility: researchers can easily add new tasks, simulated environments (e.g., websites), complex social dynamics (e.g., negotiation with a customer), and customized evaluators. Simulation is a core design principle in OA-Safety and agents operate inside isolated containers with full tool access, enabling realistic actions (e.g., deleting files, leaking data, sending messages) to be executed safely without real-world consequences. Each OA-Safety task is implemented as a modular Docker container that includes: the task description, multiple user goals, social dynamics, and customized evaluators, including both: rule-based evaluator that detects concrete environment impacts such as file deletion, and LLM-as-a-judge assessments to analyze the agent’s intermediate reasoning to flag attempted unsafe actions, even if those actions were incomplete or ultimately unsuccessful. This design allows for efficient environment reuse and flexible task extension. We evaluate five prominent LLMs on OpenAgentSafety and find that current agents exhibit a wide range of unsafe behaviors across realistic, multi-turn scenarios (§ 3 ). Unsafe actions occur in 51.2% to 72.7% of safety-vulnerable tasks ( Figure 1 ). We confirm prior findings that access to the browsing tool can increase the risk of unsafe behavior by overloading the agent’s context [ 31 ] . Beyond this, our analysis reveals new failure modes that are underexplored in existing safety benchmarks: we establish agents frequently fail to reason over extended multi-turn interactions, leading individually safe steps to compound into unsafe outcomes; they disregard legal, privacy, and security policies even in high-risk settings; and they show structurally unsafe behavior patterns across diverse user intents and tool types. Unlike prior studies, we analyze the impact of different risk categories, user intents, and tool usage, allowing us to uncover patterns in unsafe agent behavior not captured by single-axis benchmarks. Our contributions are: • We introduce OpenAgentSafety , 1 1 1 Code and data at: https://github.com/sani903/OpenAgentSafety a modular and extensible simulation framework with 350+ executable tasks covering eight key safety risk categories. • We propose hybrid evaluations with rigorous, rule-based and flexible, neural-based assessments. • We present a detailed empirical analysis across five LLMs, revealing failure modes and agent vulnerabilities in realistic deployment scenarios. Figure 2: An overview of the OpenAgentSafety framework. 2 OpenAgentSafety

Read the complete text