BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments
Yuxuan Li, Yi Lin, Peng Wang, Shiming Liu, Xuetao Wei · 2026
Read BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments by Yuxuan Li, Yi Lin, Peng Wang, Shiming Liu, Xuetao Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25747.txt.
Introduction
Li and co-authors present BeSafe-Bench, a benchmark for the safety of situated agents. Such agents act in web, mobile, and embodied settings. Current tests rely on fake environments or narrow tasks, the authors argue. Their bench uses working environments across four domains. Tasks are spiced with nine kinds of safety-critical risks. Grading mixes rule checks with an LLM judge. They tested 13 popular agents. Even the best finished under forty percent of tasks while fully obeying safety rules. Strong task scores often came with severe safety breaks.
Catalog record
| CREATOR | Yuxuan Li, Yi Lin, Peng Wang, Shiming Liu, Xuetao Wei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51490 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25747
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments Yuxuan Li, Yi Lin, Peng Wang, Shiming Liu, Xuetao Wei Abstract The rapid evolution of Large Multimodal Models (LMMs) has enabled agents to perform complex digital and physical tasks, yet their deployment as autonomous decision-makers introduces substantial unintentional behavioral safety risks. However, the absence of a comprehensive safety benchmark remains a major bottleneck, as existing evaluations rely on low-fidelity environments, simulated APIs, or narrowly scoped tasks. To address this gap, we present BeSafe-Bench (BSB), a benchmark for exposing behavioral safety risks of situated agents in functional environments, covering four representative domains: Web, Mobile, Embodied VLM, and Embodied VLA. Using functional environments, we construct a diverse instruction space by augmenting tasks with nine categories of safety-critical risks, and adopt a hybrid evaluation framework that combines rule-based checks with LLM-as-a-judge reasoning to assess real environmental impacts. Evaluating 13 popular agents reveals a concerning trend: even the best-performing agent completes fewer than 40% of tasks while fully adhering to safety constraints, and strong task performance frequently coincides with severe safety violations. These findings underscore the urgent need for improved safety alignment before deploying agentic systems in real-world settings. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : An overview of the BeSafe-Bench. BeSafe-Bench generates a safety-critical task dataset by integrating initial tasks with predefined safety risk types and factors. Through multi-round dynamic interactions between agents and functional environments, environment states, and agent trajectories are recorded. These data are subsequently processed via a hybrid evaluation framework to assess task completion rates and safety coverage. Table 1: Comparison of various agent safety evaluation benchmarks versus BeSafe-Bench. Functional Env. represents that the tools and interfaces within the environment possess actual functionalities rather than being text-based simulations generated by LLMs. Task Perf. represents that the benchmark evaluates both task completion and safety performance simultaneously. Dynamic Interact. represents whether the benchmark requires agents to engage in dynamic interactions with the environment. Benchmark Functional Env. Task Perf. Dynamic Interact. Web Mobile Embodied VLM Embodied VLA Task Number SafeBench ( 45 ) ✗ ✗ ✗ ✓ ✗ ✗ ✗ 2300 SafeAgentBench ( 44 ) ✗ ✓ ✓ ✗ ✗ ✓ ✗ 750 R-judge ( 46 ) ✗ ✗ ✗ ✓ ✓ ✗ ✗ 569 Agent-SafetyBench ( 50 ) ✗ ✗ ✓ ✓ ✓ ✗ ✗ 2000 OpenAgentSafety ( 32 ) ✓ ✗ ✓ ✓ ✗ ✗ ✗ 350 ToolEmu ( 27 ) ✗ ✗ ✓ ✓ ✓ ✗ ✗ 144 ST-WebAgentBench ( 15 ) ✓ ✓ ✓ ✓ ✗ ✗ ✗ 222 IS-Bench ( 21 ) ✓ ✓ ✓ ✗ ✗ ✓ ✗ 161 BeSafe-Bench (ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ 1312 The rapid development of AI agents, catalyzed by transformative breakthroughs in Large Multimodal Models (LMMs) ( 37 ) , has extended their reach across a wide spectrum of domains, including web automation ( 30 ; 9 ) , mobile applications ( 20 ; 43 ) , and embodied intelligence. However, the velocity of this technological advancement has significantly outpaced the development of corresponding safety safeguards. As autonomous agents are increasingly empowered with sophisticated tools to execute complex, multi-step tasks, many systems are being deployed without rigorous assessment of their potential safety risks ( 26 ) . There exists a profound discrepancy between an agent’s task-completion proficiency and its behavioral safety assurance. Such a gap not only risks catastrophic failures but also leads to subtle yet pervasive harms—ranging from the unauthorized leakage of sensitive personal or corporate data on public platforms to physical collisions caused by robotic manipulators in embodied settings ( 52 ; 25 ; 27 ) . Unlike the harmful textual outputs directly generated by LLMs ( 57 ; 36 ; 38 ) , the consequences of agent-driven behavioral risks manifest more directly in the physical and digital world, yet they remain significantly more challenging to analyze and evaluate. Consequently, establishing a robust benchmark for assessing the behavioral safety of AI agents is both essential and imperative. Existing research on agent safety evaluation primarily focuses on the safety of generated content within text-based scenarios ( 38 ; 3 ; 51 ) . However, as agents are increasingly equipped with tools to execute actions, evaluating content safety alone has become insufficient. To address this challenge, several studies have proposed safety assessments centered on the consequences of agent actions, a concept formally defined as behavioral safety ( 46 ; 50 ) . Despite their contributions, these existing frameworks often suffer from a limited scope of applicability. Most are confined to a single domain, such as web navigation or embodied AI, or they rely on LLMs to simulate environmental interactions ( 27 ; 23 ) . While such simulations can broaden benchmark coverage, the LLM-modeled APIs tend to be overly simplistic and static. This stands in stark contrast to the dynamic and diverse interaction patterns between agents and environments in the real world, creating a significant fidelity gap. Consequently, these benchmarks often produce evaluation results that exhibit a substantial discrepancy from real-world safety performance. In this paper, we present BeSafe-Bench (BSB), a comprehensive benchmarking framework for evaluating the behavioral safety risks of situated agents within functional environments. These environments provide executable tools, interfaces, and feedback with real functional effects, rather than relying on purely textual LLM-based simulations. As illustrated in Figure 1 , we categorize situated agents into four primary domains: Web, Mobile, Embodied VLM (high-level planning), and Embodied VLA (low-level manipulation). For each domain, we leverage corresponding realistic environment simulators to establish interactive platforms. Within these functional environments, we construct safety-critical tasks based on executable primitive actions, together with predefined safety risk categories and contextual factors. During evaluation, we record both environment states and agent trajectories, and employ a hybrid evaluation framework that integrates rule-based systems with LLM-based reasoning to simultaneously assess task completion and safety coverage. Each task in BSB consists of five essential components: a task execution environment, user instructions, a safety risk type, a task completion evaluator, and a safety evaluator. Furthermore, tasks and environments are strictly isolated from one another, enabling high-fidelity simulations of real-world agent interaction patterns while mitigating potential real-world consequences. BSB provides 1312 executable tasks across diverse scenarios, and a detailed comparison with existing agent safety evaluation benchmarks is presented in Table 1 . Utilizing BSB, we evaluated the performance of 13 popular LLM-driven agents, encompassing both general-purpose and specialized models. Due to variations in task complexity and available tools, evaluation results exhibit significant differentiation across environments. Excluding the Mobile environment, the safety coverage of agents in other domains remains below 60%. Furthermore, fewer than 40% of agents are capable of successfully completing tasks while maintaining safety throughout their execution. Notably, in up to 41% of cases, agents completed the assigned tasks but simultaneously engaged in unsafe behaviors. Our further analysis reveals that agents exhibit insufficient protection of sensitive information and harmful content, while also struggling to maintain safety awareness within complex, sequential task scenarios. In summary, the main contributions of this work are as follows: • We propose BeSafe-Bench, a new and comprehensive benchmark focusing on agent behavioral safety within functional environments, spanning four distinct agent domains (Web, Mobile, Embodied VLM, and Embodied VLA) and covering four