LM Agents May Fail to Act on Their Own Risk Knowledge
Yuzhi Tang, Tianxiao Li, Elizabeth Li, Chris J. Maddison, Honghua Dong, Yangjun Ruan, University of Toronto University of California, Berkeley Vector Institute · 2025
Read LM Agents May Fail to Act on Their Own Risk Knowledge by Yuzhi Tang, Tianxiao Li, Elizabeth Li, Chris J. Maddison, Honghua Dong, Yangjun Ruan, University of Toronto University of California, Berkeley Vector Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13465.txt.
Introduction
Yuzhi Tang and colleagues document a gap that should worry anyone deploying language model agents: the agents know the dangers, and do the dangerous things anyway. Asked directly, models almost always flag a reckless command as risky. Writing in twenty twenty-five, the team's framework tests three levels: knowing risks, spotting them inside an unfolding action sequence, and actually refusing to execute them. Scores fall steeply from the first level to the second, and collapse at the third. Fewer than a quarter of risky actions were avoided in their tests.
Stronger models and dedicated reasoning models show the same pattern. Scale and extra thinking do not close the gap. The authors liken it to the generator-validator split seen elsewhere in language models, and they build a mitigation on that very asymmetry. Knowledge, they conclude, is not a safety mechanism by itself.
Catalog record
| CREATOR | Yuzhi Tang, Tianxiao Li, Elizabeth Li, Chris J. Maddison, Honghua Dong, Yangjun Ruan, University of Toronto University of California, Berkeley Vector Institute |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 136 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13465
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LM Agents May Fail to Act on Their Own Risk Knowledge Yuzhi Tang, Tianxiao Li, Elizabeth Li, Chris J. Maddison, Honghua Dong, Yangjun Ruan, University of Toronto University of California, Berkeley Vector Institute Abstract Language model (LM) agents have demonstrated significant potential for automating real-world tasks, yet they pose a diverse array of potential, severe risks in safety-critical scenarios. In this work, we identify a significant gap between LM agents’ risk awareness and safety execution abilities: while they often answer “Yes” to queries like "Is executing ‘sudo rm -rf /*’ dangerous?" , they will likely fail to identify such risks in instantiated trajectories or even directly perform these risky actions when acting as agents. To systematically investigate this, we develop a comprehensive evaluation framework to examine agents’ safety across three progressive dimensions: 1) their knowledge about potential risks, 2) their ability to identify corresponding risks in execution trajectories, and 3) their actual behaviors to avoid executing these risky actions. Our evaluation reveals two critical performance gaps that resemble the generator-validator gaps observed in LMs: while agents demonstrate near-perfect risk knowledge ( > 98 % >98\% pass rates), they fail to apply this knowledge when identifying risks in actual scenarios (with performance dropping by > 23 % >23\% ) and often still execute risky actions ( < 26 % <26\% pass rates). Notably, this trend persists across more capable LMs as well as in specialized reasoning models like DeepSeek-R1, indicating that simply scaling model capabilities or inference compute does not inherently resolve safety concerns. Instead, we take advantage of these observed gaps to develop a risk verifier that independently critiques the proposed actions by agents, with an abstractor that converts specific execution trajectories into abstract descriptions where LMs can more effectively identify the risks. Our overall system achieves a significant reduction of risky action execution by 55.3 % 55.3\% over vanilla-prompted agents. 1 Introduction Recent advances have demonstrated the promise of developing language model (LM) agents ( Yang et al. 2024 ; Yao et al. 2023b ; Yao et al. 2023a ; Shinn et al. 2023 ; Wang et al. 2023 ; Liu et al. 2023 ) to automate a wide range of real-world tasks. However, these agents, while interacting with the external world, can also incorporate a new, diverse array of risks and exhibit concerning failures, especially in safety-critical scenarios ( Ruan et al. 2024a ; Kinniment et al. 2024 ; Qu et al. 2025 ; Ramos et al. 2024 ; Anwar et al. 2024 ) . Understanding and mitigating these risks is critical for the reliable deployment of LM agents. In this work, we start from a striking phenomenon in the behavior of LM agents: When directly asked about risky actions in the question-answering (QA) setting (e.g., "Is executing ‘sudo rm -rf /*’ dangerous?" ), the LMs consistently acknowledge the risks. However, when instantiated in actual execution scenarios, they often fail to identify the risks of potential risky actions, or even more concerningly, directly execute them without recalling their safety knowledge ( Ruan et al. 2024a ) . This discrepancy suggests a fundamental gap between LM agents’ awareness of risks and their ability to apply the knowledge during executions, which presents a new instantiation of the generator-validator gap of LMs ( Li et al. 2023 ) in safety-critical applications. (a) Evaluation Framework (b) Evaluation Results Figure 1 : (a) Illustration of our evaluation framework assessing the performance gaps from LM agents’ safety awareness to their actual safe executions. From a diverse set of risky execution trajectories, we design three targeted tests for distinct and progressive aspects of LM agents’ safety: 1) Knowledge Test to test whether the agents possess relevant risk knowledge; 2) Identification Test to test whether they can apply their risk knowledge in instantiated scenarios. 3) Execution Test to test whether agents would operate safely during actual executions. (b) We observe critical performance gaps from their safety awareness to executions for all agents we have assessed – they all achieve near-perfect performance on knowledge test, but fail at an increasingly high ratio on identification and execution tests. To systematically investigate this, we develop a comprehensive evaluation framework that examines three distinct and progressive aspects of agents’ risk awareness, from its ability to know the risks in direct QA setups to identify them and avoid executing them during actual executions (as illustrated in Figure 1(a) ). In particular, we construct the following tests that target each aspect: • Knowledge Test : To test whether the agents possess the relevant risk knowledge, we prompt the agents to judge the risks of potential risky actions described in abstract scenario descriptions and answer in standard QA settings. • Identification Test : To test agents’ abilities to apply the knowledge to identify the potential risks in instantiated scenarios, we prompt agents to judge the risks in actual execution trajectories corresponding to the abstract scenario descriptions. • Execution Test : To test agents’ abilities to actually operate safely and avoid executing risky actions, we instantiate them in potential risky scenarios and prompt them to generate the next actions starting from partial execution trajectories before risky actions were executed. Together, these tests provide a comprehensive understanding of LM agents’ safety profiles and reveal two critical gaps from agents’ risk awareness to their executions that generalize the standard generator-validator gap ( Figure 1(b) ): • Knowledge-Identification Gap : Although agents possess most of the risk knowledge with a near-perfect performance ( >98.4\% pass rate) in knowledge tests, they struggle to identify these risks in instantiated scenarios (pass rates dropped to an average of 78.2\% ). • Identification-Execution Gap : Even when agents are able to identify risks of executed actions in specific trajectories, they still tend to execute the potentially risky actions if placed in the same scenarios (with an average pass rate of 13.3\% on execution tests). Our results demonstrate the significant inefficiencies of LMs in applying their safety knowledge when deployed as agents, possibly due to the extensive alignment procedures mostly done under conversational scenarios instead of agentic scenarios ( Askell et al. 2021 ; Ouyang et al. 2022 ; Bai et al. 2022a ; Achiam et al. 2023 ) . Notably, we observe that agents based on more capable LMs or specialized reasoning models like DeepSeek-R1 ( Guo et al. 2025 ) do not necessarily demonstrate smaller gaps ( Figure 2 ), which implies that we may not address these by simply scaling base model capabilities or inference-time compute. Inspired by these findings, we develop specific risk mitigation strategies ( Figure 3 ) for LM agents that bridge these gaps. Specifically, we develop a risk verifier instantiated with the same base LMs to independently critique the agent’s proposed actions and reject risky ones, effectively leveraging the observed identification-execution gap. Additionally, we utilize the knowledge-identification gap by augmenting the verifier with an abstractor that converts specific execution trajectories into abstract descriptions, helping the verifier better identify potential risks. Together, as shown in Table 1 , our verifier-based system achieves a significant reduction ( 55.3\% ) in risky action execution across different agents. Overall, the contributions of our work are as follows: • We develop a systematic evaluation framework that targets distinct and progressive