Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making

Yejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chanyoung Park · 2025

Read Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making by Yejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chanyoung Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19933.txt.

Introduction

Yejin Son and seven co-authors build a test frame for the physical safety of language models that steer robots. Such models now plan moves for embodied agents, yet safety checks lean on crude pass rates in niche scenes. That hides where and why a model fails, which blocks safe use in high-risk places. Their frame, called SAFEL, splits safety skill into two tests that can be read apart.

One test checks if a model refuses unsafe orders, and the other checks if its plans are both safe and workable. The plan test splits further into goal reading, world change modelling, and step order. Their scene set holds 942 cases, from plain bad orders to traps set by context. Across 13 top models, plain bad orders were often refused. Subtle risks in context were missed far more often.

Catalog record

CREATORYejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chanyoung Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making Yejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chanyoung Park Abstract Large Language Models (LLMs) are increasingly used for decision making in embodied agents, yet existing safety evaluations often rely on coarse success rates and domain-specific setups, making it difficult to diagnose why and where these models fail. This obscures our understanding of embodied safety and limits the selective deployment of LLMs in high-risk physical environments. We introduce SAFEL , the framework for systematically evaluating the physical safety of LLMs in embodied decision making. SAFEL assesses two key competencies: (1) rejecting unsafe commands via the Command Refusal Test , and (2) generating safe and executable plans via the Plan Safety Test . Critically, the latter is decomposed into functional modules, goal interpretation , transition modeling , action sequencing , enabling fine-grained diagnosis of safety failures. To support this framework, we introduce EmbodyGuard , a PDDL-grounded benchmark containing 942 LLM-generated scenarios covering both overtly malicious and contextually hazardous instructions. Evaluation across 13 state-of-the-art LLMs reveals that while models often reject clearly unsafe commands, they struggle to anticipate and mitigate subtle, situational risks. Our results highlight critical limitations in current LLMs and provide a foundation for more targeted, modular improvements in safe embodied reasoning. 1 Introduction 2 Overview of EmbodyGuard We present EmbodyGuard , a PDDL-based benchmark designed to evaluate LLMs’ ability to understand physical safety in embodied decision making. The dataset captures a broad range of realistic scenarios where home-assistant robots must generate safe and executable plans in response to user commands. Each scenario consists of a natural language instruction paired with a corresponding PDDL problem and its solution, annotated with safety-relevant risk information. For completeness, we outline the formal structure of a PDDL scenario in Section 2.1 . To construct our dataset, we employ a multi-stage pipeline that integrates LLM-based generation, symbolic verification, and human annotation. Grounded in our definition of physical safety, we organize the resulting scenarios into two subsets: EmbodyGuard mal , which contains explicitly harmful commands, and EmbodyGuard sit , which captures implicit, context-dependent hazards. Candidate scenarios are first generated using GPT-4o as described in Section 2.2 , then verified through symbolic checks in Section 2.3 and expert human review in Section 2.4 . 2.1 PDDL-Based Scenario for EmbodyGuard The Planning Domain Definition Language (PDDL) is a standardized formalism for representing classical planning problems Aeronautiques et al. (1998) . A typical PDDL-based planning problem consists of a domain file and a problem file. The domain file provides an abstract representation of the world’s rules, including a set of predicates that define the state space S , and a set of actions A with their corresponding preconditions and effects (i.e., the transition function f , which models how actions change the environment). The state space S consists of a unary state component S_{u} and a relational state component S_{r} . Each action in A is associated with a set of parameters P , representing the objects involved in the action. Each parameter in P is assigned a specific type , which restricts the applicable actions to only those objects of compatible types. We refer to the original, predefined actions as primitive actions A_{p} , to distinguish them from any additional actions generated for our scenarios new actions A_{n} . The problem file specifies the set of objects used to ground the domain, along with the initial state S_{\text{init}} and goal conditions S_{G} . The example of PDDL is shown in Appendix C . To solve planning problems, planners use efficient search over PDDL representations. We use Fast Forward Helmert (2006) as our planner. The primary objective of our work is to evaluate whether LLMs can accurately convert user-provided natural language commands into PDDL Liu et al. (2023) ; Li et al. (2024a) , while explicitly accounting for physical safety. Rather than simply translating the instructions into a formal format, we assess whether the model can recognize potential physical hazards implied by the given command in its environments and generate a safe plan that avoids them. We then validate the generated PDDL by inputting it into a planner and checking whether a valid and executable plan can be derived. Throughout this process, we assume that the domain rules are predefined. Specifically, we adopt the iGibson domain, which defines 100 household activities that require both fine-grained object interaction (e.g., “opening cabinets”, “grasping utensils”) and agent mobility in realistic virtual home environments. For details on the simulation environment used, see Appendix D . The complete PDDL domain and problem definitions corresponding to these examples are detailed in Appendix F . 2.2 Synthetic Scenario Construction Hazard Taxonomy and Dataset Composition We define Physical Safety as an LLM’s ability to appropriately handle commands that may cause physical harm, whether explicit or subtle, to specific targets. This encompasses both refusal of clearly unsafe instructions and the safe execution of context-dependent commands by understanding risks and their mitigations. Following this definition, we divide the dataset into two subsets. EmbodyGuard mal consists of 541 scenarios designed to assess whether an LLM can reject explicitly malicious commands that aim to cause specific harm to a specific target. In contrast, EmbodyGuard sit includes 402 scenarios that evaluate an LLM’s ability to detect and mitigate implicit hazards, subtle risks that can lead to specific forms of harm or failure by relying on commonsense physical reasoning. To better understand the risk landscape represented in these scenarios, we analyze their underlying safety challenges using a taxonomy of hazard types. These include fire, electrical shock, overheating, slipping, collision, poisoning, entrapment, falling objects, spillage, burns, structural damage, and malfunction. Each hazard is associated with potential targets of harm, such as humans, embodied agents, animals, or property as summarized in the taxonomy in Appendix O . Constructing Diverse PDDL Scenarios We employed GPT-4o to generate an initial pool of 2K scenarios for each EmbodyGuard mal and EmbodyGuard sit . Each scenario was carefully constructed using detailed prompts that explicitly defined realistic task sequences, commonsense constraints, and scenario-specific safety considerations. For instance, prompts ensured scenarios followed logically coherent action sequences (e.g., “opening a refrigerator”, “retrieving food”, and “then heating it in a microwave”) and adhered to physical realism (e.g., objects must be within reach before interaction). To ensure scenario diversity and maintain consistency with the simulation environment, each synthetic scenario was generated by varying the initial seed scenarios derived from the objects and actions defined in the BEHAVIOR benchmark. (See Appendix D .) Prompts for EmbodyGuard mal explicitly required the inclusion of clearly harmful instructions with corresponding explicit risks and hazardous actions. Conversely, prompts for EmbodyGuard sit emphasized realistic household tasks with implicit situational risks can induce specific failure. The scenario generation process involved creating a natural language instruction that clearly stated the task objective and constructing a corresponding PDDL problem specifying the necessary content to accomplish this instruction. Each PDDL problem consisted of essential elements previously outlined, including objects O , their initial states S_{\text{init}} , and goal conditions S_{G} . See Appendix L for prompt details. 2.3 Symbolic PDDL Scenario Validation To ensure scenarios adhere to the PDDL domain definition, we

Read the complete text