From Words to Safety: Language-Conditioned Safety Filtering for Robot Navigation

Zeyuan Feng, Haimingyue Zhang, Somil Bansal · 2025

Read From Words to Safety: Language-Conditioned Safety Filtering for Robot Navigation by Zeyuan Feng, Haimingyue Zhang, Somil Bansal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05889.txt.

Introduction

Zeyuan Feng, Haimingyue Zhang, and Somil Bansal let people state safety rules for a robot in plain words, and have the robot obey them. As robots enter human spaces, they must follow instructions and respect limits. Older work mapped language to rewards, or covered only narrow limits like avoiding obstacles.

Their framework has three parts. A large language model turns free-form instructions into structured safety rules. A perception module tracks objects in 3D so the rules attach to real things. A predictive controller then filters the robot's motion in real time, enforcing both the worded rules and plain geometry. Tests in simulation and on hardware showed the system held diverse rules across many scenes.

Catalog record

CREATORZeyuan Feng, Haimingyue Zhang, Somil Bansal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05889

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Words to Safety: Language-Conditioned Safety Filtering for Robot Navigation Zeyuan Feng, Haimingyue Zhang, Somil Bansal Abstract As robots become increasingly integrated into open-world, human-centered environments, their ability to interpret natural language instructions and adhere to safety constraints is critical for effective and trustworthy interaction. Existing approaches often focus on mapping language to reward functions instead of safety specifications or address only narrow constraint classes (e.g., obstacle avoidance), limiting their robustness and applicability. We propose a modular framework for language-conditioned safety in robot navigation. Our framework is composed of three core components: (1) a large language model (LLM)-based module that translates free-form instructions into structured safety specifications, (2) a perception module that grounds these specifications by maintaining object-level 3D representations of the environment, and (3) a model predictive control (MPC)-based safety filter that enforces both semantic and geometric constraints in real time. We evaluate the effectiveness of the proposed framework through both simulation studies and hardware experiments, demonstrating that it robustly interprets and enforces diverse language-specified constraints across a wide range of environments and scenarios. I Introduction Robots are increasingly deployed in human-centered environments such as homes, hospitals, and workplaces, where safety is fundamental for trust and adoption. Existing approaches predominantly address static, predefined safety constraints, most notably collision avoidance. Through methods such as Model Predictive Control (MPC) [ 20 , 4 ] , Control Barrier Functions (CBF) [ 2 , 22 ] , and reachability analysis [ 5 , 12 ] , classical techniques have proven effective at filtering unsafe actions. However, these approaches presuppose fixed safety specifications, limiting their ability to adapt to changing contexts. In contrast, failures in human–robot interaction are often dynamic and semantic : they arise from context- and preference-dependent constraints that evolve with the situation and the user. We refer to this broader requirement as semantic safety – the ability of robots to not only avoid collisions, but also anticipate and adapt to context-specific safety expectations expressed by humans around them. For example, a nurse may instruct a medical delivery robot to “ be quiet around the patient’s bed ” when there is a patient who should not be disturbed, or a homeowner may want a robot vacuum to “ not go under a standing desk ” to avoid damage. Such constraints are difficult to anticipate and encode a priori , yet are naturally expressed in language by non-technical users. Enabling robots to accept, ground, and enforce such natural language constraints at runtime is therefore a key missing capability. Recent progress in vision–language models (VLMs) has primarily targeted high-level decision making, e.g., specifying tasks and goals in natural language or shaping reward functions. In contrast, work on vision-language-conditioned safety remains underexplored. Prior efforts have been limited either to policy steering or to handling a narrow class of language-specified constraints. Consequently, the central challenges of language-based safety remain open, i.e., (1) grounding vague or ambiguous instructions in perception, (2) generalizing across diverse instructions and constraint types, (3) maintaining interpretable, auditable specifications, and (4) enforcing them in real time during execution. To address these challenges, we introduce a modular framework for language-conditioned safety (Fig. 1 ). Our system integrates three components: (1) a language module that parses free-form instructions into safety configuration files using predefined constraint templates, leading to structured safety constraint specifications; (2) a perception module that persistently grounds referenced objects and regions in 3D by combining open-vocabulary and panoptic segmentation; and (3) a sampling-based safety filter that can enforce novel semantic and geometric constraints online while minimally deviating from nominal control. Together, these components enable robots to incorporate runtime language instructions into their low-level safety guarantees. We validate our framework extensively in both simulation and hardware experiments, demonstrating robustness across diverse environments and instruction types. Ultimately, our framework leverages VLMs as “open-world” runtime monitors, where natural language provides a flexible interface for specifying evolving safety requirements. II Related Work II-A Large Language Models as Generalist Controllers Recent work has explored using LLMs as generalist reasoning engines that translate natural language into low-level plans or actions. Hierarchical systems such as SayCan [ 1 ] and PaLM-E [ 8 ] couple LLM reasoning with perception and planning modules, while RT-2 [ 31 ] and OpenVLA [ 13 ] extend this paradigm to vision-language-action policies trained end-to-end. These systems highlight the zero-shot generalization potential of LLMs, but their focus is primarily on task execution rather than enforcing explicit safety guarantees. Fig. 1: Framework Overview. Language instructions are parsed by an LLM into safety configurations, grounded via the perception module, and enforced by a sampling-based safety filter. II-B Language for Constraints vs. Rewards Most language-conditioned robotics methods exploit LLM reasoning for reward design and shaping [ 27 , 26 , 9 , 28 , 18 ] , especially in reinforcement learning and navigation. While effective for guiding behaviors, these approaches typically treat safety as a soft cost in the optimization problem (as opposed to a strict requirement), or omit it altogether. In contrast, only a handful of works have attempted to derive explicit safety constraints from language. For instance, [ 25 ] aligns language with learned world models for safe policy steering, but requires heavy manual engineering and remains domain-specific. [ 17 ] integrates VLMs with Hamilton–Jacobi (HJ) reachability for semantic safety filtering, though it addresses only obstacle-avoidance-style constraints. [ 6 ] uses VLMs to infer relevant semantic constraints for a robot manipulator in a given scene, and enforces them with CBF-based filters. Our work complements these efforts by targeting semantic safety for navigation in dynamic, unstructured environments, scaling to diverse constraint types, and supporting runtime template-based specification parsing coupled with safety filtering. II-C Grounding Language in Perception Mapping instructions such as “ be quiet around the bed ” to actionable robot constraints requires robust grounding in perception. Two primary approaches have been explored for providing 3D representation of the target objects: (i) semantic SLAM algorithms that construct a full 3D semantic map [ 15 , 14 , 30 ] or scene graph [ 10 ] of the environment, and (ii) open-vocabulary segmentation models [ 19 , 11 , 29 ] combined with camera projections, robot odometry, and point cloud fusion techniques [ 21 ] . While semantic SLAM provides detailed global representations, it is computationally expensive and prone to odometry drift in cluttered environments. For navigation, open-vocabulary segmentation with point cloud integration is often more practical. Our perception module adopts this latter approach, fusing CLIP-based and panoptic segmentation to balance semantic coverage with boundary accuracy. II-D Safety Filtering Safety filtering techniques modify a robot’s nominal control in real time to ensure satisfaction of safety constraints. Approaches based on HJ reachability and CBFs [ 22 , 3 , 5 ] provide formal guarantees for runtime shielding, but require precomputed safety value functions or fixed constraint sets, making them difficult to extend to dynamic, language-specified constraints. MPC-based safety filters, by contrast, offer greater flexibility through online optimization, but their computational demands limit scalability in cluttered or dynamic settings. To balance flexibility and efficiency, we design a predictive safety filter inspired by Model Predictive Path Integral (MPPI) control [ 24 , 23 ] , which leverages parallelizable, sampling-based trajectory optimization to reduce computation while retaining the expressiveness of MPC. III

Read the complete text