Towards General Language-Conditioned Latent Safety Filters
Ihab Tabbara, Yuxuan Yang, Hussein Sibai · 2026
Read Towards General Language-Conditioned Latent Safety Filters by Ihab Tabbara, Yuxuan Yang, Hussein Sibai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00315.txt.
Introduction
Tabbara, Yang, and Sibai test language-conditioned safety filters for robot policies. Vision-language-action models let one policy run many tasks from natural language. Safe deployment must adapt to safety demands as well as tasks. Those demands vary by user, setting, and application. Existing filters are tied to set constraints and must be rebuilt when demands change.
Their filter conditions a Hamilton-Jacobi safety actor and critic on language-stated constraints. Tests span pick-and-place, table-wiping, and block-stacking tasks. All run in a vision-based setting. The filters cut constraint violations in their experiments. Partial transfer to unseen constraint cases within known families also appeared. General language-conditioned safety is the direction they probe.
Catalog record
| CREATOR | Ihab Tabbara, Yuxuan Yang, Hussein Sibai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00315
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards General Language-Conditioned Latent Safety Filters Ihab Tabbara, Yuxuan Yang, Hussein Sibai Abstract Robot policies are becoming increasingly general, with vision–language–action (VLA) models enabling a single policy to execute diverse tasks specified in natural language. Safe deployment, however, requires adapting not only to new tasks but also to varying safety requirements across users, environments, and applications. Existing safety filters remain largely constraint-specific, and thus must be redesigned or relearned when safety requirements change. In this paper, we investigate language-conditioned safety filtering, in which a Hamilton–Jacobi safety actor and critic are conditioned on language-specified constraints. We evaluate this formulation across pick-and-place, table-wiping, and block-stacking tasks in the vision-based setting, examining its ability to enforce language-specified constraints and transfer to unseen constraint instances within the evaluated constraint families. Our experiments provide evidence that language-conditioned safety filters reduce constraint violations and exhibit partial transfer to unseen constraint instances. 1 Department of Computer Science and Engineering, Washington University in St. Louis, MO 63130, USA {i.k.tabbara, y.yuxuan, sibai}@wustl.edu Introduction Robot learning is increasingly shifting from training task-specific controllers towards training generalist policies capable of executing multiple tasks in diverse settings. Recent vision–language–action (VLA) models advance this trend by conditioning robot behavior on visual observations and natural-language instructions, allowing a single policy to perform a broad range of manipulation ( Brohan et al. 2023 ; Open X-Embodiment Collaboration et al. 2024 ; Octo Model Team et al. 2024 ; Black et al. 2024 ; Intelligence et al. 2025 ; Kim et al. 2024 ; Sapkota et al. 2025 ; Zitkovich et al. 2023 ) and navigation tasks ( Wang et al. 2025 ; Hirose et al. 2026 ) . As robot policies become increasingly general, three complementary approaches can support safe deployment. First, policies can learn to execute tasks safely even without explicit safety instructions, provided their training data includes risky situations and demonstrations of appropriate responses. Second, policies can be trained to follow explicit safety constraints expressed in language, which requires examples pairing such instructions with corresponding safe behavior. Third, a separate runtime safety filter can enforce deployment-specific constraints independently from the nominal policy. This third approach remains important because safety requirements may vary across users, environments, and applications, even for the same task. Current VLA policies are not consistently reliable with either of the first two approaches. Since they are typically trained through behavioral cloning, they may not learn implicit safety when risky scenarios are absent from the demonstrations, and they may ignore unfamiliar safety prompts that were not represented during training. As shown in the Appendix, explicit language-specified constraints can have little effect on generated actions. These limitations motivate investigating language-conditioned safety filtering as a complementary runtime layer that accepts natural-language safety constraints and intervenes when the nominal policy proposes an action that may violate them. Safety filters provide an attractive framework for this setting because they monitor a nominal policy’s proposed actions online and intervene only when those actions may lead to failure. Classical safety filters, including ones based on Hamilton–Jacobi (HJ) reachability ( Bansal et al. 2017b ; Fisac et al. 2019 ) and control barrier functions (CBFs) ( Ames et al. 2017b ) , guarantee safety without interrupting the nominal policy unless necessary. However, safety filters have not yet undergone the transition from task-specific to general-purpose systems. They assume a fixed notion of safety, namely forward invariance, that depends on the system dynamics together with a formally specified failure set of states ( Bansal et al. 2017b ; Fisac et al. 2019 ) . Recently proposed latent safety filters generalize the classical ones to vision-based control settings where the policies and the failure sets are functions of visual observations ( Nakamura et al. 2025 ; Tabbara et al. 2025 ) , but they remain tied to specific safety specifications. Consequently, changing the safety specification, i.e., the failure set of states, typically requires redesigning or retraining the safety filter, limiting the practical deployment of increasingly general robot policies. This raises a question: if robot policies are becoming general-purpose, can safety filters also adapt to deployment-time safety requirements expressed through language? In other words, to what extent can a single learned safety filter represent multiple language-specified constraints, generalize beyond the constraints observed during training, and act as a function of learned vision–language representations? In this work, we investigate language-conditioned safety filtering . We formulate safety filtering as learning a constraint-conditioned Hamilton–Jacobi reachability actor and critic that condition on both the robot’s visual observation and a user-specified natural-language safety constraint. Given an observation o_{t} , a natural-language constraint c , and a nominal action a_{t}^{\mathrm{nom}} , the learned critic estimates whether the proposed action satisfies the specified constraint, and if not, the filter switches from the nominal to the learned actor policy that is trained to maximize safety. Our contributions can be summarized as follows: • We formulate language-conditioned safety filtering , extending Hamilton–Jacobi reachability-based safety filtering to condition on language-specified constraints. • We evaluate language-conditioned safety filters across pick-and-place, table wiping, and block stacking tasks, studying language-conditioned constraint satisfaction, out-of-distribution generalization, and the trade-off between safety, task success, and intervention rate. • We compare a language-conditioned safety filter with separate specialized non-language-conditioned ones. • We evaluate the performance of eleven vision–language models as failure functions for robotic manipulation. • Upon publication, we will release a dataset of 5,000 LIBERO-style manipulation trajectories, approximately 90% of which contain safety violations. Related Work Safe control and latent safety filters Control barrier functions and Hamilton–Jacobi (HJ) reachability provide formal tools for filtering a nominal controller’s output to maintain safety ( Ames et al. 2017b ; Ames et al. 2017a ; Bansal et al. 2017b ; Bansal et al. 2017a ) . Classical HJ methods assume low-dimensional states, known dynamics, and a manually specified failure set. Recent neural and latent safety filters relax some of these assumptions by learning safety value functions over high-dimensional observation spaces, and particularly, their learned representation ones ( Nakamura et al. 2025 ; Tabbara et al. 2025 ; Li and Chen 2025 ; Lee and Kim 2025 ) . AnySafe ( Agrawal et al. 2025 ) adapts latent safety filters at runtime using a user-provided constraint image. HJRNO ( Li and Chen 2025 ) learns a neural operator for Hamilton–Jacobi reachability, enabling fast generalization across obstacle shapes, dynamics, and problem parameters. ( Santos et al. 2025 ) and ( Feng et al. 2025 ) both study language-conditioned safety for robot navigation. ( Santos et al. 2025 ) tackle collision avoidance for planar robot navigation by using a vision-language model to update an environment-specific failure set and re-run an online Hamilton–Jacobi safety filter ( Mitchell and others 2007 ) . ( Feng et al. 2025 ) propose a modular pipeline that translates free-form instructions into structured safety specifications with an LLM, grounds them in object-level 3D perception, and enforces semantic and geometric constraints with an MPC-based safety filter in real time. In contrast, our method directly conditions a safety filter on language constraints in an embedding space, learns a single filter offline, and is intended as a general framework beyond navigation, with the learned filter adapting at runtime to the current instruction rather than requiring online replanning or retraining a separate HJ filter