Your Learned Constraint is Secretly a Backward Reachable Tube
Mohamad Qadri, Gokul Swamy, Jonathan Francis, Michael Kaess, Andrea Bajcsy · 2025
Read Your Learned Constraint is Secretly a Backward Reachable Tube by Mohamad Qadri, Gokul Swamy, Jonathan Francis, Michael Kaess, Andrea Bajcsy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15618.txt.
Introduction
Mohamad Qadri and four co-authors show that learned constraints are backward reachable tubes in two thousand twenty-five. Inverse constraint learning infers rules from safe example acts. Learned rules should guide safe plans for new tasks and new dynamics. The team asks what object the learning truly recovers.
In theory and in tests, it recovers where failure is doomed, not where it has hit. Safe control calls that set a backward reachable tube. That tube rests on the dynamics of the data source system. The twist shapes how well rules transfer and how fast plans can be found. Both gains and limits follow.
Catalog record
| CREATOR | Mohamad Qadri, Gokul Swamy, Jonathan Francis, Michael Kaess, Andrea Bajcsy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15618
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Your Learned Constraint is Secretly a Backward Reachable Tube Mohamad Qadri, Gokul Swamy, Jonathan Francis, Michael Kaess, Andrea Bajcsy Abstract Inverse Constraint Learning (ICL) is the problem of inferring constraints from safe (i.e., constraint-satisfying) demonstrations. The hope is that these inferred constraints can then be used downstream to search for safe policies for new tasks and, potentially, under different dynamics. Our paper explores the question of what mathematical entity ICL recovers. Somewhat surprisingly, we show that both in theory and in practice, ICL recovers the set of states where failure is inevitable , rather than the set of states where failure has already happened. In the language of safe control, this means we recover a backwards reachable tube (BRT) rather than a failure set . In contrast to the failure set, the BRT depends on the dynamics of the data collection system. We discuss the implications of the dynamics-conditionedness of the recovered constraint on both the sample-efficiency of policy search and the transferability of learned constraints. 1 Introduction Constraints are fundamental for safe robot decision-making ( Stooke et al. 2020 ; Qadri et al. 2022 ; Howell et al. 2022 ) . However, manually specifying safety constraints can be challenging for complex problems, paralleling the reward design problem in reinforcement learning ( Hadfield-Menell et al. 2017 ) . For example, consider the example of an off-road vehicle that needs to traverse unknown terrains. Successful completion of this task requires satisfying constraints such as “avoid terrains that, when traversed, will cause the vehicle to flip over” which can be difficult to specify precisely via a hand-designed function. Hence, there has been a growing interest in applying techniques analogous to Inverse Reinforcement Learning (IRL) — where the goal is to learn hard-to-specify reward functions — to learning constraints ( Liu et al. 2024 ) . This is called Inverse Constraint Learning (ICL): given safe expert robot trajectories and a nominal reward function, we aim to extract the implicit constraints that the expert demonstrator is satisfying. Intuitively, these constraints forbid highly rewarding behavior that the expert nevertheless chose not to take ( Kim et al. 2023 ) . However, as we now explore, the question of what object we’re actually inferring in ICL has a nuanced answer that has several implications for downstream usage of the inferred constraint. Figure 1 : In this work we show theoretically and empirically that inverse constraint learning (ICL) recovers a backwards reachable tube rather than the true failure set as commonly assumed in the literature. (a) ICL models the expert demonstrator as optimizing reward functions (potentially for different tasks) while satisfying a shared true constraint c^{\star} (e.g. don’t hit a tree) with an associated unsafe set C^{\star} . (b) ICL takes as input expert demonstrations and the reward functions r_{1}\ldots r_{K} and aims to recover the shared true constraint c^{\star} . (c) ICL infers a constraint \hat{c} and it’s associated unsafe set \hat{C} , from the demonstrations. However, we show that \hat{c} encodes a different object than the true failure set. In particular, \hat{c} encodes the the backward reachable tube of the true failure set under system dynamics f(x,a,d,t) : the set of states from which violating c^{\star} is inevitable (e.g. positions / velocities for which we can’t avoid crashing). Consider Fig. 1 a , in which an expert (e.g., a human driver) drives a car through a forest from a starting position to an end goal, without colliding with any trees. Assume that the expert has an internal representation of the true constraint, c^{\star} , which they use during their planning process to generate demonstrations (Fig. 1 b). Here, c^{\star} encodes the location of the trees or, equivalently, the failure set : the set of states which encode having already failed the task. Given expert demonstrations that satisfy c^{\star} , we can run an ICL algorithm to obtain an inferred constraint, \hat{c} . Our key question is whether the learner actually recovers the constraint the expert used (i.e. is \hat{c}=c^{\star} ?). In other words, does \hat{c} encode the true failure set (e.g., where the trees are)? As we prove below, the answer to this question is, surprisingly enough, often “ no ." This motivates the key question of our work: When learning constraints from demonstrations, what mathematical entity are we actually learning? We show theoretically and empirically that, rather than inferring the set of states where the robot has already failed at the task, ICL instead infers where, under the expert’s dynamics, failure is inevitable . For example, rather than inferring the location of the tree, ICL would infer the larger set of states for which the expert will find that avoiding the tree is impossible (illustrated in Fig. 1 c). More formally, we prove that ICL is actually approximating a dynamics-conditioned backward reachable tube (BRT), rather than the the dynamics-independent failure set. The observation that we are recovering dynamics-conditioned BRTs rather than failure sets has two important implications. On one hand, it means that we can add ICL algorithms to the set of computational tools available to us for computing BRTs, given a dataset of safe demonstrations. On the other hand, it means that one cannot hope to easily transfer the constraints learned via ICL between different dynamics naively. We begin by exploring the relationship between ICL and BRTs before discussing implications. 2 Problem Setup Dynamical System Model. We consider continuous-time dynamical systems described by the ordinary differential equation \dot{s}=f(s,a,d,t) , where t is time, s\in\mathcal{S} is the state, a\in\mathcal{A} is the control input, and d\in\mathcal{D} is the disturbance that accounts for unmodeled dynamics (e.g., wind or friction). Environment and Task Definition. A task k is defined as a specific objective that our robot needs to complete. For example, in Fig. 1 , a mobile robot might be tasked with reaching a specific target pose from a starting position while avoiding environmental obstacles. In this work, we assume this task objective to be implicitly defined using a reward function r_{k}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} . Let K be a set of tasks \{k\} with a shared implicit constraint c^{\star} which can be a function of the state and action or of the state only—in other words, c^{\star}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} or c^{\star}:\mathcal{S}\rightarrow\mathbb{R} . While the task-specific reward r_{k} assigns a high reward when the robot successfully completes the objective, the constraint c^{\star} assigns a high cost to state-action pairs (or states) that violate the true shared constraints. In other words, c^{\star}=\infty if a state-action pair (or state) is unsafe and c^{\star}=-\infty otherwise. For example, in Fig. 1 a, the set C^{\star}=\{s\in\mathcal{S}\;|\;1[c^{\star}(s)=\infty]\} represents the true location of the obstacle (i.e., the tree). Furthermore, let’s define \hat{c}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} or \hat{c}:\mathcal{S}\rightarrow\mathbb{R} as the constraint learned through ICL. Similarly, \hat{c}=\infty if a state-action pair (or state) is deemed unsafe by the algorithm and \hat{c}=-\infty otherwise. For example, in Fig. 1 c, \hat{C}=\{s\in\mathcal{S}\;|\;1[\hat{c}(s)=\infty]\} represents the inferred set of unsafe states calculated by ICL. Safe Demonstration Data from an Expert. In the ICL setting, an expert provides our algorithm with safe demonstrations from K different tasks, each satisfying a shared constraint. Take, for instance, a mobile robot operating in a single environment as shown Fig. 1 . Each task k might involve navigating according to a different set of