Safety Assessment in Reinforcement Learning via Model Predictive Control
Jeff Pflueger, Michael Everett · 2025
Read Safety Assessment in Reinforcement Learning via Model Predictive Control by Jeff Pflueger, Michael Everett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20955.txt.
Introduction
Jeff Pflueger and one co-author approximate safety feedback for mobile robots without a safety oracle, using model predictive control inside a simulator. Safe control methods usually assume direct safety checks or hand labeled safety data, both costly and often inaccurate.
Modern simulators capture complex interactions and carry implicit safety limits that are hard to write down by hand. By rolling out candidate actions in simulation and scoring the outcomes, the method builds a stand in safety function. Robots can then screen actions before acting in the real world, a useful route when explicit constraints resist modeling.
Catalog record
| CREATOR | Jeff Pflueger, Michael Everett |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safety Assessment in Reinforcement Learning via Model Predictive Control Jeff Pflueger, Michael Everett Abstract Model-free reinforcement learning approaches are promising for control but typically lack formal safety guarantees. Existing methods to shield or otherwise provide these guarantees often rely on detailed knowledge of the safety specifications. Instead, this work’s insight is that many difficult-to-specify safety issues are best characterized by invariance. Accordingly, we propose to leverage reversibility as a method for preventing these safety issues throughout the training process. Our method uses model-predictive path integral control to check the safety of an action proposed by a learned policy throughout training. A key advantage of this approach is that it only requires the ability to query the black-box dynamics, not explicit knowledge of the dynamics or safety constraints. Experimental results demonstrate that the proposed algorithm successfully aborts before all unsafe actions, while still achieving comparable training progress to a baseline PPO approach that is allowed to violate safety. I Introduction Model-free Reinforcement Learning (RL) [ 1 ] is a powerful approach for learning a control policy without explicit knowledge of safety constraints or system dynamics. However, training RL policies in the real world can lead to safety issues that are both time-consuming and costly to fix. To address these issues, many current algorithms model the process as a Constrained Markov Decision Process (CMDP) [ 2 ] , which uses a set of constraints on the state and action space to model safety. However, defining safety constraints presents major challenges, especially in complex environments. Any heuristic constraints can hinder an agent’s exploration if they don’t appropriately model the true constraints of the environment. For example, a constraint to maintain a distance from other vehicles may prevent an agent from learning a riskier maneuver like overtaking another vehicle while racing. On the contrary, if these heuristic constraints are too liberal, they can cause potentially catastrophic incidents, such as collisions. These incidents can damage the robot and require both time and money to repair during the training process. Current methods of ensuring the safety of a learned policy can provide guarantees but typically require knowledge of the safety specifications. For example, discriminator methods [ 3 , 4 , 5 ] assume access to a safety oracle during the training process and are prone to constraint violations as the discriminators learn. Gaussian process methods [ 6 , 7 ] require strong assumptions about the underlying safety function (e.g., smoothness) and must be able to observe its true value. Shielding based methods [ 8 , 9 , 10 ] enforce that executed actions are safe before they are executed. They do this through measuring the safety of an action before it is executed, and modifying it to fit some safety definition. These assumptions limit the performance of the learned policy or slow down the training process by requiring hand-labeling of safety data. Therefore, there is a need for learning algorithms that do not require explicitly defined constraints. But how can we define safety without direct knowledge of the constraints? Fig. 1 : Illustration of RL-SAVMPC. The agent is attempting to overtake the vehicle in red. First, RL-SAVMPC projects an action forward. It then ensures there is a set of actions back to the original state before execution. Consider a state space divided into a safe and unsafe set, \mathcal{X}_{\text{safe}} and \mathcal{X}_{\text{unsafe}} . Our key assumption is that \mathcal{X}_{\text{unsafe}} is invariant, which captures the class of safety issues that cannot be autonomously “un-done” (e.g., damaging the robot or environment, rolling over such that the wheels/legs cannot contact the ground, running out of fuel). This assumption will enable the use of reversibility as a proxy for safety [ 11 ] . A reversibility between two states means that sequences of actions exist that transition an agent between those states, in both directions. Since \mathcal{X}_{\text{unsafe}} is invariant, establishing a reversibility also establishes that both states belong to the same set. Further, establishing a chain of reversibilities to a known safe set also establishes the safety of every state in the reversibility chain. In this paper we propose R einforcement L earning with S afety A ssessment V ia M odel- P redictive C ontrol (RL-SAVMPC), a safe RL algorithm that uses reversibility as a proxy for unknown safety constraints. RL-SAVMPC employs Model Predictive Control (MPC [ 12 ] ) to plan through a simulator back to a previous state in the trajectory. In doing this, it automatically verifies that a plan exists that reaches the safe set. If it cannot verify the safety of a planned action, RL-SAVMPC aborts the trajectory in favor of resetting the training environment. It does this while maintaining zero constraint violations in the underlying training environment, all without requiring access to the latent constraint space. A visualization of this process is depicted in Fig. 1 . Our primary contributions include: • We present RL with Safety Assessment Via Model-Predictive Control (RL-SAVMPC), a safe RL algorithm that uses MPC and a simulator to ensure safety by planning back to a previous state in a trajectory. • We analyze this method on two training environments for average reward, as well as total constraint violations and trajectory aborts. RL-SAVMPC is compared to a baseline RL algorithm and a shielding method with perfect constraint knowledge. We show that in some cases, RL-SAVMPC can outperform a resampling shield with perfect knowledge of state constraints. II Problem Statement We model the safe decision making problem as a continuous Constrained Markov Decision Process (CMDP). Our model is a tuple ( \mathcal{X} , \mathcal{U} , f_{\text{dyn}} , R , \gamma , g ). The state space \mathcal{X}\subseteq\mathbb{R}^{n} is divided into two subsets: \mathcal{X}_{\text{safe}} and \mathcal{X}_{\text{unsafe}} , i.e., \mathcal{X}_{\text{safe}}=\mathcal{X}\setminus\mathcal{X}_{\text{unsafe}} , and \mathcal{X}_{\text{unsafe}} is invariant. \mathcal{U}\subseteq\mathbb{R}^{m} defines an action space, f_{\text{dyn}}:\mathcal{X}\times\mathcal{U}\to\mathcal{X} is the deterministic transition function, R:\mathcal{X}\to\mathbb{R} is the reward function, and g:\mathcal{X}\to\{0,1\} is a safety function. We assume that the dynamics function f_{\text{dyn}} is a “black-box” in that it is queryable but not directly accessible. Our goal is to prevent the RL agent from ever entering \mathcal{X}_{\text{unsafe}} without interfering excessively to hinder the training progress. III Related Work This section summarizes the literature on ensuring safety in RL. Unlike our work, which only assumes black-box access to the dynamics function (i.e., a simulator), most existing work requires more detailed knowledge of the system dynamics or safety constraints. III-A Learned Safety Function Representations One way to determine whether an action is safe to implement or not is to approximate the underlying safety function. For example, the use of Gaussian Processes (GP) to model safety in discrete MDPs was pioneered in the SafeMDP [ 6 ] algorithm. This uses a set of uncertainty bounds provided by the GP to plan to new known-safe locations for exploration. This work also uses the concept of reversibility, enforcing that an agent can return to known safe states from future states that it visits. SNO-MDP [ 7 ] builds on this work with the focus of expanding pessimistic-safe regions for exploitation. While these methods can safely explore MDPs, they are limited to discrete, deterministic MDPs, with Lipschitz assumptions about the underlying safety function. SABRE [ 13 ] takes an active learning approach, seeking to find the best safety function