SPoRt - Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL

Jacques Cloete, Nikolaus Vertovec, Alessandro Abate · 2025

Read SPoRt - Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL by Jacques Cloete, Nikolaus Vertovec, Alessandro Abate (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06386.txt.

Introduction

Jacques Cloete, Nikolaus Vertovec, and Alessandro Abate present a way to train robot policies with a certified safety bound, in a 2025 paper. Reinforcement learning lets an agent learn by trial and error, but safety-critical uses need proof that harm stays unlikely during both training and use. Their method, called SPoRt, measures how far a new task policy strays from a known safe base policy. That distance, the safe policy ratio, bounds the chance of breaking a safety rule.

The bound also covers rules that unfold over time and robust control problems. A data-driven step sets the bound for the base policy, and a new training method, Projected PPO, keeps the task policy inside the limit the user picks. Experiments show the trade-off between safety and task performance, and check the bound against real violation rates.

Catalog record

CREATORJacques Cloete, Nikolaus Vertovec, Alessandro Abate
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPoRt - Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL Jacques Cloete, Nikolaus Vertovec, Alessandro Abate Abstract To apply reinforcement learning to safety-critical applications, we ought to provide safety guarantees during both policy training and deployment. In this work, we present theoretical results that place a bound on the probability of violating a safety property for a new task-specific policy in a model-free, episodic setting. This bound, based on a maximum policy ratio computed with respect to a ‘safe’ base policy, can also be applied to temporally-extended properties (beyond safety) and to robust control problems. To utilize these results, we introduce SPoRt, which provides a data-driven method for computing this bound for the base policy using the scenario approach, and includes Projected PPO, a new projection-based approach for training the task-specific policy while maintaining a user-specified bound on property violation. SPoRt thus enables users to trade off safety guarantees against task-specific performance. Complementing our theoretical results, we present experimental results demonstrating this trade-off and comparing the theoretical bound to posterior bounds derived from empirical violation rates. 1 Introduction Reinforcement Learning (RL) is an area of machine learning where an agent is trained to interact with its environment to maximize some (cumulative) reward 5 ; Mason and Grijalva (2019) . There has been great interest in applying RL to real-world control problems in fields such as robotics Kober and Peters (2014) ; Hwangbo et al. (2019) ; Singh et al. (2022) , traffic management Chu et al. (2020) ; Vertovec and Margellos (2023) ; Lee et al. (2023) and autonomous driving Isele et al. (2018) ; Ma et al. (2021) ; Li et al. (2022) , to name just a few. Many of these domains typically fall into the realm of “safety-critical” applications, whereby we need to guarantee safety specifications, such as obstacle avoidance. Satisfying safety constraints becomes particularly challenging when we have little to no knowledge of our environment. This problem has been studied in a substantial body of literature, known as model-free safe RL. Traditional policy gradient algorithms for model-free RL, such as Trust Region Policy Optimization (TRPO) Schulman et al. (2015) and Proximal Policy Optimization (PPO) Schulman et al. (2017) , allow the agent to explore any behavior during training, including behaviors that would be considered unsafe; this is unacceptable for safety-critical applications. To encode safety into training, a popular formulation is the Constrained Markov Decision Process (CMDP) Altman (2021) , which includes safety constraints and is typically solved using primal-dual methods Achiam et al. (2017) and modifying the trust region to exclude unsafe policy updates Milosevic et al. (2024) . However, the CMDP formulation is limited in its ability to model safety constraints; CMDPs constrain the expected discounted cost return, but for many practical applications we require an explicit bound on the probability that a sampled trajectory violates a safety constraint. Alternative approaches based on control theory ensure safety by preventing the agent from taking actions that would eventually lead to safety violations; this is achieved using, e.g., Lyapunov and barrier functions Chow et al. (2018) , shielding Alshiekh et al. (2018) ; Konighofer et al. (2023) or safety filters Hsu et al. (2024) . However, these approaches require a model of the environment to predict future safety, and thus are generally limited to model-based setups. Meanwhile, formal methods-based approaches, such as [ 4 ] , encode safety by leveraging Linear Temporal Logic (LTL) [ 6 ] as a formal reward-shaping structure. Unlike CMDPs, whereby the original objective is separate from the constraint, LTL formula satisfaction is encoded into the expected return itself, and under certain conditions the trained policy is guaranteed to maximize the probability of LTL formula satisfaction; however, no guarantees can be obtained during training. Since we presume no knowledge of the environment, as in standard model-free RL, we will rely on finite-sample learning to evaluate the agent’s ability to remain safe using probably approximately correct (PAC) guarantees. Finite-sample complexity bounds provide the number of samples needed to, with a given confidence, learn some target function with a certain accuracy Vidyasagar (2003) . Tools from statistical learning theory based on Vapnik Chervonenkis (VC) theory have successfully been able to provide finite sample bounds for learning in unknown environments Vidyasagar (2003) ; Tempo et al. (2005) , with recent work providing finite sample bounds even under changing target assumptions Vertovec et al. (2024) . Yet VC-theoretic techniques require the computation of the VC dimension, which is a difficult task for generic optimization problems. Under a convexity assumption, the so-called scenario approach offers a-priori probabilistic feasibility guarantees without resorting to VC theory Calafiore and Campi (2006) ; Campi and Garatti (2008) ; Campi and Garatti (2018) . The scenario approach traditionally relies on independent and identically distributed (i.i.d.) samples to establish its sample-complexity bounds. This creates a limitation in RL contexts, where the sampling distribution changes as policies are updated. As a result, safety guarantees established for one policy cannot be directly transferred when the policy changes. In this work, we overcome this limitation by extending the PAC guarantees to accommodate policy changes. Specifically, we derive a constraint on how much policies can shift while maintaining safety guarantees, and present SPoRt, an approach for adapting an existing safe policy to improve task-specific performance while maintaining a bound on the probability of safety violation, known prior to deploying or even training the adapted policy; this bound can be tuned by the user to trade off safety and task-specific performance. Our technical contributions underpinning SPoRt are as follows: 1. A data-driven method for obtaining a bound on the probability that a property (e.g. safety), in general expressed as an LTL formula, is violated for trajectories drawn using a given ‘safe’ base policy (Section 3 ). 2. Novel theoretical results that provide, for an episodic, model-free RL setup, a prior bound on the probability of property violation for a new task-specific policy, based on a ‘maximum policy ratio’ computed with respect to the ‘safe’ base policy (cf. previous point) (Section 4 ). 3. A projection-based method for constraining the task-specific policy to ensure that this prior bound holds (Section 5 ). 4. Projected PPO, an algorithm for training a new, task-specific policy, while maintaining a user-specified prior bound on property violation, thus trading off safety guarantees for task-specific performance (Section 6 ). We also test SPoRt on a time-bounded reach-avoid property and present experimental results demonstrating the safety-performance trade-off, as well as comparison of the theoretical prior bound to posterior bounds based on empirical violation rates (Section 7 and 8 ). All appendices and code 1 1 1 Link to code: https://github.com/JacquesCloete/sport . can be found in the supplementary material, which contains all proofs. 2 Models, Tasks and Properties We consider a model-free episodic RL setup where an agent interacts with an unknown environment modeled as a Markov Decision Process (MDP) [ 5 ] , specified by the tuple \langle\mathcal{S},\mathcal{A},p,\mu,r_{\mathrm{task}}\rangle , with a continuous state space \mathcal{S} and continuous action space \mathcal{A} . p(s^{\prime}|a,s):\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) and \mu(s)\in\Delta(\mathcal{S}) are the (unknown) state-transition and initial state distributions, respectively. We will consider learning a stochastic policy \pi(a|s):\mathcal{S}\to\Delta(\mathcal{A}) in a model-free setup. We use \tau^{p,\pi}_{{\mathbf{s}_{t}},T}=({\mathbf{s}_{t}},{\mathbf{s}_{t+1}},\ldots,{\mathbf{s}_{t+T}})^{p,\pi} to denote a realization

Read the complete text