Provably Safe Reinforcement Learning from Analytic Gradients
Tim Walter 1 , Hannah Markgraf † 1 , Jonathan Külz † 1,2 , and Matthias Althoff 1,2 · 2025
Read Provably Safe Reinforcement Learning from Analytic Gradients by Tim Walter 1 , Hannah Markgraf † 1 , Jonathan Külz † 1,2 , and Matthias Althoff 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01665.txt.
Introduction
Tim Walter and co-authors wrote this robotics paper in 2025. Its text carries the title "Leveraging Analytic Gradients in Provably Safe Reinforcement Learning," and it tackles a gap in safe robot training. Guards that keep a learning robot from unsafe acts exist for trial-based methods. The faster gradient-based methods, which learn from fewer trials, had no such guard.
The team took known guards that can pass gradients, reshaped their maps and math, and built them into a top learning method with a smooth test world. On three control tasks, guarded training matched the unguarded scores while keeping acts safe. The result, they say, brings proof-backed safety to a class of learning that plants and labs favor for its speed.
Catalog record
| CREATOR | Tim Walter 1 , Hannah Markgraf † 1 , Jonathan Külz † 1,2 , and Matthias Althoff 1,2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Provably Safe Reinforcement Learning from Analytic Gradients Tim Walter 1 , Hannah Markgraf † 1 , Jonathan Külz † 1,2 , and Matthias Althoff 1,2 Abstract Deploying autonomous robots in safety-critical applications requires safety guarantees. Provably safe reinforcement learning is an active field of research which aims to provide such guarantees using safeguards. These safeguards should be integrated during training to prevent a large sim-to-real gap. While there are several approaches for safeguarding sampling-based reinforcement learning, analytic gradient-based reinforcement learning often achieves superior performance and sample efficiency. However, there is no safeguarding approach for this learning paradigm yet. Our work addresses this gap by developing the first effective safeguard for analytic gradient-based reinforcement learning. We analyse existing, differentiable safeguards, adapt them through modified mappings and gradient formulations, and integrate them with a state-of-the-art learning algorithm and a differentiable simulation. We evaluate how different safeguards affect policy optimisation using numerical experiments on two classical control tasks. The results demonstrate safeguarded training without compromising performance. \pubyear 2021 \accepteddate XX October 2021 \publisheddate XX November 2021 \currentdate XX November 2021 \doiinfo OJCSYS.2021.Doi Number \sptitle Article Category \editor This paper was recommended by Associate Editor F. A. Author. keywords Safe reinforcement learning, policy optimisation, differentiable simulation, gradient-based methods, constrained optimisation, first-order analytic gradient-based reinforcement learning † † volume: 00 † † issue: XX † † paper-type: 1234567 † † received: XX September 2021 † † corresponding: CORRESPONDING AUTHOR: T. Walter (e-mail: [email protected] ) † † authornote: This work was supported by the Deutsche Forschungsgemeinschaft (German Research Foundation) under grant numbers AL 1185/31-1 and AL 1185/9-1. † † footnotetext: Equal contribution I INTRODUCTION The transfer of physical labour from humans and human-operated machines to robots is a long-standing goal of robotics research. While robots have been successfully deployed in controlled environments, such as factories, deploying strong robots in human proximity is still a challenge [ 1 , 2 ] . One reason is a lack of safety guarantees to ensure that robots do not harm humans or themselves [ 3 ] . Classical control algorithms, such as PID or MPC controllers, can be designed for safety [ 4 , 5 , 6 ] . While reinforcement learning controllers do not provide safety guarantees, they often achieve state-of-the-art performance [ 7 , 8 ] due to their broad applicability, representation power, and fast inference. In recent years, provably safe reinforcement learning has emerged as a research field to provide safety guarantees [ 9 , 10 ] . Current safeguards are applied in conjunction with reinforcement learning algorithms that rely on the policy gradient theorem to estimate reward landscapes [ 11 , 12 , 13 , 14 , 15 , 16 ] . With the advent of differentiable physics simulators [ 17 , 18 , 19 , 20 , 21 ] , analytically computing the gradient of the reward with respect to the actions has become possible. Reinforcement learning algorithms that exploit these gradients promise faster training and better performance [ 22 , 23 , 24 , 25 ] . While these simulators require approximations to remain differentiable, maintaining simulation accuracy is possible. As a result, one can train the policy exclusively in simulation before deploying it to real systems and avoid costly real-world training [ 26 , 27 ] . Applying safeguards during training is desirable for safety-critical systems to prevent an effectively enlarged sim-to-real gap [ 28 , 29 ] . Otherwise, the optimisation may converge to strategies that rely on unsafe states or actions. When deployment safeguards subsequently restrict these strategies, the policy can become suboptimal or even fail for non-convex objective landscapes, as it has not learned alternative, safe solutions [ 30 ] . While crafting reward functions that reliably encode safety requirements could theoretically prevent performance degradation in deployment, this is notoriously difficult without introducing unintended incentives [ 31 , 32 ] . Moreover, safeguards can guide the exploration process to aid learning in challenging solution spaces [ 13 , 33 ] . However, there are currently no safeguarding attempts for analytic gradient-based reinforcement learning. Our work addresses this by combining state-of-the-art analytic gradient-based reinforcement learning algorithms, differentiable safeguards, and differentiable simulations 1 1 1 Code available at https://github.com/TimWalter/SafeGBPO . To ensure safety, we incorporate a range of existing set-based safeguarding methods that map policy actions into a safe action set. We formalise desirable safeguarding properties in the context of differentiable optimisation and analyse existing methods with respect to these criteria. Based on this analysis, we propose targeted modifications, such as custom backward passes or adapted maps, that enhance their suitability for analytic gradient-based reinforcement learning. We also extend the applicability of one of the safeguards to state constraints state sets. We evaluate the approaches in differentiable simulations of various classical control problems. Throughout, we observe sample efficiency and final performance that exceeds or is on par with unsafe training and sampling-based baselines. In summary, our core contributions are: • the first provably safe analytic policy optimisation, • an in-depth analysis of some suitable safeguards, • adapted backward passes, an adapted mapping and extended applicability of these safeguards, • and an evaluation on two classical control tasks, demonstrating the potential of provably safe reinforcement learning from analytic gradients. II RELATED WORK We provide a literature review on the most relevant research areas: analytic gradient-based reinforcement learning, safeguards, and implicit layers. II-A ANALYTIC GRADIENT-BASED REINFORCEMENT LEARNING Analytic gradient-based reinforcement learning relies on a continuous computational graph from policy actions to rewards, which allows computing the first-order gradient of the reward with respect to the action. Relying on first-order gradient estimators often results in less variance than zeroth-order estimators [ 34 ] , which are usually obtained using the policy gradient theorem. Less variance leads to faster convergence to local minima of general non-convex smooth objective functions [ 22 , 23 ] . However, complex environments may lead to optimisation landscapes that are stiff, chaotic or contain discontinuities, which can stifle performance as first-order gradients suffer from empirical bias [ 34 ] . Using a smooth surrogate to approximate the underlying noisy reward landscape can alleviate this issue [ 25 ] . Moreover, naively backpropagating through time [ 35 ] can lead to vanishing or exploding gradients in long trajectories [ 36 ] . Various approaches have been introduced to overcome this issue: Policy Optimisation via Differentiable Simulation [ 24 ] utilises the gradient provided by differentiable simulators in combination with a Hessian approximation to perform policy iteration. It outperforms sampling-based methods. Short-Horizon Actor-Critic (SHAC) [ 25 ] tackles the empirical bias of first-order gradient estimators by training a smooth value function through a mean-squared-error loss, with error terms calculated from the sampled short-horizon trajectories through a td- \lambda formulation [ 37 ] . It prevents exploding and vanishing gradients by cutting the computational graph deliberately after a fixed number of steps and estimating the terminal value by the critic. The algorithm shows applicability even in contact-rich environments, which tend to lead to stiff dynamics. The successor Adaptive Horizon Actor-Critic [ 38 ] has a flexible learning window to avoid stiff dynamics and shows improved performance across the same tasks. Short-Horizon Actor-Critic also inspired Soft Analytic Policy Optimisation [ 21 ] , which integrates