Sampling-Based Safe Reinforcement Learning
Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl, Melanie Zeilinger, Andreas Krause, Yarden As · 2026
Read Sampling-Based Safe Reinforcement Learning by Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl, Melanie Zeilinger, Andreas Krause, Yarden As (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19469.txt.
Introduction
Vignola and team propose SBSRL, a model-based reinforcement method that keeps learning safe by using constraints across a finite set of dynamics samples. That scheme approximates worst-case optimization over unknown dynamics and supports practical safety guarantees in continuous domains. Exploration instead limits epistemic uncertainty, removing the need for explicit exploration bonuses. Under mild conditions the authors prove high-probability safety in learning and a finite-time sample bound for near-optimal policies. Simulation and real-robot tests confirm safe, efficient exploration that extends to deep-ensemble control.
Catalog record
| CREATOR | Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl, Melanie Zeilinger, Andreas Krause, Yarden As |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98116 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sampling-Based Safe Reinforcement Learning Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl, Melanie Zeilinger, Andreas Krause, Yarden As Abstract Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning ( SBSRL ), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems. 1 Introduction Reinforcement learning (RL) is a framework for sequential decision-making that has achieved remarkable success across a wide range of application domains [ 39 , 51 , 33 , 57 , 18 , 43 ] . Despite this progress, training RL policies during deployment remains challenging due to two fundamental barriers: sample efficiency and safety . Standard RL algorithms typically require a large number of interactions with the environment, which is often impractical or costly in physical systems. Moreover, ensuring safety during training is itself a central challenge: because of epistemic uncertainty about the system dynamics, the consequences of any action are uncertain; in this sense, every interaction with the environment is inherently exploratory, and may cause catastrophic failures. Together, these challenges limit the applicability of RL largely to settings that can be accurately simulated or where high-quality demonstration data is available. A key challenge in safe exploration [ 3 ] is to satisfy constraints despite uncertainty, while simultaneously visiting uncertain regions of the state-action space to gather informative data and improve performance. Model-based RL provides a natural framework for addressing this tension by explicitly modeling uncertainty in the learned environment dynamics. Existing approaches [ 5 , 69 ] typically use such uncertainty estimates to pessimistically tighten constraints to guarantee safety throughout learning, and to add uncertainty-driven bonuses to encourage exploration. However, these methods often (i) introduce excessive conservatism through their constraint tightening, and (ii) require careful tuning of exploration bonuses to ensure both convergence and effective exploration. To address these challenges, we propose Sampling-Based Safe Reinforcement Learning ( SBSRL ), a model-based algorithm for safe and sample-efficient exploration. SBSRL learns an uncertainty-aware dynamics model, leveraging its epistemic uncertainty to drive exploration while also maintaining safety at all times. Sampling-based methods provide flexible uncertainty estimates, scale to expressive model classes [ 15 ] and admit natural parallelization. SBSRL leverages these properties to obtain a tractable approximation to the robust optimization problem at the core of safe exploration: finding a policy that satisfies safety constraints under worst-case dynamics among a set of plausible models of the environment (see Equation 6 ). Rather than solving this worst-case problem explicitly, SBSRL enforces safety on each sample of the dynamics, as we illustrate in Figure 1 . In addition, rather than adding an exploration bonus to the objective, SBSRL encodes exploration as a constraint that requires the agent to collect sufficient information along its trajectories. As a result, exploration is triggered only when a greedy policy is not informative enough, yielding a simpler and more robust strategy for safe exploration. Goal tightened constraint f_{n}^{1} f_{n}^{2} f_{n}^{M} f^{\star} Figure 1 : We illustrate a safe exploration task in which a car with unknown dynamics must learn to reach a goal while avoiding obstacles (tires). At each episode n , SBSRL maintains a set of M dynamics samples f_{n}^{1},\dots,f_{n}^{M} and enforces tightened safety constraints jointly across their corresponding model-generated rollouts, denoted by transparent cars. For instance, the depicted policy would lead to constraint violations under the plausible model f_{n}^{M} . SBSRL updates its policy by simulating it in dynamics f_{n}^{1},\dots,f_{n}^{M} , thereby ensuring safety under the true system dynamics f^{\star} . As more episodes are collected and uncertainty shrinks, the sampling-based approximation becomes progressively less conservative, enabling SBSRL to discover higher-reward trajectories and ultimately reach the goal. Our contribution. • We propose SBSRL , a novel model-based RL algorithm for safe exploration in continuous state-action spaces. SBSRL enforces safety using a finite number of dynamics samples, and promotes exploration via a constraint on the epistemic uncertainty. • We show that when the dynamics are modeled using a Gaussian process, SBSRL guarantees safety with high probability. In addition, we provide a sample-complexity bound showing that SBSRL obtains near-optimal policies in a finite number of episodes. • We empirically validate SBSRL both in simulation and on hardware. With Gaussian process dynamics models, consistent with our theory, SBSRL achieves safe and efficient exploration. We also introduce a scalable neural ensemble variant that enforces constraints jointly across multiple dynamics and evaluate it in SafetyGym and RWRL [ 50 , 23 ] . Finally, we show that these principles carry over to real-world hardware, enabling safe online learning in practice. 2 Related Work Safety in CMDPs. Safety in RL has been modeled in various ways [ 24 , 10 , 27 ] . Among these, constrained Markov decision processes (CMDPs) [ 2 ] provide a natural framework, as they enjoy many classical results from planning in MDPs, and can be used to formulate different notions of safety [ 68 , 17 , 65 ] . Many works address learning and planning in CMDPs, both in discrete [ 63 , 21 , 41 ] and continuous [ 1 , 6 , 52 , 29 ] state-action spaces. However, these algorithms ensure safety only at convergence, and may violate constraints during learning. This problem of safely learning, i.e., safe exploration, has been tackled in tabular CMDPs [ 40 , 60 , 66 , 67 , 20 , 11 ] , where strong guarantees are available, but these approaches do not scale well to continuous problems. Extensions to continuous state-action spaces have been explored in the model predictive control (MPC) literature [ 34 , 28 , 64 , 46 , 48 ] . However, these techniques rely on online planning, requiring the solution of an optimization problem at each step and thus incurring significant computational cost. Sampling-based algorithms. Sampling-based methods are popular in robotics for their efficiency and parallelizability [ 59 , 36 , 70 ] , but they remain largely restricted to unconstrained settings. Notable exceptions include scenario optimization [ 12 , 61 ] , which requires sampling from the true data-generating process, conformal prediction [ 38 ] , which typically yields post-hoc guarantees; and dynamics sampling approaches within MPC [ 48 , 45 ] . Closely related to our work, 45 sample functions from GP dynamics models and enforce constraints jointly across all sampled dynamics within an MPC framework. We apply this principle—to the best of our knowledge, for the first time—to continuous-domain CMDPs. Existing CMDP methods [ 6 , 5 ] typically tackle epistemic uncertainty via worst-case optimization over plausible models. However, this optimization is generally intractable and approximated via heuristics, sacrificing