CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
Rahul Narava, Siddharth Verma, Ojas Jain, Shashi Shekhar Jha, Mayank Shekhar Jha · 2026
Read CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning by Rahul Narava, Siddharth Verma, Ojas Jain, Shashi Shekhar Jha, Mayank Shekhar Jha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23576.txt.
Introduction
Narava and colleagues propose CAPSULE, a safe way to train agents in complex systems whose physics are unknown. It first learns a probabilistic model of the system offline. From that model it builds safety barriers that allow for model error, setting cautious limits. An online fix-up step enforces those limits while the agent explores, matching baseline rewards on control tests while cutting safety breaches sharply.
Catalog record
| CREATOR | Rahul Narava, Siddharth Verma, Ojas Jain, Shashi Shekhar Jha, Mayank Shekhar Jha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32773 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23576
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning Conference: Proc.@ of the Adaptive and Learning Agents Workshop (ALA 2026); May 25 – 26, 2026; Paphos, Cyprus, https://alaworkshop2026.github.io/ <<submission id>> Rahul Narava, Siddharth Verma, Ojas Jain, Shashi Shekhar Jha, Mayank Shekhar Jha Abstract . Ensuring safe exploration in high-dimensional systems with unknown dynamics remains a significant challenge. Existing safe reinforcement learning methods often provide safety guarantees only in expectation, which can still lead to safety violations. Control-theoretic approaches, in contrast, offer hard constraint–based safety guarantees but typically assume access to known system dynamics or require accurate estimation of control-affine models. In this paper, we propose a safe reinforcement learning framework that learns a probabilistic control-affine dynamics model in an offline setting. The learned model is leveraged to explicitly construct control barrier functions (CBFs) that incorporate model uncertainty to provide conservative safety constraints. These CBF constraints are enforced through an online constraint-based action correction mechanism, enabling safe exploration without overly restricting task performance. Empirical evaluations on nonlinear, complex continuous-control benchmarks demonstrate that our approach achieves returns comparable to those of existing baselines while significantly reducing safety violations. Aydeniz, Delgrange, Mohammedalamen, Yang (eds.) 1. Introduction Reinforcement Learning (RL) studies the problem of sequential decision-making through interactions between an agent and its environment ( 13 ) . At each time step, the agent observes the current state, selects an action, and influences the subsequent evolution of the environment. In return, the environment provides feedback in the form of a scalar reward. Unlike supervised learning, RL does not rely on predefined labels; instead, the agent improves its behavior through trial-and-error interactions. A defining characteristic of RL is the presence of temporal dependencies, where actions affect not only immediate rewards but also future states and long-term outcomes. The objective of an RL agent is to learn a policy that maximizes the expected cumulative reward over time. RL methods have demonstrated remarkable success across a wide range of challenging domains ( 10 ; 11 ; 6 ) . RL agents are driven by the objective of maximizing cumulative reward, which can lead to unsafe exploratory behaviors that incur high costs, damage the environment, or harm the agent itself. Without explicit safety considerations, this reward-driven optimization often encourages high-risk actions that push the agent into hazardous states and cause safety violations. This becomes especially critical in domains like autonomous driving, where unsafe exploration or policy deployment can endanger human lives. For example, a self-driving car must avoid collisions and respect speed limits rather than adopting aggressive maneuvers that cause accidents. Safe Reinforcement Learning (Safe RL) addresses these limitations by introducing mechanisms that enforce safety constraints during training, thereby keeping exploration within acceptable risk boundaries while still enabling effective policy learning. By embedding these constraints directly into the learning process, Safe RL provides a more reliable foundation for deploying RL systems in safety-critical real-world environments. Exisiting Safe RL methods seeks to maximize reward while enforcing expected safety constraints. However, in highly safety-critical settings, such expectation-based constraints are often insufficient—an agent may still produce unsafe actions as long as they are rare on average. Because such frameworks cannot guarantee strict constraint satisfaction, this motivates looking toward control-theoretic methods within RL, which are specifically designed to provide stronger, more reliable safety guarantees. In ( 3 ; 4 ) , the authors enforce safety by solving a convex optimization that generates a corrective safety-preserving action on top of a base RL policy. Although effective, these approaches generally assume known system dynamics and are typically evaluated in low-dimensional or simplified settings. L1-MBRL ( 12 ) reports that enforcing a control-affine structure, where the learned dynamics depend linearly on the action. This can lead to unstable training and degraded performance. These issues become more prevalent in high-dimensional continuous-control environments, where the true dynamics are complex and highly nonlinear. As a consequence, ensuring safety guarantees under unknown dynamics remains a central challenge. To address these limitations, this paper proposes C ontrol-theoretic A ction P erturbations for S afe U ncertainty-aware Reinforcement LE arning dubbed as CAPSULE . This approach to explicitly learn a control-affine dynamics model suitable for online constraint-based safety filtering while avoiding the instability observed in prior online learning approaches. Our key idea is to decouple model learning from policy optimization through an offline pretraining phase that leverages a heteroscedastic probabilistic ensemble of dynamics models. This approach stabilizes the learning of the control-affine model, leveraging offline data before any online interaction begins. Offline pretraining mitigates the model bias reported in earlier works, yielding a much more stable model that is structurally compatible with our framework. The resulting model provides uncertainty estimates necessary for making safety-critical decisions in subsequent online learning. Unlike ( 3 ) , which relies on Bayesian models such as the Gaussian Process (GP) Model, i.e, effective but limited to low-data and low-dimensional regimes, our heteroscedastic ensemble provides scalable, PETS-style ( 5 ) uncertainty estimates suitable for modern continuous-control benchmarks. This uncertainty enables us to construct state-dependent safety margins and maintain a conservative approach. This allows our framework to enforce safety guarantees even in the presence of model error. Thus, the agent benefits from the exploration and performance while maintaining provable safety constraints derived from the learned dynamics and their uncertainty. Our contributions are of threefold: • Uncertainty Aware Control-Affine Probabilistic Ensemble Dynamics Model: An offline-trained probabilistic dynamics model that maintains control-affine structure for higher dimensional environments along with heteroscedastic uncertainty estimate for safety guarantees. • Online Safe RL via CBFs using learned models: A hybrid method that pairs Reinforcement Learning with a CBF based safety correction using a pre-trained model allowing for hard-constraint satisfaction which is required in safety critical settings. • Evaluation on Complex Continuous-Control Domains: Experimentation over complex settings such as Safety Gymnasium (MuJoCo based) showing improved performance and fewer safety violations compared to existing safe RL approaches. 2. Background 2.1. Markov Decision Process (MDP) MDP is formulated by the tuple {M}\!=<\!S,A,{R},\mathcal{P},\gamma\!> where S is the state space, A is the action space and R\!:\!(S\times{A})\!\rightarrow\![r_{\text{min}},r_{\text{max}}] is the reward function such that r_{\text{min}},r_{\text{max}}\in\mathbb{R} . Further, \mathcal{P}\!:\!(S\times{A}\times{S})\rightarrow[0,1] captures the transition dynamics of the environment i.e \mathcal{P}(\boldsymbol{\cdot}|s,a) is a probability distribution over the state space that outputs the probability of agent traversing into next state s^{\prime} given current state s and action taken a . Discounting parameter is given by \gamma\in[0,1) . A policy \pi:(S\times A)\rightarrow[0,1] is a function that outputs the probability of taking an action a in the state s i.e a mapping \pi(\cdot\!\mid\!s) from states to probability distribution over actions. The objective of a reinforcement learning (RL) agent is to learn a policy \pi^{*} that maximizes the expected discounted return given by \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right] . The state-value function, or simply the value function, V^{\pi}(s) , under a policy \pi , is defined as the expected return when starting from state s and following policy \pi thereafter: (1) V^{\pi}(s)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{t}\mid S_{0}=s\right] The Q-value function, or action-value function, Q^{\pi}(s,a) under a policy \pi , is defined as the expected return (cumulative discounted reward) starting from state s , taking action a , and thereafter following policy \pi : (2) Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{t}\mid S_{0}=s,A_{0}=a\right] Trust Region Policy Optimization TRPO is an on-policy actor-critic algorithm developed to improve the stability and reliability of policy optimization.