Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation

Qisong He, Xinmiao Huang, Jinwei Hu, Zhuoyun Li, Yi Dong, Changshun Wu, Xiaowei Huang · 2026

Read Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation by Qisong He, Xinmiao Huang, Jinwei Hu, Zhuoyun Li, Yi Dong, Changshun Wu, Xiaowei Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14174.txt.

Introduction

He and six co-authors train risk-aware robot navigation policies under safety constraints checked after training. Safe driving must hold up under high-consequence perception uncertainty in cluttered spaces, yet most safe reinforcement learning judges safety by average cumulative cost, which can hide dangerous tail-risk behavior. Their framework tunes Conditional Value-at-Risk constraints on an off-policy TD3 backbone, tuning the policy to high-cost tail outcomes rather than averages alone. After training, neural network reachability checks compute action reachable sets under bounded observation uncertainty.

Catalog record

CREATORQisong He, Xinmiao Huang, Jinwei Hu, Zhuoyun Li, Yi Dong, Changshun Wu, Xiaowei Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38248
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation Qisong He, Xinmiao Huang, Jinwei Hu, Zhuoyun Li, Yi Dong, Changshun Wu, Xiaowei Huang Abstract Safe navigation for mobile robots demands policies that remain reliable under the high-consequence perception uncertainty of cluttered environments. Yet most existing safe reinforcement learning (RL) methods assess safety through average cumulative cost. Such metrics can mask dangerous tail-risk behaviors. To address this, we propose a framework that trains risk-sensitive policies through Conditional Value-at-Risk (CVaR) constrained optimization on an off-policy TD3 backbone and evaluates their safety margins post-training through neural network reachability verification. During training, the policy is optimized under CVaR constraints on cumulative costs, promoting sensitivity to high-cost tail outcomes rather than average behavior alone. After training, we compute action reachable sets under bounded observation uncertainty using Taylor Model analysis, yielding a safety rate metric that quantifies the proportion of evaluated states at which the policy’s reachable action set remains within prescribed safety margins. A key finding is that policies trained with CVaR constraints maintain larger safety margins from obstacles across evaluated states. This makes them significantly more amenable to formal reachability verification. Experiments across ten navigation scenarios and six baselines show that our method achieves a 98.3% success rate, the highest safety verification rate among all compared methods, while revealing that average cost rankings and reachability-based safety rankings can diverge. This indicates that reachability verification captures risks which are missed by empirical cost metrics alone. We further validate our approach on a physical Clearpath Jackal robot, demonstrating successful sim-to-real transfer. I Introduction Safe deployment of autonomous mobile robots requires navigation policies that remain reliable not only along a single realized trajectory, but across the full range of behaviors that may arise under perception uncertainty (e.g., sensor noise). In practice, however, reinforcement learning (RL) policies are typically evaluated by average reward and average cumulative cost. These metrics can mask dangerous tail-risk behaviors. As illustrated in Fig. 1 , two policies may both reach the goal without collision along their nominal trajectories, yet their action reachable sets under bounded observation uncertainty differ substantially. The baseline policy’s reachable tube encroaches on obstacles, exposing latent collision risk that conventional metrics fail to capture. This discrepancy motivates combining risk-sensitive training with formal post-training verification to assess whether policies deemed safe by training metrics remain verifiable under observation uncertainty. Fig. 1 : Comparison of action reachable sets under bounded observation uncertainty. Black lines denote actual trajectories. Orange and green tubes denote reachable sets of a baseline policy and VIA, respectively. Fig. 2 : Overview of the VIA framework. A principled approach to safe RL formulates the problem as a constrained Markov decision process (CMDP) [ 3 ] , where the agent maximizes expected return subject to constraints on cumulative costs. Most existing methods enforce these constraints in expectation [ 1 , 26 , 14 , 24 ] , which limits the overall frequency of unsafe events but provides no control over rare, high-consequence outcomes. Conditional Value-at-Risk (CVaR) [ 25 ] addresses this limitation by constraining the expected cost over the tail of the distribution beyond a chosen quantile, encouraging policies that are robust to high-cost events rather than merely well-behaved on average [ 8 ] . Nonetheless, existing CVaR-based safe RL methods either rely on distributional assumptions (e.g., Gaussian approximations or trust-region surrogates) that may not hold in practice [ 27 , 22 ] , or require on-policy updates that limit sample efficiency [ 29 ] . Moreover, a fundamental limitation shared by both expectation-based and CVaR-based approaches is that they provide only statistical training-time signals. A policy with low average cost or favorable CVaR may still produce unsafe actions under the observation perturbations encountered at deployment. Formal verification methods can address this gap by reasoning over the set of all possible policy outputs. In robotics, Control Barrier Functions (CBFs) [ 4 ] and Hamilton-Jacobi (HJ) reachability [ 5 ] provide runtime safety guarantees, while shielding approaches [ 2 ] override unsafe actions online. These methods enforce safety at runtime by reasoning over closed-loop system dynamics, which typically requires known analytical models or access to a differentiable safety function. Polynomial Arithmetic for Reachability (POLAR) [ 16 ] targets a different layer of the safety problem. Rather than constraining closed-loop trajectories, it computes guaranteed bounds on a trained network’s outputs given interval input perturbations, enabling direct analysis of whether the learned policy’s action set remains within prescribed safety margins. Importantly, stable CVaR-constrained training is itself essential for this verification to be meaningful. Unconstrained or expectation-only policies tend to produce broader, less conservative action distributions, yielding reachable sets that frequently encroach on unsafe regions and fail verification. We propose VIA (Verification-Informed safe reinforcement learning for Autonomous navigation), a framework (Fig. 2 ) that unifies CVaR-constrained policy optimization with post-training reachability-based safety verification for mobile robot navigation. During training, VIA maximizes expected return subject to CVaR-based safety constraints within an off-policy TD3 architecture [ 13 ] , providing sample-efficient learning while maintaining sensitivity to high-cost tail events. A key technical challenge in off-policy CVaR optimization is the stable estimation of the Value-at-Risk (VaR) quantile threshold. To address this, we introduce an adaptive VaR update mechanism driven by quantile deviation that reduces oscillation and promotes stable convergence toward the target quantile. After training, we apply POLAR with layered safety boundaries to compute guaranteed bounds on the policy’s action outputs given interval input perturbations that represent bounded observation uncertainty. This produces a safety rate metric that quantifies the proportion of evaluated states at which the policy’s reachable set remains within prescribed safety margins. Because CVaR-constrained training shapes the policy toward conservative, tightly bounded action distributions, the resulting networks exhibit substantially higher safety rates than expectation-constrained or unconstrained baselines. Our main contributions are as follows: • We design a CVaR-constrained optimization framework on the TD3 architecture, featuring a distributional cost critic for tail-risk estimation and an adaptive VaR update mechanism that stabilizes quantile convergence. • We develop a safety verification framework combining POLAR reachability analysis with layered safety boundaries and propose safety rate metrics to quantify policy robustness under bounded observation uncertainty. • We validate VIA in both simulation and real-world deployment on a Clearpath Jackal robot. II Related work II-A Constrained Safe Reinforcement Learning Within the CMDP framework, a variety of constrained optimization methods enforce safety during training. Constrained Policy Optimization (CPO) [ 1 ] performs trust-region policy updates with constraint satisfaction at each iteration, though it requires recovery steps for infeasible updates. Projection based CPO (PCPO) [ 28 ] and FOCOPS [ 30 ] simplify the update procedure while maintaining feasibility guarantees. An alternative line of work leverages Lagrangian relaxation to convert the constrained problem into a saddle-point formulation. Reward Constrained Policy Optimization (RCPO) [ 26 ] introduces a multi-timescale update scheme with convergence guarantees under mild assumptions. SAC-Lagrangian [ 14 ] and PPO-Lagrangian [ 24 ] further combine standard RL backbones with Lagrangian methods, extending this framework to off-policy and on policy setting respectively. However, all of the above methods constrain only the expected cumulative cost, without explicitly controlling the tail of the cost distribution. II-B CVaR-Based Safe Reinforcement Learning To move beyond expectation-based constraints, several methods have incorporated CVaR into the RL

Read the complete text