A Provable Approach for End-to-End Safe Reinforcement Learning

Akifumi Wachi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, Youhei Akimoto, LY Corporation University of Tsukuba RIKEN AIP · 2025

Read A Provable Approach for End-to-End Safe Reinforcement Learning by Akifumi Wachi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, Youhei Akimoto, LY Corporation University of Tsukuba RIKEN AIP (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21852.txt.

Introduction

Akifumi Wachi and co-authors present a 2025 method for safe reinforcement learning that stays safe from training through use. Past safe methods guard one stage or the other, but not the whole life of a policy. Their method, called PLS, joins offline learning with a guarded rollout stage.

The policy is first learned offline from set target scores. At rollout, only those target scores are tuned, with a cautious math tool that tracks risk. The team proves the method finds near-best targets while keeping safety with high odds. Tests show it beats past methods on both safety and reward.

Catalog record

CREATORAkifumi Wachi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, Youhei Akimoto, LY Corporation University of Tsukuba RIKEN AIP
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Provable Approach for End-to-End Safe Reinforcement Learning Akifumi Wachi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, Youhei Akimoto, LY Corporation University of Tsukuba RIKEN AIP Abstract A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL ( PLS ), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation. 1 Introduction Reinforcement learning (RL) has exhibited remarkable capabilities in a wide range of real problems, including robotics [ 32 ] , data center cooling [ 34 ] , finance [ 23 ] , and healthcare [ 59 ] . RL has attracted significant attention through its successful deployment in language models [ 38 , 21 ] or diffusion models [ 7 ] . As RL becomes a core component of advanced AI systems that affect our daily lives, ensuring the safety of these systems has emerged as a critical concern. Hence, while harnessing the immense potential of RL, we must simultaneously address and mitigate safety concerns [ 4 ] . Safe RL [ 18 , 20 ] is a fundamental and powerful paradigm for incorporating explicit safety considerations into RL. Given its wide range of promising real-world applications, safe RL naturally spans a broad scope and involves several critical considerations in its formulation. For example, design choices must be made regarding the desired level of safety (e.g., safety guarantees are required in expectation or with high probability), the phase in which safety constraints are enforced (e.g., post-convergence or even during training), and other related aspects [ 54 , 27 ] . A longstanding goal in safe RL is to develop a methodology with a safety guarantee throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this goal. In online safe RL, where an agent learns its policy while interacting with the environment, ensuring safety is especially challenging during the initial phases of policy learning. While safe exploration [ 50 ] , sim-to-real safe RL [ 24 ] , or end-to-end safe RL [ 11 ] have been actively studied, they typically rely on strong assumptions, such as (partially) known state transitions. Also, in offline safe RL, where a policy is learned from a pre-collected dataset, it remains difficult to deploy a safe policy in a real environment due to distribution mismatch issues between the offline data and the actual environment, even though training can proceed without incurring any immediate safety risks. Our contributions. We propose Provably Lifetime Safe RL ( PLS ), an algorithm designed to address the longstanding goal in safe RL. PLS integrates offline policy learning with online policy evaluation and adaptation with high probability safety guarantee, as illustrated in Figure 1 . Specifically, PLS begins by training a policy using an offline safe RL algorithm based on return-conditioned supervised learning (RCSL). Given this resulting return-conditioned policy, PLS then seeks to optimize a set of target returns by maximizing the reward return subject to a safety constraint during actual environmental interaction. Through rigorous analysis, we demonstrate that leveraging Gaussian processes (GPs) for this optimization is theoretically sound, which enables PLS to optimize target returns in a Bayesian optimization framework. We further prove that, with high probability, the resulting target returns are near-optimal while guaranteeing safety. Finally, empirical results demonstrate that 1) PLS outperforms baselines in both safety and task performance, and 2) PLS learns a policy that achieves high rewards while ensuring safety throughout the entire process from learning to operation. Figure 1 : A conceptual illustration of PLS . After learning a return-conditioned policy using offline safe RL, PLS optimizes target returns through safe online policy evaluation via Gaussian processes. A key advantage of PLS is that safety is guaranteed at least with high probability in the entire process. 2 Related Work Safe RL [ 18 ] is a promising approach to bridge the gap between RL and critical decision-making problems related to safety. A constrained Markov decision process (CMDP, [ 3 ] ) is a popular model for formulating a safe RL problem. In this problem, an agent must maximize the expected cumulative reward while guaranteeing that the expected cumulative safety cost is less than a fixed threshold. Online safe RL. Although safe RL in CMDP settings has been substantially investigated, most of the existing literature has considered “online” settings, where the agent learns while interacting with the environment [ 54 ] . Prominent algorithms fall into this category, as represented by constrained policy optimization (CPO, [ 1 ] ), Lagrangian-based actor-critic [ 6 , 8 ] , and primal-dual policy optimization [ 39 , 57 ] . In online safe RL, satisfaction of safety constraints is not usually guaranteed during learning, and many unsafe actions may be executed before converging. To mitigate this issue, researchers have investigated safe exploration [ 50 , 5 , 52 ] , formal methods [ 2 , 17 ] , or end-to-end safe RL [ 11 , 25 ] . These techniques, however, typically rely on strong assumptions (e.g., known state transitions), and excessively conservative policies tend to result in unsatisfactory performance or inapplicability to complex systems. Therefore, simultaneously achieving both reward performance and guaranteed safety within the online safe RL paradigm is inherently difficult. Offline safe RL. Offline reinforcement learning (RL) [ 33 , 40 ] trains an agent exclusively on a fixed dataset of previously collected experiences. Since the agent does not interact with the environment during training, no potentially unsafe actions are executed during learning. Extending this setup to incorporate explicit safety requirements has led to the area of offline safe RL [ 30 , 31 , 55 , 42 , 37 ] . In this context, the objective is to maximize expected cumulative reward while satisfying pre-specified safety constraints, all from a static dataset. Because the policy is never deployed during training, offline safe RL is especially appealing for safety-critical domains. Le et al. [ 30 ] pioneered this direction with an algorithm that optimizes return under safety constraints using only offline data. Liu et al. [37] proposed a constrained decision transformer (CDT) that solves safe RL problems by sequence modeling by extending decision transformer [ 10 ] architectures from unconstrained to constrained RL settings. Despite such progress, offline safe RL still suffers from a central difficulty: learned policies often become either unsafe or overly conservative, largely due to the intrinsic challenges of off-policy evaluation (OPE) in stateful

Read the complete text