Safe-Support Q-Learning: Learning without Unsafe Exploration
Yeeun Lim a, Narim Jeong b, Donghwan Lee b · 2026
Read Safe-Support Q-Learning: Learning without Unsafe Exploration by Yeeun Lim a, Narim Jeong b, Donghwan Lee b (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25379.txt.
Introduction
Lim, Jeong and Lee put forth Safe-Support Q-Learning, barring unsafe visits. Most safe RL ways roam into unsafe states. Their frame uses a safe-set-backed behavior policy, so paths stay safe while roaming enough to learn, with no need for near-optimal behavior. They train Q-function and policy apart. A KL-tuned Bellman target keeps Q-values near behavior policy. A parametric extraction step nears the best policy. Tests show steady learning, well-tuned values, and safer behavior at par or better than baselines.
Catalog record
| CREATOR | Yeeun Lim a, Narim Jeong b, Donghwan Lee b |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73874 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25379
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safe-Support Q-Learning: Learning without Unsafe Exploration Yeeun Lim a, Narim Jeong b, Donghwan Lee b Abstract Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the policy induced from the trained Q-values and propose a parametric policy extraction method to approximate the optimal policy. Our approach provides a unified framework that can be adapted to different action spaces and types of behavior policies. Experimental results demonstrate that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than existing baselines. 1 Introduction Reinforcement learning (RL) 36 has become a powerful framework for autonomous decision-making. By optimizing cumulative rewards through the interaction with the environment, RL agents learn appropriate behaviors with minimal human intervention. However, as RL systems are increasingly deployed in high-risk domains such as autonomous driving and robotics, ensuring safety during the learning and implementation process has become a critical challenge. The core difficulty of this issue arises from the fundamental nature of RL: optimal policies are often discovered through exploration. This process inevitably requires visiting a wide range of state-action pairs, including those that can lead to undesirable or unsafe outcomes. Traditional safe RL 12 attempts to mitigate this issue by incorporating constraints or risk-sensitive objectives into the learning process. For example, constrained policy optimization and actor-critic methods aim to maximize the expected return while satisfying safety constraints 42 , 18 , whereas conditional value-at-risk focuses on reducing the risk of catastrophic events in constrained settings 44 , 20 , 19 . Despite these advances, most existing approaches still rely on interactions with unsafe states. For instance, safety violations are often controlled through penalties, expectation-based constraints, or bounded violation probabilities 41 , 29 , 32 , but they are not completely eliminated. Even methods such as safety critics 2 , 43 require exposure to unsafe transitions to estimate the risk of state-action pairs. Consequently, safety has been incorporated as a regularized objective rather than being strictly guaranteed in the majority of prior works. In parallel, an alternative perspective on safety has been extensively studied in control theory. Proportional–integral–derivative (PID) control 17 , Lyapunov-based stabilization 33 , and model predictive control (MPC) 15 are examples of classical approaches for designing controllers whose trajectories stay inside the certified safe or stable regions. Rather than penalizing unsafe outcomes, these methods ensure that the system never deviates from a predetermined set. While such approaches do not directly optimize long-term reward in the RL sense, they provide principled mechanisms for guaranteeing safety at the trajectory level. This motivates the consideration of whether RL can be constrained to operate within the support of safe states. Based on this consideration, this study adopts a stricter safety requirement: the learning process must not produce trajectories that visit unsafe states. Unlike conventional constrained RL 1 , 6 , 35 , 42 , we require that unsafe states be entirely excluded from the trajectory distribution. However, restricting exploration to a safe subset of states may limit the achievable performance. This raises a fundamental question: how can an agent sufficiently explore and improve its policy under such strict safety constraints? To address this challenge, we propose a Q-learning-based safe RL algorithm with three key components. First, we employ a behavior policy supported on a safe set. This policy does not need to be optimal but is intentionally stochastic to enable sufficient exploration within the safe region. It can be constructed either by using well-performing controllers or by training from safe datasets. Second, we modify the Q-learning objective by introducing a KL divergence regularization term that aligns the Q-function with the behavior policy. This regularization constrains the Q-function to remain within the support of safe behaviors while enabling improvement in expected return. Third, we provide a principled method for deriving a policy from the trained Q-function with theoretical justification. Since exact policy extraction is often computationally intractable, we further propose a surrogate policy that approximates the optimal policy induced by the Q-function. The main contributions of this work are summarized as follows: 1. We introduce a behavior policy that enables RL without unsafe exploration in both online and offline settings, requiring only stochastic (not near-optimal) policies under the assumption that the induced trajectories remain within the safe set. 2. We adopt a two-stage safe-support Q-learning framework in which the Q-function and policy are trained separately. We first propose a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the optimal policy and show that the safe Bellman operator admits a unique fixed point. In addition, we formulate a KL-regularized objective that balances reward and safety and propose a parametric policy extraction method for continuous actions. 3. The proposed method provides a unified framework that can be adapted to different action spaces and types of behavior policies. 4. Experiments show that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than baselines. 2 Related Work Safe RL. Prior research in safe RL has primarily focused on incorporating safety constraints into the optimization objective. One example of this research is the constrained Markov decision process 1 , 39 , where the agent seeks to maximize cumulative rewards while satisfying explicit safety constraints. Risk-sensitive objectives such as conditional value-at-risk have also been adopted to control tail risks 44 , 20 , 19 . Chance-constrained formulations explicitly bound violation probabilities 5 but still allow unsafe states with limited probability. Entropy regularization is often incorporated to improve exploration and stability in constrained variants such as WCSAC 42 . Another line of work employs reward shaping 24 , penalizing unsafe states or actions without explicit constraints. In value-based settings, safe Q-learning methods leverage safety through cost critics or safety critics that estimate the likelihood of unsafe transitions and discourage risky behaviors during learning 41 , 35 . Despite these advances, existing safe RL methods primarily aim to reduce or bound the risk associated with unsafe states, typically in expectation or probabilistic terms. As a result, unsafe states may still be visited during training, even if their frequency or impact is controlled. In contrast, our work imposes a stricter requirement: under the safe set assumption, unsafe states are excluded from the learning process. This enforces safety at the level of trajectory support rather than through risk penalization. Control-based safety. Control theory has provided principled tools for guaranteeing stability and safety. Classical controllers such