Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL

Junyu Guo, Zhi Zheng, Donghao Ying, Ming Jin, Shangding Gu, Costas Spanos, Javad Lavaei · 2025

Read Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL by Junyu Guo, Zhi Zheng, Donghao Ying, Ming Jin, Shangding Gu, Costas Spanos, Javad Lavaei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12391.txt.

Introduction

Junyu Guo and co-authors tackle safe learning from fixed data in a 2025 paper. An agent must learn strong moves without fresh trial and error, since live trials could break safety limits in real tasks.

Their method first learns how past actors behaved from the stored data with a diffusion model. It then draws out a simpler, faster policy and adjusts its training so reward gains do not come at the cost of safety. Tests on robot tasks show steady safety, fast choices, and strong rewards. One set of settings works across tasks, which the team flags as a plus for real use.

Catalog record

CREATORJunyu Guo, Zhi Zheng, Donghao Ying, Ming Jin, Shangding Gu, Costas Spanos, Javad Lavaei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12391

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL Junyu Guo, Zhi Zheng, Donghao Ying, Ming Jin, Shangding Gu, Costas Spanos, Javad Lavaei Abstract Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints. We focus on an offline setting where the agent learns from a fixed dataset—a common requirement in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high-quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in real-world scenarios. 1 Introduction Offline reinforcement learning (RL) has advanced decision-making by learning from pre-collected datasets without online interaction ( Fujimoto et al., 2019 ; Levine et al., 2020 ) . For real-world control tasks (e.g., autonomous driving, industrial control), safety is equally critical. Safe RL addresses this by imposing constraints, often formulated as a constrained Markov decision process (CMDP) ( Gu et al., 2024c ; Altman, 2021 ) , to ensure high performance without violating safety requirements. These can be hard constraints (no violation at each step) ( Zheng et al., 2024 ; Ganai et al., 2024 ) or soft constraints (expected total cost below a threshold) ( Chow et al., 2018 ; Yang et al., 2020 ; Koirala et al., 2024 ; Guo et al., 2025 ; Shi et al., 2021 ) . We focus on the soft-constraint setting in this work. Offline safe RL faces two main hurdles: distribution shift and reward-safety trade-off. Firstly, the learned policy may deviate from the offline dataset’s state–action distribution, causing critic overestimation and extrapolation errors ( Fujimoto et al., 2019 ; Lyu et al., 2022 ) . To address value overestimation, previous methods have either constrained the learned policy to remain close to the behavioral policy Wu et al. (2019) ; Kumar et al. (2019) or conservatively penalized over-optimistic out-of-distribution (OOD) state-action pairs Kostrikov et al. (2021) ; Lyu et al. (2022) ; Xu et al. (2022) . Secondly, achieving high returns while strictly respecting safety becomes more challenging when these objectives conflict. Although constrained optimization methods ( Achiam et al., 2017 ; Liu et al., 2022 ) handle this in online RL, they rely on on-policy data collection, making them not directly applicable to offline settings. Hence, the key question is: How can we balance reward maximization and constraint satisfaction without risking out-of-distribution actions or unsafe behavior in a setting where no additional data can be collected? To address this, we propose Diffusion-Regularized Constrained Offline Safe Reinforcement Learning (DRCORL). First, DRCORL trains a diffusion policy to imitate the behavioral policy in the offline dataset; then it regularizes the learned policy via the diffusion model’s score function—removing the need for costly sampling from the diffusion model at inference. Second, we apply gradient manipulation to balance reward optimization and cost minimization, effectively handling conflicts between these two objectives. Furthermore, the behavioral policy serves as a regularizer, discouraging OOD actions that may compromise safety. We evaluate DRCORL on the DSRL benchmark ( Liu et al., 2023a ) , comparing against state-of-the-art offline safe RL methods. Experiments show that DRCORL consistently attains higher rewards while satisfying safety constraints. Our main contributions are: ① We exploit diffusion-based regularization to build a simple, high-speed policy with robust performance; and ② We introduce a gradient-manipulation mechanism for reward–cost trade-offs in purely offline settings, ensuring safety without sacrificing returns. 2 Preliminary A Constrained Markov Decision Process (CMDP) ( Altman, 2021 ) is defined by the tuple \langle\mathcal{S},\mathcal{A},P,r,c,\gamma\rangle , where \mathcal{S} and \mathcal{A} represent the state and action spaces, respectively. The transition kernel P specifies the probability {P}(s^{\prime}|s,a) of transitioning from state s to state s^{\prime} when action a is taken. The reward function is r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} , and the cost function is c:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} . The discount factor is denoted by \gamma . A policy is a function \pi:\mathcal{S}\rightarrow\Delta(\mathcal{A}) that represents the agent’s decision rule, i.e., the agent takes action a with probability \pi(a|s) in state s , and we define \Pi as the set of all feasible policies. Under policy \pi , the value function is defined as V^{\pi}_{\diamond}(\rho)=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}\diamond(s_{t},a_{t})|s_{0}\sim\rho] , where \diamond\in\{r,c\} , \rho is the initial distribution, and the expectation is taken over all possible trajectories. Similarly, the associated Q-function is defined as Q^{\pi}_{\diamond}(s,a)=\diamond(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}[V^{\pi}_{\diamond}(s^{\prime})] for \diamond\in\{r,c\} . In standard CMDP, the objective is to find a policy \pi\in\Pi that maximizes the cumulative rewards V^{\pi}_{r}(\rho) while ensuring that the cumulative cost V^{\pi}_{c}(\rho) remains within a predefined budget l . In the offline setting, the agent cannot interact directly with the environment and instead relies solely on a static dataset \mathcal{D}^{\mu}=\{(s_{i},a_{i},r_{i},s_{i}^{\prime},c_{i})\}_{i=1}^{N} consisting of multiple transition tuples, which is collected using a behavioral policy \pi_{b}(a|s) . This offline nature introduces the risk of distributional shift between the dataset and the learned policy. To mitigate this, an additional constraint is often imposed to limit the deviation of the learned policy \pi from the behavioral policy \pi_{b} , resulting in the optimization problem \max_{\pi\in\Pi}\mathbb{E}[V^{\pi}_{r}(\rho)]\text{ s.t. }D_{\operatorname{KL}}(\pi\|\pi_{b})\leq\epsilon,\mathbb{E}[V^{\pi}_{c}(\rho)]\leq l, (1) where D_{\operatorname{KL}}(p\|q) the KL-divergence between two distributions defined as D_{\operatorname{KL}}(p\|q)=\mathbb{E}_{x\sim p}\left[\log(p(x)/q(x))\right] . We use the KL-divergence to penalize the learned policy \pi ’s distance to the behavioral policy, though it is actually not a distance measure. To address safety constraints, primal-dual methods Ding et al. (2021) ; Paternain et al. (2022) ; Wu et al. (2024) typically reformulate the constrained optimization problem as follows: \max_{\pi\in\Pi}\mathbb{E}[V^{\pi}_{r}(\rho)]-\lambda(\mathbb{E}[V^{\pi}_{c}(\rho)]-l)\text{ s.t. }D_{\operatorname{KL}}(\pi\|\pi_{b})\leq\epsilon, (2) where \lambda\geq 0 is a surrogate for the Lagrange multiplier. When the safety constraint is violated, the multiplier \lambda increases to impose a greater penalty on the cost, thereby reducing the cost value. 3 Methodology 3.1 Diffusion Model for Policy Extraction Our main idea is to fully exploit the offline dataset to obtain a behavioral policy, and use the behavioral policy to guide the training of the target policy. A policy \pi(a|s) is a distribution on the action space. Previous work estimated the behavior policy with maximum likelihood estimation Fujimoto and Gu (2021) or leveraged a conditional variational autoencoder Kingma (2013) ; Sohn et al. (2015) . Here, we exploit the concept of diffusion models Sohl-Dickstein et al. (2015) ; Ho et al. (2020) ; Song et al. (2020b) to learn the unknown behavioral policy \pi_{b}(a|s) given its strong generative capabilities. Diffusion models have emerged as powerful generative tools to generate data sample x_{0}\sim p(x_{0}) with few-shot samples. They work by using a forward process q(x_{t}|x_{0}) to perturb the original distribution to a known noise distribution. Subsequently, this model generates the samples using the reverse denoising process p_{\psi}(x_{t-1}|x_{t}) . The forward process can generally be written with a forward stochastic differential equation (SDE) dx=-\frac{\beta(t)}{2}xdt+\sqrt{\beta(t)}dW_{t}, (3) where \beta(\cdot):[0,T]\rightarrow\mathbb{R}^{+} is a scalar function and the process \{W_{t}\}_{t\in[0,T]} is a standard Brownian motion. Our forward process is the discretized version of SDE in Eq. ( 3 ) perturbing the

Read the complete text