Semi-gradient DICE for Offline Constrained Reinforcement Learning

Woosung Kim, JunHo Seo, Jongmin Lee, Byung-Jun Lee · 2025

Read Semi-gradient DICE for Offline Constrained Reinforcement Learning by Woosung Kim, JunHo Seo, Jongmin Lee, Byung-Jun Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08644.txt.

Introduction

Woosung Kim, JunHo Seo, Jongmin Lee, and Byung-Jun Lee fix cost estimation in offline learning with limits in a 2025 paper. Offline methods learn from logged data, and a family called DICE corrects the mismatch between a policy's visit pattern and the data. Constrained versions must also price costs well, not just chase reward.

Recent tweaks that raised reward scores quietly broke cost reading, the authors find, because a half gradient shortcut solves a different problem than intended. Their repair keeps that shortcut for policy learning while restoring sound cost estimates. On a standard safety benchmark it reached top results.

Catalog record

CREATORWoosung Kim, JunHo Seo, Jongmin Lee, Byung-Jun Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semi-gradient DICE for Offline Constrained Reinforcement Learning Woosung Kim, JunHo Seo, Jongmin Lee, Byung-Jun Lee Abstract Stationary Distribution Correction Estimation (DICE) addresses the mismatch between the stationary distribution induced by a policy and the target distribution required for reliable off-policy evaluation (OPE) and policy optimization. DICE-based offline constrained RL particularly benefits from the flexibility of DICE, as it simultaneously maximizes return while estimating costs in offline settings. However, we have observed that recent approaches designed to enhance the offline RL performance of the DICE framework inadvertently undermine its ability to perform OPE, making them unsuitable for constrained RL scenarios. In this paper, we identify the root cause of this limitation: their reliance on a semi-gradient optimization, which solves a fundamentally different optimization problem and results in failures in cost estimation. Building on these insights, we propose a novel method to enable OPE and constrained RL through semi-gradient DICE. Our method ensures accurate cost estimation and achieves state-of-the-art performance on the offline constrained RL benchmark, DSRL. 1 Introduction Constrained reinforcement learning (RL) trains agents to maximize return while adhering to predefined cost constraints. Conventional online RL involves interactions with the environment that can be unsafe or costly due to potential constraint violations in sensitive environment. To avoid the risk of violations during interactions, offline constrained RL offers a practical alternative by relying on a fixed dataset of pre-collected experiences, eliminating the need for potentially unsafe online exploration during training. Specifically, offline constrained RL aims to maximize the expected return of a policy subject to cost constraints using only offline data. This objective makes stationary distribution correction estimation (DICE) a promising framework, leveraging stationary distribution to simultaneously estimate and optimize policy performance for both reward and cost functions [ 35 , 21 , 47 ] . Despite its theoretical appeal, prior DICE-based research has largely been limited to finite domains or struggled to achieve competitive performance in continuous state and action spaces compared to alternative frameworks. Fortunately, recent empirical findings in offline RL [ 37 , 30 ] show that incorporating semi-gradient updates into the DICE objective significantly improves training stability and achieves state-of-the-art performance in large and continuous domains. However, our analysis reveals that applying semi-gradient methods causes DICE to lose its off-policy e valuation (OPE) capability, rendering them unsuitable for constrained scenarios. Despite being were adopted to stabilize conflicting gradients, inspired by successful bootstrapped deep RL methods, we discovered that semi-gradient DICE algorithms inherently solve a fundamentally different optimization problem. This divergence yields a solution with distinct characteristics. We show that semi-gradient DICE algorithms are closely related to behavior-regularized offline RL, returning a policy correction rather than the intended stationary distribution correction . This observation partially explains the empirical success of the semi-gradient updates ( Section 4 ). Building on these analyses, we propose CORSDICE, an offline constrained RL algorithm. CORSDICE recovers a valid stationary distribution from the policy correction found by semi-gradient DICE, thereby enabling OPE while preserving the strong empirical performance ( Section 5 ). We provide extensive empirical results validating our method across offline RL, OPE, and offline constrained RL tasks ( Section 6 ). 2 Preliminary and Related Works Markov Decision Process (MDP) We model the environment as an infinite-horizon discounted Markov Decision Process (MDP) [ 40 ] , defined by the tuple \mathcal{M}:=\langle S,A,T,r,\gamma\rangle , where S and A are the state and action spaces, T:S\times A\to\Delta(S) is the transition distribution, r:S\times A\to\mathbb{R} is the reward function, and \gamma\in[0,1) is a discount factor. Additionally, we define p_{0}:\Delta(S) as the initial state distribution and a policy \pi:S\to\Delta(A) as an action distribution conditioned on the state. Given a policy \pi , its stationary distribution d_{\pi}(s,a):=(1-\gamma)\sum^{\infty}_{t=0}\gamma^{t}\textrm{Pr}(s_{t}=s,a_{t}=a|\pi) represents the discounted probability of visiting a state-action pair (s,a) when following \pi . These distributions satisfy the single-step transposed Bellman recurrence: d_{\pi}(s,a)=(1-\gamma)p_{0}(s)\pi(a\mid s)+\gamma\pi(a\mid s)(\mathcal{T}_{*}d)(s) , where (\mathcal{T}_{*}d)(s):=\sum_{\bar{s},\bar{a}}T(s\mid\bar{s},\bar{a})d(\bar{s},\bar{a}) . The expected discounted sum of rewards (return), or value \rho(\pi) , is defined as \rho(\pi):=(1-\gamma)\mathbb{E}_{\pi}\left[\sum^{\infty}_{t=0}\gamma^{t}r(s_{t},a_{t})\right] and can be equivalently evaluated using stationary distribution \rho(\pi)=\mathbb{E}_{(s,a)\sim d_{\pi}}[r(s,a)] . OptiDICE OptIDICE [ 20 ] addresses distribution shift in offline RL by optimizing for a stationary distribution d that maximizes expected return while being regularized towards the dataset distribution d_{D} via an f -divergence penalty: \displaystyle\max_{d\geq 0}\; \displaystyle\sum_{s,a}d(s,a)r(s,a)-\alpha D_{f}(d||d_{D}) (1a) s.t. \displaystyle\sum_{a}d(s,a)=(1-\gamma)p_{0}(s)+\gamma(\mathcal{T}_{*}d)(s),\ \forall s (1b) where D_{f}(d||d_{D}):=\mathbb{E}_{(s,a)\sim d_{D}}\big[f\big(\frac{d(s,a)}{d_{D}(s,a)}\big)\big] denotes f -divergence. The solution d^{*} to this problem is the stationary distribution of the policy \pi^{*} maximizing the objective under Bellman flow constraints ( 1b ). OptiDICE [ 20 ] is implemented by solving the Lagrangian dual of Eq. 1 with multiplier \nu(s) for ( 1b ): \min_{\nu}\max_{w\geq 0}\;\mathcal{L}(w,\nu):=(1-\gamma)\mathbb{E}_{s_{0}\sim p_{0}}[\nu(s_{0})]+\mathbb{E}_{(s,a)\sim d_{D}}\left[-\alpha f(w(s,a))+w(s,a)e_{\nu}(s,a)\right] (2) where d is replaced with the stationary distribution correction w(s,a):=\tfrac{d(s,a)}{d_{D}(s,a)} to accommodate the offline dataset and e_{\nu}(s,a):=r(s,a)+\gamma\mathbb{E}_{s^{\prime}}[\nu(s^{\prime})]-\nu(s) (derivation in Section A.1 ). Substituting w with its closed-form solution w^{*}_{\nu}(s,a)=\max\big(0,(f^{\prime})^{-1}\big(\tfrac{e_{\nu}(s,a)}{\alpha}\big)\big) gives: \min_{\nu}\;(1-\gamma)\mathbb{E}_{s_{0}\sim p_{0}}[\nu(s_{0})]+\alpha\mathbb{E}_{d_{D}}\left[f^{*}_{0}\big(\tfrac{e_{\nu}(s,a)}{\alpha}\big)\right] (3) where f^{*}_{0}(y):=\max_{x\geq 0}xy-f(x) is a convex conjugate of f in \mathbb{R}^{+} . A notable benefit of optimizing for w^{*} is its applicability to OPE for any reward function, expressed as \hat{\rho}(\pi)=\mathbb{E}_{(s,a)\sim d_{D}}[w^{*}_{\nu^{*}}(s,a)r(s,a)] . After obtaining \nu^{*} , a policy \pi^{*} inducing the stationary distribution w^{*}_{\nu^{*}}(s,a)d_{D}(s,a) must be extracted. For continuous action spaces without an analytical solution, weighted behavior cloning is commonly adopted by minimizing: \displaystyle\min_{\pi_{\theta}}\;-\mathbb{E}_{(s,a)\sim d_{D}}[w^{*}_{\nu^{*}}(s,a)\log\pi_{\theta}(a|s)] (4) Despite this elegant formulation, OptiDICE’s empirical performance in large and continuous domains lag behind compared to other value-based offline RL algorithms [ 30 ] . Semi-gradient Optimization To improve OptiDICE’s performance, semi-gradient variants have been explored [ 37 , 30 ] . This approach is a natural extension, inspired by fitted Q-iteration [ 5 ] and motivated by the resemblance of the the residual e_{\nu}(s,a) in Eq. 3 to the Bellman error [ 40 ] . Prior semi-gradient methods [ 37 , 30 ] typically involved three modifications: (1) omitting the gradient from the next state \nu(s^{\prime}) in e_{\nu}(s,a) , (2) replacing the initial state distribution p_{0}(s) with the dataset distribution d_{D} , and (3) introducing a temperature hyperparameter \beta (often removing \alpha ). Modification (3) is primarily a design choice and not directly tied to the core semi-gradient process. To isolate and simplify the analysis of semi-gradient optimization, we adopt a semi-gradient algorithm that incorporates only modifications (1) and (2), specifically by entirely omitting the gradient of \nu(s^{\prime}) and replacing p_{0} with d_{D} . Introducing a function approximator Q , the resulting objectives are: \displaystyle\min_{\nu}\; \displaystyle\mathbb{E}_{(s,a)\sim d_{D}}\big[\nu(s)+\alpha f^{*}_{0}\big(\tfrac{Q(s,a)-\nu(s)}{\alpha}\big)\big] (5) \displaystyle\min_{Q}\; \displaystyle\mathbb{E}_{(s,a,s^{\prime})\sim d_{D}}[(r(s,a)+\gamma\nu(s^{\prime})-Q(s,a)^{2}] (6) This formulation aligns with the semi-gradient optimization of Eq. 3 when Q is an exact optimum. Moreover, adopting Q removes the bias introduced by estimating the expectation within a convex function in Eq. 3 using finite samples - a limitation causing bias in OptiDICE in stochastic environments [ 20 , 14 ] . We refer to this specific algorithm as SemiDICE throughout the paper (details in Appendix B ). Based on this semi-gradient approach to DICE, Sikchi et al. 2023 , Mao et al. 2024b achieved state-of-the-art performance. Similarly, our empirical results confirm that SemiDICE exhibits strong offline RL performance consistent with those previous studies. Independently, Xu et al. 2025 proposed a similar theoretical formulation for semi-graident DICE within the context of unconstrained offline RL 1 1 1 We had independently derived the theoretical results (Propositions 4.1 , 4.2 , and

Read the complete text