Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, Sheng-Jun Huang · 2025
Read Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL by Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, Sheng-Jun Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19923.txt.
Introduction
Learning from a fixed log of past actions, offline reinforcement learning must beware of trusting values for moves the log never tried. Qin-Wen Luo and three co-authors note that the usual fix, holding the learner back everywhere with one strength, backfires. Too weak lets fantasy values creep in; too strong reduces learning to copying the log.
Their method adapts the restraint state by state, trusting Bellman updates where the data is good and clamping down selectively on strong actions elsewhere. They connect the idea to two mainstream approaches and extend it to both. On the standard D4RL benchmark, the method beat leading rivals in both pure offline tests and offline-to-online fine-tuning. The paper appears in 2025.
Catalog record
| CREATOR | Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, Sheng-Jun Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, Sheng-Jun Huang Abstract Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset. To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all states. However, due to substantial variations in data quality, the fixed regularization strength often leads to a dilemma: Weak regularization strength fails to address extrapolation errors and value overestimation, while strong regularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates. To address this issue, we propose the selective state-adaptive regularization method for offline RL. Specifically, we introduce state-adaptive regularization coefficients to trust state-level Bellman-driven results, while selectively applying regularization on high-quality actions, aiming to avoid performance degradation caused by tight constraints on low-quality actions. By establishing a connection between the representative value regularization method, CQL, and explicit policy constraint methods, we effectively extend selective state-adaptive regularization to these two mainstream offline RL approaches. Extensive experiments demonstrate that the proposed method significantly outperforms the state-of-the-art approaches in both offline and offline-to-online settings on the D4RL benchmark. The implementation is available at https://github.com/QinwenLuo/SSAR . † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) has made remarkable strides across diverse domains ( Degrave et al. 2022 ; Kaufmann et al. 2023 ; Haarnoja et al. 2024 ) . However, RL typically improves performance through trial and error, which limits its practical application to critical scenarios, such as healthcare decision-making ( Tang & Wiens 2021 ; Fatemi et al. 2022 ) and autonomous driving ( Fang et al. 2022 ; Diehl et al. 2023 ) . These tasks often have such strict requirements regarding potential risks and time costs that the interaction with the environment is limited or inaccessible. Offline RL has mitigated this problem by deriving policies from static datasets without actual environment interactions. Direct application of the off-policy algorithms could result in value overestimation and extrapolation errors ( Fujimoto et al. 2019 ; Kumar et al. 2019 ) . To address these challenges, based on how they resolved around the policy iteration process, existing RL-based offline methods can be roughly divided into two groups, value regularization ( Kumar et al. 2020 ; Cheng et al. 2022 ; Nakamoto et al. 2024 ) and policy constraint ( Fujimoto & Gu 2021 ; Kostrikov et al. 2021b ; Nair et al. 2020 ) . The former suppresses the Q-values of out-of-distribution (OOD) actions in the policy evaluation function to alleviate value overestimation; while the latter restricts updates near the dataset in the policy improvement function to avoid OOD actions. Both approaches incorporate regularization terms into their online counterparts, shifting the update from an optimistic to a pessimistic manner. Despite their demonstrated advances, existing methods still face the critical challenge of identifying optimal regularization strengths that balance trust in the outcomes derived from Bellman updates and the imitation of dataset actions to fully unlock the potential of RL-based learning. Given that in model-free RL methods, performance gains beyond the behavior policy are fundamentally enabled by Bellman updates, it is crucial to balance confidence in these updates and conservative regularization. Since the balance is typically achieved by the fixed global coefficient in the regularization term, identifying its optimal value is paramount for effective performance. This presents challenges from three key aspects. Firstly, from a task-level perspective, the optimal global coefficient can vary significantly across different tasks. Certain tasks may benefit from a larger coefficient to enforce stricter regularization and avoid overestimation, while others may require a lower coefficient to enable more flexible policy updates, resulting in better performance ( Nakamoto et al. 2024 ; Lyu et al. 2022 ) . Secondly, from the perspective of training dynamics, the optimal coefficient should evolve throughout the training process. When the policy is less reliable and poorly constrained by the dataset at the early stage, a larger coefficient is essential to restrict policy updates. As the training advances and the distribution of the learned policy becomes sufficiently close to the dataset, a lower coefficient may be more beneficial for states where the policy actions are close to the dataset, thanks to the reliable and reasonable generalization enabled by Bellman updates ( Beeson & Montana 2022 ) . Thirdly, considering the variability in data densities, it is intuitive to trust the Q-values learned by Bellman updates for states with high data density, while conversely tightening the constraints for states with low data density. Despite some nuances, existing methods predominantly determine the global coefficient through extensive experiments, which struggles to address these challenges effectively. Another critical issue with the fixed global regularization lies in its impact on the efficiency of transitioning from offline-to-online (O2O) RL. The uniform regularization strength presents a dilemma: low regularization coefficients may amplify the risks of extrapolation errors through insufficient behavioral constraints, yielding poorly initialized policies. Conversely, overly high regularization coefficients induce significant discrepancies between the offline and online Q-values in value regularization methods, as well as misalignment between the learned policy and the policy directly induced by the critic in policy constraint methods. To enhance the efficiency of O2O RL, researchers have explored various approaches, including gradually relaxing constraint terms ( Beeson & Montana 2022 ; Zhao et al. 2022 ) and adopting alternative action sampling strategies ( Zhang et al. 2023 ; Uchendu et al. 2023 ) . However, these methods make modifications based on global coefficients, which prevents further improvement in fine-tuning efficiency. In this work, we propose state-adaptive regularization that dynamically quantifies the reliability of Bellman updates, guiding the policy to trust optimistic outcomes at the state level. Instead of a fixed coefficient, our method employs a neural network as a state-dependent coefficient generator and adaptively updates it to adjust regularization strengths based on the discrepancy between the dataset and the distribution of the learned policy. To enhance its universality, we establish a connection between CQL, a representative value regularization method, and explicit policy constraint methods, extending it to both popular offline approaches. Furthermore, we propose a selective regularization method by selecting a subset of data with high rewards. This allows the policy update to focus on high-quality actions and avoid excessive constraints on low-quality parts. With the integration of selective state-adaptive regularization, we effectively bridge the gap between offline and online RL, achieving efficient O2O RL through simple linear coefficient annealing. 2 Preliminaries Offline RL The environment in RL is typically modeled as a Markov Decision Process (MDP), which is defined by the tuple ( S,A,R,P, \mu , \gamma ) ( Sutton & Barto 2018 ) , where S is the state space, A is the action space, \emph{P}:\emph{S}\times\emph{A}\rightarrow\Delta(\emph{S}) is the transition function, \emph{R}:\emph{S}\times\emph{A}\rightarrow\mathbb{R} is the reward function, \mu is the initial state distribution and \gamma is a discount factor. The goal of RL is to find a policy \pi:\emph{S}\rightarrow\Delta(\emph{A}) that maximizes the expected discounted return: \mathbb{E}_{s_{0}\sim\mu,a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P(\cdot|s_{t},a_{t})}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right] . For any policy \pi , we define the state value function as V^{\pi}(s)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})|s_{0}=s] and the state-action value function as Q^{\pi}(s)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})|s_{0}=s,a_{0}=\pi(\cdot|s_{0})] . The agent interacts with the environment by observing states, taking actions and receiving rewards, and improves the policy through the interactive data. In offline