Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu, Siyang Gao, Shuang Qiu · 2026
Read Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting by Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu, Siyang Gao, Shuang Qiu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08118.txt.
Introduction
Qiao and colleagues improve model-based offline reinforcement learning, where a learned dynamics model helps explore policies without fresh environment data. Model errors can invite model exploitation, in which the policy leans on flaws in the model and performance degrades. Adversarial model learning counters this with a maximin formulation, and RAMBO is its best-known practical method. Yet the authors find severe Q-value underestimation in RAMBO's implementation. They answer with robust value-aware model learning plus implicitly differentiable adaptive weighting, tying model updates to value accuracy.
Catalog record
| CREATOR | Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu, Siyang Gao, Shuang Qiu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88989 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08118
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu, Yao Shu, Siyang Gao, Shuang Qiu Abstract Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, model exploitation could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO ( 42 ) has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose RO bust value-aware M odel learning with I mplicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git . 1 Introduction Model-based offline RL ( 55 ; 19 ; 42 ; 37 ; 38 ) aims to enhance offline RL ( 24 ; 35 ) by learning an environmental dynamics model in which the policy could explore. Compared with model-free offline RL ( 23 ; 31 ; 18 ; 11 ; 52 ; 39 ) , model-based offline RL exhibits higher data efficiency and could generalize better to states not present in the dataset. However, model-based offline RL suffers from model exploitation ( 19 ) , that is, the policy might explore the regions where the model cannot accurately predict the dynamics. Utilizing transitions from these inaccurate regions could degrade the algorithm’s performance. Therefore, incorporating conservatism (a.k.a. pessimism ) is necessary for model-based offline RL to mitigate model exploitation. One mainstream approach is uncertainty estimation with various heuristics ( 55 ; 19 ; 44 ; 20 ) . However, uncertainty estimation could be unreliable for neural networks ( 42 ; 54 ; 30 ) . Adversarial model learning ( 47 ) offers another theoretical framework to mitigate model exploitation by solving a maximin formulation, without uncertainty estimation. Among its practical implementations ( 42 ; 4 ; 53 ) , RAMBO ( 42 ) has emerged as a representative and widely adopted method, which (1) forces the dynamics model to reduce the value function in OOD regions by minimizing the adversarial loss with model gradient , (2) introduces a tradeoff coefficient \lambda to the adversarial term to balance the adversarial training and maximum likelihood estimation (MLE). Although RAMBO provides a practical implementation for adversarial model learning, we observe that the tradeoff coefficient \lambda is set to an extremely small value (e.g., 3e-4) in its implementation, minimizing the effect of the adversarial term. In our further investigation, we empirically find that a slightly larger yet still small \lambda (e.g., 0.05, 0.1) could cause severe Q-value underestimation and even gradient explosion, resulting in training collapse. This indicates that RAMBO has difficulties in controlling the conservatism level and can be unstable during training. To address these issues, we propose RO bust value-aware M odel learning with I mplicitly differentiable adaptive weighting (ROMI). In contrast to RAMBO, ROMI discards the model gradient method that could be overly conservative and unstable, and introduces a novel robust value-aware model learning ( 9 ; 48 ) approach to incorporate conservatism. Specifically, we require the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, and adjusting the scale of the uncertainty set is equivalent to controlling the degree of conservatism. This offers flexible and controllable conservatism adjustment and stabilizes model update. However, this value-aware learning scheme solely enforces value awareness and overlooks dynamics awareness that crucially benefits OOD generalization. To close this gap, we further propose implicitly differentiable adaptive weighting, which achieves both dynamics and value awareness in a bi-level optimization framework. Concretely, the inner level incorporates an adaptive weighting network that re-weighs each training sample in an MLE loss, while the outer level updates the weighting network by minimizing the robust value-aware model loss with implicit differentiation. This hierarchical framework adaptively balances dynamics and value-aware model learning. Lastly, we empirically evaluate ROMI on various D4RL ( 10 ) and NeoRL ( 40 ) datasets. Our experimental results demonstrate that ROMI significantly surpasses RAMBO, matching or surpassing state-of-the-art (SOTA) methods such as MOBILE ( 44 ) and Count-MORL ( 20 ) , especially on datasets where RAMBO typically underperforms. 2 Background RL. We consider a Markov Decision Process (MDP) ( 36 ) that can be specified by a tuple \langle\mathcal{S},\mathcal{A},T,r,\rho,\gamma\rangle , where \mathcal{S} and \mathcal{A} are the state space and action space, respectively, T:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) is the transition dynamics, and \Delta(\cdot) is the probability simplex. r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function, \rho is the initial state distribution, and \gamma\in[0,1) is the discount factor. The goal of RL is to obtain a policy \pi_{\theta} which maximizes the objective function: J(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0}\sim\rho\right] . Model-based Offline RL. In offline RL, the agent only has access to a pre-collected offline dataset: \mathcal{D}=\{(s_{i},a_{i},r_{i},s_{i+1})\}_{i=1}^{N} . Model-based offline RL typically trains a dynamics model to augment the dataset. The policy could perform h -step rollouts within the dynamics model starting from states in \mathcal{D} , and store the collected samples in \mathcal{D}_{\mathrm{model}} . However, the dynamics model could deviate from the true dynamics. Therefore, conservatism is necessary to prevent model exploitation. Adversarial Model Learning for Model-based Offline RL. 47 first proposes a theoretical framework that incorporates adversarial model learning into model-based offline RL to introduce conservatism. The offline RL problem is cast as a zero-sum game against an adversarial dynamics model within an uncertainty set, and thus the object is to find a policy \pi from the policy class \Pi by solving the maximin problem: \displaystyle\begin{aligned} &\pi=\arg\max_{\pi\in\Pi}{\min_{\widehat{T}\in\mathcal{M}_{\xi}}V^{\pi}_{\widehat{T}}}\\ &\text{s.t.}~~\mathcal{M}_{\xi}=\big\{\widehat{T}\mid\mathbb{E}_{(s,a)\sim\mathcal{D}}\big[\text{Dis}\big(\widehat{T}_{\mathrm{MLE}}(\cdot|s,a),\widehat{T}(\cdot|s,a)\big)\big]\leq\xi\big\},\end{aligned} (1) where \mathcal{M}_{\xi} is the dynamics uncertainty set and \text{Dis}(\cdot,\cdot) is the distributional discrepancy measure, \widehat{T}_{\text{MLE}} represents the dataset dynamics learned with maximum likelihood estimation (MLE), i.e, \widehat{T}_{\mathrm{MLE}}=\argmax_{T}\mathbb{E}_{(s,a,s^{\prime})\in\mathcal{D}}[\log T(s^{\prime}|s,a)] . While this offline RL framework offers strong theoretical guarantees, its practical algorithm remains to be explored. A widely-adopted implementation of this framework is RAMBO ( 42 ) , which optimizes the following objective: \pi=\arg\max_{\pi\in\Pi}\min_{\widehat{T}}\Big\{\lambda V^{\pi}_{\widehat{T}}-\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\big[\log\widehat{T}(s^{\prime}|s,a)\big]\Big\}, (2) where the first term is the adversarial loss that corresponds to minimizing the value in OOD regions, optimized via model gradient, and the second term relates to minimizing the negative log-likelihood on the offline data, which is a standard MLE. Here \lambda is treated as a Lagrange multiplier but, for practicality, is hand-tuned and fixed per task. 3 Motivation Example: the Limitations of RAMBO Figure 1: Left: Q-value estimation for different values