Quantile Q Q -Learning: Revisiting Offline Extreme Q Q -Learning with Quantile Regression

Xinming Gao [email protected], Shangzhe Li [email protected], Yujin Cai [email protected], Wenwu Yu [email protected] · 2025

Read Quantile Q Q -Learning: Revisiting Offline Extreme Q Q -Learning with Quantile Regression by Xinming Gao [email protected], Shangzhe Li [email protected], Yujin Cai [email protected], Wenwu Yu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11973.txt.

Introduction

Xinming Gao, Shangzhe Li, Yujin Cai, and Wenwu Yu revisit Extreme Q-Learning, an offline reinforcement learning method that learns from a fixed dataset using extreme value theory to model Bellman errors. Offline learning matters where fresh interaction is dangerous or costly, but the method and its stabilized variant demand heavy per-dataset tuning and can train unstably.

Quantile regression supplies a principled estimate of the temperature coefficient that previously had to be hand-tuned, under mild assumptions. A value regularization technique with a generalization argument further steadies training. Removing fragile knobs is what turns a clever offline algorithm into one practitioners can trust on a new dataset without a tuning expedition.

Catalog record

CREATORXinming Gao [email protected], Shangzhe Li [email protected], Yujin Cai [email protected], Wenwu Yu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11973

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantile Q Q -Learning: Revisiting Offline Extreme Q Q -Learning with Quantile Regression Xinming Gao [email protected], Shangzhe Li [email protected], Yujin Cai [email protected], Wenwu Yu [email protected] Abstract Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme Q Q -Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient β \beta via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains. 1 Introduction Deep reinforcement learning (DRL) has achieved impressive results across a broad range of domains, including navigation ( 20 ) , healthcare ( 29 ) , robotics ( 11 ) , and games ( 21 ; 25 ) . Recent advances in offline reinforcement learning (offline RL) ( 16 ; 17 ; 15 ; 9 ) have extended the capability of DRL by enabling agents to learn solely from static datasets, without requiring further interaction with the environment. This paradigm shift is particularly promising in domains where data collection is expensive, risky, or impractical. Among the recent developments, the offline version of extreme Q-learning (XQL) ( 9 ) introduces a novel perspective by modeling the Bellman error distribution using the Extreme Value Theorem (EVT), assuming it follows a Gumbel distribution. This theoretical insight leads to an in-sample learning algorithm that has demonstrated competitive performance on standard offline RL benchmarks such as D4RL. However, XQL exhibits notable instability during training, as reported in follow-up work ( 23 ) . To address this issue, MXQL ( 23 ) introduces a variant that stabilizes training through a Maclaurin-series-based approximation of the original XQL loss. Figure 1 : Comparison of XQL performance using dataset-specific tuning versus consistent hyperparameters across the domain. In some scenarios, performance degrades notably without dataset-specific tuning. Despite these efforts, several challenges remain. Both XQL and MXQL are sensitive to hyperparameter choices—particularly the temperature coefficient—which often require dataset-specific tuning to achieve optimal performance, making them impractical for real-world applications. This sensitivity can lead to significant performance degradation when hyperparameters are changed, as demonstrated in Figure 1 . Besides, even the stabilized MXQL variant can experience training instability under certain conditions (see Figure 2(a) ). These limitations motivate our central research question: Can we design an algorithm based on offline XQL that maintains stability and strong performance across diverse domains and offline datasets, using a single, consistent set of hyperparameters? To address this issue, we propose a principled method for estimating the temperature coefficient \beta via quantile regression under mild assumptions, thereby eliminating the need for dataset-specific tuning. Building on this, we introduce a value regularization approach with mild generalization, inspired by prior work on doubly constrained value learning ( 19 ) . The performance of the proposed algorithm matches or exceeds that of the state-of-the-art model-free offline RL algorithms across a variety of domains and dataset types in diverse offline RL benchmarks, including D4RL ( 4 ) and NeoRL2 ( 8 ) , without the need for domain-specific hyperparameter tuning. 2 Related Works Our work builds primarily on the literature of offline reinforcement learning (offline RL), a subfield of reinforcement learning that aims to learn optimal policies from a fixed dataset without any further interaction with the environment. A central challenge in offline RL is that many online off-policy methods tend to underperform due to extrapolation error ( 6 ) or distributional shift ( 17 ) . To address these issues, offline RL algorithms often augment standard off-policy methods with a penalty term that measures the divergence between the learned policy and the behavior policy used to collect the data ( 5 ) . Existing offline RL approaches vary in how they formulate the problem, including methods that constrain the learned policy to remain close to the behavior policy ( 31 ; 5 ; 18 ) , approaches that incorporate pessimistic value estimation to avoid overestimation of out-of-distribution actions ( 2 ; 30 ; 15 ; 16 ) , and techniques that leverage deep architectures such as large autoregressive models ( 3 ; 14 ) or generative models ( 13 ; 1 ; 28 ; 32 ) . In our work, we specifically focus on improving an existing offline RL method, Extreme- Q Learning (XQL) ( 9 ) , which is potentially unstable and sensitive to hyperparameters as discussed in the introduction. Prior work ( 23 ) proposed a method to stabilize XQL by applying a Maclaurin expansion to the original objective function. 3 Preliminaries The problem is formulated as an Markov decision process (MDP), which is represented by the tuple (\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma) , where \mathcal{S} denotes the state space, \mathcal{A} represents the action space, \mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}} characterizes the transition dynamics; r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function; \gamma\in[0,1) is the discount factor. We aim to learn a policy \pi:\mathcal{S}\rightarrow\Delta_{\mathcal{A}} to maximize cumulative discounted rewards ( 26 ) . In offline RL setting, the policy learning is conducted on a dataset \mathcal{D} consisting a set of trajectories (\tau_{1},...,\tau_{T}) . Each trajectory is composed of a group of transitions \{(s_{t},a_{t},r_{t},s^{\prime}_{t})\} . Extreme Q -Learning Extreme Q -Learning (XQL) ( 9 ) is an algorithm being capable of solving both online and offline RL tasks. It directly models the maximal value using Extreme Value Theory (EVT) with the assumption that the error in Q -functions follow a Gumbel distribution. In our work, we especially focus on the offline part of XQL. Specifically, offline XQL leverages an objective for optimizing a soft value function V over dataset \mathcal{D} : \mathcal{J}(V)=\mathbb{E}_{(s,a)\sim\mathcal{D}}\exp\left(({Q}(s,a)-V(s))/\beta\right)-({Q}(s,a)-V(s)/\beta)-1, (1) where Q is the Q -function optimized via minimizing mean-squared Bellman error and \beta is the temperature hyperparameter. Furthermore, XQL learns a policy with an advantage-weighted regression ( 24 ; 22 ) (AWR) style update. \pi^{*}=\arg\max_{\pi}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[e^{(Q(s,a)-V(s))/\beta}\log\pi\right]. (2) In practice, we find that XQL is sensitive to the hyperparameter \beta , and its optimal value varies significantly across environments and offline datasets, as shown in Appendix C.1 and Figure 1 . 4 Revisiting Extreme Q -Learning with Quantile Regression 4.1 Estimating \beta under Mild Assumptions Since the formulation of XQL is sensitive to the hyperparameter \beta , we show in this section that \beta can be generalized to a state-dependent function \beta(s) and learned under mild assumptions. We begin by stating the assumptions used in our subsequent analysis involving the estimated Q -function Q , the optimal Q -function Q^{\star} and the soft value function V . It is worth noting that these assumptions are either identical or closely aligned with those used in prior works ( 12 ; 9 ) . Assumption 1 ( ( 12 ) ) . Given a state-dependent

Read the complete text