Peng’s Q( λ \lambda ) for Conservative Value Estimation in Offline Reinforcement Learning

Byeongchan Kim, Min-hwan Oh · 2026

Read Peng’s Q( λ \lambda ) for Conservative Value Estimation in Offline Reinforcement Learning by Byeongchan Kim, Min-hwan Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14779.txt.

Introduction

Kim and Oh propose Conservative Peng's Q(lambda), a model-free offline reinforcement learning algorithm with multiple steps. It adapts Peng's Q(lambda) operator for conservative value estimation instead of the Bellman operator. Its fixed point is claimed to sit closer to the behaviour policy's value function, giving implicit behaviour control. It is said to curb over-pessimistic values, match or beat the behaviour policy, and offer near-optimal bounds. On D4RL, it is claimed to beat existing offline single-step baselines.

Catalog record

CREATORByeongchan Kim, Min-hwan Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116014
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14779

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Peng’s Q( λ \lambda ) for Conservative Value Estimation in Offline Reinforcement Learning Byeongchan Kim, Min-hwan Oh Abstract We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng’s Q( λ \lambda ) (CPQL). Our algorithm adapts the Peng’s Q( λ \lambda ) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with a multi-step operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees — a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL . 1 Introduction Offline RL aims to learn policies from a static dataset collected under unknown behavior policies without further interactions with the actual environment. However, offline RL faces a major challenge known as distributional shift ( 38 ) . A distributional shift arises when the state-action distribution under the learned policy diverges significantly from that under the behavior policy. This issue is exacerbated when the application of the Bellman updates to value functions requires querying values of out-of-distribution (OOD) state-action pairs, which can lead to an accumulation of extrapolation errors and ultimately result in poor performance of learned policies. To tackle OOD actions in policy evaluation, conservative Q-learning (CQL) ( 35 ) penalizes the learned Q-function for OOD actions induced by the learning policy. Building on CQL, subsequent algorithms ( 44 ; 42 ; 7 ; 48 ; 58 ; 75 ) address the potential over-pessimism in both in-distribution and OOD actions, which stems from excessively conservative value estimates. These approaches rely on additional components, such as estimating the unknown behavior policy to handle OOD actions ( 42 ; 75 ) or introducing extra networks for a quantile ( 44 ) or a state value function ( 7 ; 48 ) , which may lead to increased complexity and further drawbacks despite their intentions to address the over-pessimism — such drawbacks include distribution mismatches between the estimated behavior policy and the dataset ( 84 ; 36 ) , the need for extensive parameter tuning, and prolonged training times. Intuitively, leveraging offline trajectories that span multiple timesteps, rather than individual single-timestep transitions, provides more information about the behavior policy and can potentially prevent the selection of OOD actions for offline datasets. Although trajectories are readily available in offline datasets, most previous model-free offline RL methods for policy evaluation have utilized these trajectories only in the form of fragmented single-step transitions ( 16 ; 34 ; 69 ; 14 ; 31 ) . Hence, the following question arises: Can we design a value estimation method for offline RL that utilizes the multi-step learning ? In online RL counterparts of offline RL, there is a line of work that extends a single-step temporal-difference (TD) learning (e.g., Q-learning ( 67 ) ) to multi-step generalizations ( 51 ; 53 ; 46 ; 22 ; 57 ; 33 ) , introducing multi-step operators that leverage temporally extended trajectories to update Q-values. These operators improve learning efficiency and provide a more forward-looking perspective, leading to enhanced performance in determining optimal actions compared to the single-step Bellman operator across various benchmarks ( 21 ; 45 ; 23 ; 5 ; 27 ; 10 ) . However, whether such an extension to multi-step TD learning is possible in offline RL is still unclear. Hence, the follow-up questions arise: What is a suitable multi-step operator for offline RL? Is it possible to demonstrate that the multi-step operator enhances performance ? In this paper, we propose an effective conservative multi-step Q-learning algorithm for model-free offline RL, Conservative Peng’s Q( \lambda ) (CPQL). Our algorithm builds on conservative value estimation by incorporating the Peng’s Q( \lambda ) (PQL) operator ( 51 ; 33 ) instead of the standard Bellman operator. Unlike other multi-step operators ( 53 ; 46 ; 22 ; 57 ) that truncate trajectories, the PQL operator fully leverages whole trajectories to improve policy evaluation. Since the PQL operator does not use importance sampling, which requires estimating the behavior policy from offline datasets, it avoids the mismatch issues arising from inaccurate behavior policy estimation ( 84 ; 36 ) . Because the fixed point of the PQL operator in offline RL converges to the Q-function of a mixture policy that interpolates between the behavior policy and target policy, coupling it with a conservative value estimation method ensures that even mild conservatism is sufficient to mitigate Q-value overestimation caused by distribution shift. In contrast to other conservative methods ( 35 ; 44 ; 42 ; 7 ; 48 ; 58 ; 75 ) , CPQL mitigates over-pessimistic value estimation in the Q-function ( Theorem 1 ) without requiring additional estimation procedures or auxiliary networks. Our main contributions are summarized as follows: • We propose CPQL, the first multi-step Q-learning algorithm for model-free offline RL. CPQL adapts the PQL operator to conservative value estimation and fully leverages offline trajectories without estimating additional models. To the best of our knowledge, our work is the first to demonstrate both theoretically and empirically the effectiveness of conservative multi-step value estimation. • We provide rigorous theoretical analyses for CPQL. The mixture policy learned by CPQL is guaranteed to achieve performance greater than (or equal to) that of the behavior policy (Theorem 2 ) and further reduces the sub-optimality gap compared to CQL (Theorem 3 ). Our theoretical analyses effectively address the key limitations of over-pessimistic value estimation in offline RL, ensuring balanced conservatism and improved policy exploration. • Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step RL algorithms. In contrast to CQL, whose performance is highly sensitive to the choice of the conservatism parameter ( 2 ; 18 ; 63 ) , CPQL remains robust across a wide range of conservatism parameter values. • Beyond the contribution of CPQL to mitigating over-pessimistic value estimation in offline RL, CPQL also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL enables the online PQL agent to avoid the performance drop observed at the start of the online phase and attain robust performance improvements. 2 Related Work Model-free Offline RL. To overcome the distributional shift and extrapolation error, model-free offline RL methods focus on learning policies using techniques such as penalizing learned value functions to assign low values to unseen actions ( 35 ; 44 ; 42 ; 7 ; 48 ; 58 ; 43 ; 75 ) ,

Read the complete text