GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning

Chang-Hwan Lee, Chanseung Lee · 2025

Read GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning by Chang-Hwan Lee, Chanseung Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17034.txt.

Introduction

Chang-Hwan Lee and Chanseung Lee present GB-DQN, a gradient-boosted version of Deep Q-Networks for reinforcement learning in changing environments. When dynamics or rewards drift, a single value network goes stale and forgets, a failure called catastrophic forgetting.

Instead of retraining one network, GB-DQN grows an additive team. Each new learner fits the Bellman residual of the current team after drift. The authors prove each boosting step cuts the empirical residual, and the team converges to the post-drift best value function under standard terms. Across control tasks with set drift, GB-DQN recovered faster and held steadier than DQN and common non-stationary baselines.

Catalog record

CREATORChang-Hwan Lee, Chanseung Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17034

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning Chang-Hwan Lee, Chanseung Lee Abstract Non-stationary environments pose a fundamental challenge for deep reinforcement learning, as changes in dynamics or rewards invalidate learned value functions and cause catastrophic forgetting. We propose Gradient-Boosted Deep Q-Networks (GB-DQN) , an adaptive ensemble method that addresses model drift through incremental residual learning. Instead of retraining a single Q-network, GB-DQN constructs an additive ensemble in which each new learner is trained to approximate the Bellman residual of the current ensemble after drift. We provide theoretical results showing that each boosting step reduces the empirical Bellman residual and that the ensemble converges to the post-drift optimal value function under standard assumptions. Experiments across a diverse set of control tasks with controlled dynamics changes demonstrate faster recovery, improved stability, and greater robustness compared to DQN and common non-stationary baselines. keywords Reinforcement learning, DQN, Gradient Boosting, Non-stationary reinforcement learning 1 Introduction Deep reinforcement learning (RL) has achieved remarkable success across a wide range of sequential decision-making and control problems by combining value-based methods with expressive function approximators. In particular, Deep Q-Networks (DQN) have become a standard baseline for learning effective policies directly from high-dimensional observations. However, most deep RL algorithms rely on a critical and often unrealistic assumption: the environment remains stationary throughout training and deployment. In many real-world settings, this assumption fails to hold. Changes in physical parameters, system dynamics, sensor characteristics, or task objectives induce model drift , under which the transition kernel or reward function of the underlying Markov decision process (MDP) evolves over time. When such drift occurs, the Bellman operator itself changes, rendering value functions learned under earlier dynamics systematically biased and often leading to abrupt performance degradation, slow recovery, or catastrophic forgetting. Existing approaches to non-stationary reinforcement learning typically address this problem through retraining, hard resets, or recency-biased learning. Standard DQN updates overwrite previously learned representations as new data arrives, causing destructive interference when regimes change. Reset-based methods mitigate this interference by periodically reinitializing network parameters, but they discard all accumulated knowledge and are highly sample-inefficient. Sliding-window replay and recency-based sampling emphasize recent experience to track changing dynamics, yet they inevitably forget earlier regimes and struggle when drift is gradual or recurrent. Ensemble-based methods reduce variance by averaging multiple value functions, but when all ensemble members are trained on the same non-stationary target, they drift together and fail to correct systematic bias induced by environmental changes. In this paper, we propose Gradient-Boosted Deep Q-Networks (GB-DQN) , a principled ensemble framework that addresses non-stationarity by reframing environment drift as a sequence of Bellman residual correction problems. Rather than retraining or resetting a monolithic Q-network, GB-DQN constructs an additive ensemble of value-function approximators. Each new learner is trained to approximate the Bellman residual of the current ensemble after a change in dynamics, while previously learned components are frozen and preserved. This design directly mitigates catastrophic forgetting and enables targeted adaptation, since new learners correct only the discrepancies introduced by drift rather than relearning the value function from scratch. The proposed formulation is inspired by gradient boosting in supervised learning, which builds strong predictors by sequentially fitting weak learners to residual errors. Applied to reinforcement learning, this perspective provides a unifying interpretation of value-function adaptation as functional gradient descent on the Bellman error. We show theoretically that each boosting step strictly reduces the empirical Bellman residual whenever the new learner is not orthogonal to the residual signal. Under standard contraction assumptions and a sufficiently expressive hypothesis class, repeated residual fitting guarantees convergence to the optimal value function in stationary environments. Moreover, after a change in dynamics, successive residual corrections converge to the optimal value function of the post-drift MDP, establishing both stability and adaptability of the boosted ensemble. To support these claims empirically, we evaluate GB-DQN on a suite of benchmark control tasks under controlled non-stationary dynamics. Across environments with varying reward structure and complexity, including sparse-reward and contact-rich domains, GB-DQN consistently exhibits faster post-drift recovery, smaller performance drops at regime changes, and lower variance compared to standard DQN, static ensembles, reset-based methods, and sliding-window replay. These results demonstrate that residual-based ensemble augmentation yields systematic improvements that cannot be explained by variance reduction or recency bias alone. Overall, this work makes three key contributions. First, it introduces a new perspective on non-stationary reinforcement learning that treats model drift as a residual learning problem over evolving Bellman operators. Second, it proposes GB-DQN, an adaptive and computationally efficient ensemble method that preserves prior knowledge while enabling rapid adaptation. Third, it provides both theoretical guarantees and empirical evidence demonstrating that gradient-boosted value-function ensembles offer a robust and scalable foundation for reinforcement learning in dynamic, non-stationary environments. 2 Related Work In many real-world reinforcement learning (RL) applications, the assumption of a stationary environment does not hold. Changes in system dynamics, external conditions, or task objectives induce model drift (or concept drift ), causing previously learned value functions to become suboptimal. Such non-stationarity poses a major challenge for value-based methods such as DQN, which rely on fixed transition and reward functions. Classic approaches detect change points in observed state transitions or reward distributions using statistical tests (e.g., CUSUM, Page–Hinckley) and trigger policy retraining ( 14 ; 7 ). In the context of RL, methods like Contextual MDPs maintain separate models for different environment regimes, switching policies upon detecting latent context shifts ( 8 ). More recent meta‐learning solutions aim to quickly adapt a single policy to new tasks via few‐shot gradient updates ( 5 ). While effective, these methods often require extensive offline computation or pre‐collected context labels, whereas our approach dynamically grows an ensemble online with minimal overhead. Early work on ensemble‐based RL demonstrated that maintaining a collection of diverse Q‐networks can reduce estimation variance and improve policy stability. Dietterich first formalized the notion of ensemble learning in the context of Q‐learning, showing that policy aggregation across multiple learners mitigates overfitting to spurious value estimates ( 2 ). More recent advances such as Bootstrapped ensemble DQN ( 13 ) maintain multiple value function heads trained on different bootstrap samples of the replay buffer, enabling deep exploration through approximate posterior sampling. Despite these benefits, static ensembles lack mechanisms to incorporate new information when all base learners have become outdated, motivating our focus on adaptive ensemble augmentation. Averaged-DQN ( 1 ) further formalizes ensemble learning for DQN by averaging predictions from multiple independently trained networks, showing that ensembling reduces variance and stabilizes learning dynamics. While effective under stationary settings, ensemble methods alone struggle when environment dynamics shift, since older ensemble members may encode outdated transition models. Reset-based methods have been proposed to mitigate primacy bias, a phenomenon where early training data disproportionately influences long-term policy behavior. 12 demonstrate that periodically resetting network parameters while retaining the replay buffer significantly improves performance and stability, especially in long-horizon or non-stationary environments. This approach, often referred to as Reset-DQN, enables rapid re-optimization under changing conditions but sacrifices accumulated representational knowledge at each reset. More recent work extends this idea to ensembles, where individual ensemble members are reset sequentially to balance adaptation and safety ( 10 ). Despite their effectiveness, reset-based methods can introduce abrupt performance fluctuations and do not explicitly exploit residual structure across regimes.

Read the complete text