Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn · 2026
Read Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities by Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05812.txt.
Introduction
Q-learning learns from any experience, but its bootstrapping lets small errors compound across long horizons. Armaan A. Abraham, Lucy Xiaoyang Shi, and Chelsea Finn add a backstop: n-step inequalities. Any action sequence actually realized lower-bounds what the optimal policy can achieve, so value estimates can be tightened against that fact.
Their long-horizon Q-learning uses those inequalities to stop estimation errors from propagating backward unchecked. The method keeps the appeal of off-policy learning from old or borrowed data. It targets the settings where temporal-difference learning is most brittle: problems whose payoffs arrive far in the future.
Catalog record
| CREATOR | Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn Abstract Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime. 1 Introduction Figure 1 : LQL with long trajectories scales to the longest task in OGBench; n -step TD degrades as n grows. Sparse-reward humanoidmaze-giant (all tasks) with Best-of- N policies. TD- n at increasing n and LQL with trajectory length 64 . Off-policy reinforcement learning holds the promise of turning past experience into future competence: by learning a value function, an agent can improve from data collected by older policies, other agents, or imperfect exploration, without requiring fresh on-policy rollouts at every update ( Watkins and Dayan 1992 ; Sutton and Barto 1998 ) . This promise is particularly compelling in domains where interaction is expensive and long-horizon behavior matters (e.g., robotics), where we would like to extract as much learning signal as possible from each transition. Classic value-based methods such as Q-learning do exactly this by bootstrapping : they train a Q -function so that the utility of a state-action pair agrees with the immediate reward plus a discounted estimate of what the learned policy could achieve next, i.e., a counterfactual continuation under the policy being learned. However, bootstrapping over long horizons is fragile in practice because estimation errors can amplify as they propagate backward through time ( Jaakkola et al. 1993 ; Sutton and Barto 1998 ; Asis et al. 2018 ; Park et al. 2025b ) . A common mitigation is to reduce reliance on the next-state estimate by using multi-step targets (e.g., n -step or \lambda -returns), replacing a 1-step backup with a longer segment of observed rewards before switching back to a bootstrapped value estimate ( Sutton and Barto 1998 ; Hessel et al. 2017 ; Asis et al. 2018 ) . Yet this remedy introduces its own tension in off-policy settings: an n -step target for Q(s_{t},a_{t}) necessarily incorporates the rewards generated by the logged actions a_{t+1},\ldots,a_{t+n-1} that followed in the replayed trajectory. As a result, even if a_{t} is a high-quality decision, the resulting backup can be pessimistic when the subsequent recorded actions are low-quality. Put differently, multi-step returns partially evaluate (s_{t},a_{t}) under the incorrect assumption that the agent will continue to follow the behavior that produced the trajectory for several more steps, rather than switching immediately to the (improving) policy being learned. Figure 2 : LQL establishes a backstop against compounding TD error over time. Standard 1-step TD can amplify estimation errors as they propagate backward through bootstrap updates (top). LQL’s long-horizon constraints provide additional correction signals that bound these inconsistencies across multiple steps (bottom). This paper asks: can we keep the simplicity and low-variance of 1-step off-policy TD learning, while introducing a principled long-horizon backstop that detects and corrects the kinds of inconsistencies that lead to compounding error? Our starting point is a standard consequence of optimality—and one previously leveraged for “optimality tightening”—that any realized sequence of actions provides a lower bound on what the optimal policy could achieve in expectation from the same starting state ( He et al. 2016 ) . Concretely, for any trajectory of experience, the optimal value of the initial decision cannot be worse (in expected return) than committing to the observed actions for a while and only then behaving optimally. We refer to this constraint as an optimality inequality over trajectories. Motivated by this inequality, we introduce a training objective that augments the standard TD error with hinge penalty terms that enforce long-horizon consistency asymmetrically. Intuitively, if a value estimate at some state-action is below the return realized by an actual action sequence, the estimate is pushed upward; conversely, if combining a later value estimate with the observed actions leading up to it would imply doing better than acting optimally earlier, the later estimate is pushed downward. These hinge losses provide a trajectory-level correction signal, but can be computed from the same Q -values already used in standard TD updates. Our main contribution is long-horizon Q-learning (LQL), a general off-policy value learning approach that is agnostic to the policy extraction mechanism. LQL samples trajectories from the replay buffer and constructs hinge penalties using only network outputs already computed within a typical TD update (e.g., Q(s_{t},a_{t}) and the bootstrapping value at the next state), requiring no auxiliary networks and no additional forward passes per update compared to Q-learning. Empirically, across locomotion and robot manipulation tasks from OGBench ( Park et al. 2025a ) and RoboMimic ( Mandlekar et al. 2021 ) , and across multiple ways of extracting a policy from the learned value function, LQL improves over TD learning while incurring minor additional runtime cost per training iteration. 2 Related work Off-policy learning. Off-policy value-based RL (e.g., Q-learning) can in principle learn from arbitrary experience ( Watkins and Dayan 1992 ; Sutton and Barto 1998 ; Mnih et al. 2015 ) , but is often unstable in practice. This is commonly attributed to the deadly triad —off-policy learning, bootstrapping, and function approximation—which causes TD errors to propagate and amplify over long horizons ( Baird 1995 ; Tsitsiklis and Van Roy 1996 ; Sutton and Barto 1998 ; van Hasselt et al. 2018 ; Park et al. 2025b ) . LQL addresses this issue by augmenting TD learning with a temporally extended consistency backstop that penalizes value inconsistencies across temporally distant states. Multi-step backups. Multi-step backups, such as n -step returns and \lambda -returns, accelerate reward propagation and reduce sensitivity to compounding 1-step TD error ( Sutton 1988 ; Watkins 1989 ; Peng and Williams 1996 ; Sutton and Barto 1998 ; Mnih et al. 2016 ; Hessel et al. 2017 ; Asis et al. 2018 ; Schulman et al. 2018 ; Chebotar et al. 2023 ; Schwarzer et al. 2023 ; Daley et al. 2025 ) . However, they define targets from logged trajectories, coupling early actions to subsequent (possibly suboptimal) behavior. LQL instead uses trajectories to impose an optimality inequality , regularizing long-horizon consistency while retaining standard TD updates. Off-policy corrections for multi-step learning. In off-policy settings, multi-step targets are biased under policy mismatch ( Sutton and