Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

Ian Wu, Yuxiao Qu, Amrith Setlur, Aviral Kumar · 2026

Read Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL by Ian Wu, Yuxiao Qu, Amrith Setlur, Aviral Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03773.txt.

Description

Ian Wu and 3 co-authors ask whether a model can keep improving far past the horizon it was trained on. Their 2026 paper introduces Reasoning Cache, an iterative decoding algorithm that replaces plain autoregressive decoding in both training and inference. Standard reinforcement learning pins the model to fixed problem distributions and budgets, which limits such extrapolation.

The method exploits an asymmetry: models summarize better than they generate. Reasoning chains are built that improve consistently across iterations, each round conditioning on a summary of the last. A four-billion-parameter model trained with a sixteen-thousand-token budget climbs from forty to nearly seventy percent. The benchmark is a hard mathematics test, and the climb uses half a million test-time tokens. It also stacks well with existing scaffolds.

Catalog record

CREATORIan Wu, Yuxiao Qu, Amrith Setlur, Aviral Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE125 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03773

Browse the full library · About the Tower of Babel