What Makes Value Learning Efficient in Residual Reinforcement Learning?

Guozheng Ma, Lu Li, Haoyu Wang, Zixuan Liu, Pierre-Luc Bacon, Dacheng Tao · 2026

Read What Makes Value Learning Efficient in Residual Reinforcement Learning? by Guozheng Ma, Lu Li, Haoyu Wang, Zixuan Liu, Pierre-Luc Bacon, Dacheng Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10539.txt.

Description

Guozheng Ma and five co-authors diagnose why value learning struggles in residual reward-based learning in a 2026 paper. Residual methods freeze a pretrained policy and learn only bounded fixes, enabling stable online refinement. But the critic faces two bottlenecks the paper names and dissects.

The first is cold start: the critic knows nothing about the value landscape near the base policy. The second is scale mismatch: the residual contribution is dwarfed by the base action. Through systematic tests, the authors show simple principled fixes suffice. Base-policy transitions anchor an implicit warmup, and critic normalization restores sensitivity to small value differences. Their method, DAWN, packages both insights for fast residual value learning.

Catalog record

CREATORGuozheng Ma, Lu Li, Haoyu Wang, Zixuan Liu, Pierre-Luc Bacon, Dacheng Tao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10539

Browse the full library · About the Tower of Babel