ADORA: Training Reasoning Models with Dynamic Advantage Estimation on Reinforcement Learning

Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao · 2026

Read ADORA: Training Reasoning Models with Dynamic Advantage Estimation on Reinforcement Learning by Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10019.txt.

Description

Training reasoning models with reinforcement learning hinges on estimating advantage, the credit each action deserves. A 2026 paper led by Qingnan Ren, with six co-authors, argues that static advantage estimates waste that credit, since a sample's usefulness shifts as training proceeds. The result is slower convergence and shakier learning.

The team introduces ADORA, which reweights advantage dynamically by sorting training samples into temporarily helpful and unhelpful groups during online rollouts. The scheme plugs into existing policy optimization methods without major surgery. Across model families and data scales, ADORA improved long reasoning in geometry and mathematics tasks without fussy hyperparameter tuning.

Catalog record

CREATORQingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10019

Browse the full library · About the Tower of Babel