ADORA: Training Reasoning Models with Dynamic Advantage Estimation on Reinforcement Learning
Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao · 2026
Read ADORA: Training Reasoning Models with Dynamic Advantage Estimation on Reinforcement Learning by Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10019.txt.
Description
Training reasoning models with reinforcement learning hinges on estimating advantage, the credit each action deserves. A 2026 paper led by Qingnan Ren, with six co-authors, argues that static advantage estimates waste that credit, since a sample's usefulness shifts as training proceeds. The result is slower convergence and shakier learning.
The team introduces ADORA, which reweights advantage dynamically by sorting training samples into temporarily helpful and unhelpful groups during online rollouts. The scheme plugs into existing policy optimization methods without major surgery. Across model families and data scales, ADORA improved long reasoning in geometry and mathematics tasks without fussy hyperparameter tuning.
Catalog record
| CREATOR | Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, Feng Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-10019