Conditional Advantage Estimation for Reinforcement Learning in Large Rea- soning Models
Guanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao · 2025
Read Conditional Advantage Estimation for Reinforcement Learning in Large Rea- soning Models by Guanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23962.txt.
Description
Shaping reinforcement learning for reasoning models with hand-set rules about entropy or length is brittle guesswork. Guanxu Chen and 8 co-authors propose CANON, a conditional advantage estimator that refuses to presume which direction is good. Sampled responses split into two groups by a target metric's value. Inter-group comparison reveals which trend actually performs better, and the better response is picked within each group.
Built on entropy, CANON beat prior methods across three language models on math and hard logic tasks. Built on response length, it improved token efficiency and traced a better performance-cost frontier. The authors pitch direction-free conditioning as the safer default.
Catalog record
| CREATOR | Guanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23962