Conditional Advantage Estimation for Reinforcement Learning in Large Rea- soning Models

Guanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao · 2025

Read Conditional Advantage Estimation for Reinforcement Learning in Large Rea- soning Models by Guanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23962.txt.

Description

Shaping reinforcement learning for reasoning models with hand-set rules about entropy or length is brittle guesswork. Guanxu Chen and 8 co-authors propose CANON, a conditional advantage estimator that refuses to presume which direction is good. Sampled responses split into two groups by a target metric's value. Inter-group comparison reveals which trend actually performs better, and the better response is picked within each group.

Built on entropy, CANON beat prior methods across three language models on math and hard logic tasks. Built on response length, it improved token efficiency and traced a better performance-cost frontier. The authors pitch direction-free conditioning as the safer default.

Catalog record

CREATORGuanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian, Qihan Ren, JingYi Yang, Yu Cheng, Dongrui Liu, Jing Shao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23962

Browse the full library · About the Tower of Babel