Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning
Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California · 2026
Read Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning by Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06241.txt.
Description
Ömer Faruk Akgül and three co-authors argue that reinforcement learning improves language model reasoning by selection, not by teaching new capability. Evidence suggests such training shifts probability toward solutions the base model already contains. The authors ask whether the expensive optimization loop itself is needed, if its whole effect is a small, predictable steering of existing behavior.
Token level analysis finds the benefit concentrated at high entropy decision points, touching only one to three percent of positions. The promoted token always sits among the base model's top five alternatives there. Their RL free method, ReasonMaxxer, applies contrastive loss only at those gated points using a few hundred rollouts. It matched or beat full reinforcement learning across model families and math benchmarks at a tiny fraction of the cost.
Catalog record
| CREATOR | Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06241