Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California · 2026

Read Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning by Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06241.txt.

Description

Ömer Faruk Akgül and three co-authors argue that reinforcement learning improves language model reasoning by selection, not by teaching new capability. Evidence suggests such training shifts probability toward solutions the base model already contains. The authors ask whether the expensive optimization loop itself is needed, if its whole effect is a small, predictable steering of existing behavior.

Token level analysis finds the benefit concentrated at high entropy decision points, touching only one to three percent of positions. The promoted token always sits among the base model's top five alternatives there. Their RL free method, ReasonMaxxer, applies contrastive loss only at those gated points using a few hundred rollouts. It matched or beat full reinforcement learning across model families and math benchmarks at a tiny fraction of the cost.

Catalog record

CREATORÖmer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna, [5pt] University of Southern California
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06241

Browse the full library · About the Tower of Babel