Learning Dynamics in RL Post-Training for Language Models
Akiyoshi Tomihari · 2026
Read Learning Dynamics in RL Post-Training for Language Models by Akiyoshi Tomihari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04670.txt.
Description
Akiyoshi Tomihari examines why reinforcement learning makes language models more certain and less varied, in a 2026 arXiv paper. RL post-training is where modern models gain alignment and reasoning polish, but its side effect — outputs collapsing toward a few safe patterns — is poorly understood. He studies the training dynamics through the empirical neural tangent kernel, a tool borrowed from supervised learning theory. Splitting it into two components to trace how an update on one example propagates to others.
The analysis shows that when feature representations vary little across samples, RL updates systematically push confidence up everywhere at once, which mechanically explains the diversity loss. Effective learning then hinges on shaping the classifier head of the network quickly. His remedy, classifier-first RL, prioritizes those updates before plain RL begins; experiments confirm faster optimization and the predicted confidence pattern, through a mechanism distinct from the analogous guided trick.
Catalog record
| CREATOR | Akiyoshi Tomihari |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04670