Learning Dynamics in RL Post-Training for Language Models

Akiyoshi Tomihari · 2026

Read Learning Dynamics in RL Post-Training for Language Models by Akiyoshi Tomihari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04670.txt.

Description

Akiyoshi Tomihari examines why reinforcement learning makes language models more certain and less varied, in a 2026 arXiv paper. RL post-training is where modern models gain alignment and reasoning polish, but its side effect — outputs collapsing toward a few safe patterns — is poorly understood. He studies the training dynamics through the empirical neural tangent kernel, a tool borrowed from supervised learning theory. Splitting it into two components to trace how an update on one example propagates to others.

The analysis shows that when feature representations vary little across samples, RL updates systematically push confidence up everywhere at once, which mechanically explains the diversity loss. Effective learning then hinges on shaping the classifier head of the network quickly. His remedy, classifier-first RL, prioritizes those updates before plain RL begins; experiments confirm faster optimization and the predicted confidence pattern, through a mechanism distinct from the analogous guided trick.

Catalog record

CREATORAkiyoshi Tomihari
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04670

Browse the full library · About the Tower of Babel