Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent

Yao Shu, Chenxing Wei, Hongbin Lin, Shuang Qiu, Hui Xiong · 2026

Read Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent by Yao Shu, Chenxing Wei, Hongbin Lin, Shuang Qiu, Hui Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02469.txt.

Description

Yao Shu and four co-authors clarify when offline fine-tuning can match an online reinforcement learning target for language models. In learning with verifiable rewards, the ideal updated policy tilts a reference policy toward high-reward answers. Offline methods that train on stored rollouts look simpler, but their result depends on how samples were drawn and weighted, not on rewards alone.

The paper identifies the weighted objective whose induced policy equals that ideal target. Matching forces a specific density-ratio weight, tied to how much more likely each answer is under the tilted policy. Their procedure, BOLT, estimates this projection from stored samples. A finite-sample analysis splits the error into parts from stored support, estimation, generalization, optimization, and approximation, giving readers a map of where one-shot methods lose ground.

Catalog record

CREATORYao Shu, Chenxing Wei, Hongbin Lin, Shuang Qiu, Hui Xiong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE138 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02469

Browse the full library · About the Tower of Babel