Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

Dylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng · 2026

Read Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning by Dylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01058.txt.

Description

Dylan Zhang and co-authors warn that a stronger first-stage tune can make the later reward stage worse. First-stage data comes from a fixed set, while reward learning uses the model's own fresh runs, so the two data worlds drift. Their PEAR fix re-weights first-stage loss by odds ratios, at token, block, or full-run level, with little extra cost once old odds are known. In tight game and math tests, PEAR lifted post-reward scores time and again, with gains up to 14.6 percent on AIME2025.

Catalog record

CREATORDylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51494
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01058

Browse the full library · About the Tower of Babel