Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
Dylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng · 2026
Read Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning by Dylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01058.txt.
Description
Dylan Zhang and co-authors warn that a stronger first-stage tune can make the later reward stage worse. First-stage data comes from a fixed set, while reward learning uses the model's own fresh runs, so the two data worlds drift. Their PEAR fix re-weights first-stage loss by odds ratios, at token, block, or full-run level, with little extra cost once old odds are known. In tight game and math tests, PEAR lifted post-reward scores time and again, with gains up to 14.6 percent on AIME2025.
Catalog record
| CREATOR | Dylan Zhang, Yufeng (Felix) Xu, Haojin Wang, Qingzhi Chen, Hao Peng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51494 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.01058