All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
Gokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell · 2025
Read All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning by Gokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01067.txt.
Description
Swamy and four co-authors ask in twenty twenty-five why the best way to tune large chat models is a two-step route. First a score model learns from human picks, then the main model learns by trial with that score. A plain one-step fit on the same picks looks more direct.
They test a few views on why the two-step route wins, with both math and trials. The best-backed view rests on a gap: checks are easy to learn, but good replies are hard to write. The trial step then picks from a small set of replies that pass the easy check. That cuts the data the full task needs.
Catalog record
| CREATOR | Gokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01067