All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning

Gokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell · 2025

Read All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning by Gokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01067.txt.

Description

Swamy and four co-authors ask in twenty twenty-five why the best way to tune large chat models is a two-step route. First a score model learns from human picks, then the main model learns by trial with that score. A plain one-step fit on the same picks looks more direct.

They test a few views on why the two-step route wins, with both math and trials. The best-backed view rests on a gap: checks are easy to learn, but good replies are hard to write. The trial step then picks from a small set of replies that pass the easy check. That cuts the data the full task needs.

Catalog record

CREATORGokul Swamy, Sanjiban Choudhury & Wen Sun, Zhiwei Steven Wu & J. Andrew Bagnell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01067

Browse the full library · About the Tower of Babel