Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt · 2025

Read Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision by Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-07886.txt.

Description

Ye, Laidlaw, and Steinhardt show tag polish beats pick tune when guides run weak. Post-train for word tools leans on two guide steps in the norm. Show tags train first, then pick pairs train by man picks. As tools grow, their tasks grow too hard to guide well. The team fakes weak guides with small tools and timed men. With weak shows, first train still holds some force in tests.

Pick tune, by DPO, fails to lift past first train in that state. Their swap, ILR, uses pick pairs to polish the show set. Pairs say if a made reply should swap a man show tag. Fresh first train then runs on the polished set in the loop. That loop beats pick tune on math, code, and safe tasks. Weak-guide worlds may thus spend picks on tags, not tune.

Catalog record

CREATORYaowen Ye, Cassidy Laidlaw, Jacob Steinhardt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-07886

Browse the full library · About the Tower of Babel