Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt · 2025
Read Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision by Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-07886.txt.
Description
Ye, Laidlaw, and Steinhardt show tag polish beats pick tune when guides run weak. Post-train for word tools leans on two guide steps in the norm. Show tags train first, then pick pairs train by man picks. As tools grow, their tasks grow too hard to guide well. The team fakes weak guides with small tools and timed men. With weak shows, first train still holds some force in tests.
Pick tune, by DPO, fails to lift past first train in that state. Their swap, ILR, uses pick pairs to polish the show set. Pairs say if a made reply should swap a man show tag. Fresh first train then runs on the polished set in the loop. That loop beats pick tune on math, code, and safe tasks. Weak-guide worlds may thus spend picks on tags, not tune.
Catalog record
| CREATOR | Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-07886