When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
Amirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology · 2026
Read When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger by Amirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04968.txt.
Description
Amirabbas Afzali and team ask whether a weak language model can serve as a strong annotator for preference alignment, avoiding costly human labels or large API models. Selecting only the weak model's highly confident samples does better than using full human annotations. They build on this with Confidence-Weighted Preference Tuning, which re-weights samples by weak-model confidence across preference objectives. Notably, CW-PO trained on just 20% of human annotations outperforms standard DPO trained on 100%, suggesting confidence weighting can cut alignment cost while beating fully human-labeled training.
Catalog record
| CREATOR | Amirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96861 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04968