When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

Amirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology · 2026

Read When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger by Amirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04968.txt.

Description

Amirabbas Afzali and team ask whether a weak language model can serve as a strong annotator for preference alignment, avoiding costly human labels or large API models. Selecting only the weak model's highly confident samples does better than using full human annotations. They build on this with Confidence-Weighted Preference Tuning, which re-weights samples by weak-model confidence across preference objectives. Notably, CW-PO trained on just 20% of human annotations outperforms standard DPO trained on 100%, suggesting confidence weighting can cut alignment cost while beating fully human-labeled training.

Catalog record

CREATORAmirabbas Afzali Myeongho Jeon Maria Brbić, [0.2em] EPFL Sharif University of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96861
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04968

Browse the full library · About the Tower of Babel