Nearly Optimal Active Preference Learning and Its Application to LLM Alignment

Yao Zhao, Kwang-Sung Jun · 2026

Read Nearly Optimal Active Preference Learning and Its Application to LLM Alignment by Yao Zhao, Kwang-Sung Jun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01581.txt.

Description

Yao Zhao and Kwang Sung Jun cut the cost of aligning language models by choosing which preference labels to collect next. Active learning designs borrow optimality rules ignoring the special shape of preference data, so the authors question their fit here. They propose two methods: one with the first instance dependent label complexity guarantee here, and a simpler greedy method. They prove the key term optimal up to a constant tied to the logit bound, and tests on real and synthetic data show better sample efficiency.

Catalog record

CREATORYao Zhao, Kwang-Sung Jun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69305
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01581

Browse the full library · About the Tower of Babel