The Limits of Preference Data for Post-Training

Eric Zhao, Jessica Dai, Pranjal Awasthi · 2025

Read The Limits of Preference Data for Post-Training by Eric Zhao, Jessica Dai, Pranjal Awasthi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19964.txt.

Description

Eric Zhao, Jessica Dai, and Pranjal Awasthi study a limit on learning from human preferences in a 2025 study. Preference data give rankings between outputs rather than scores, and they are easy to collect at scale. Teams hope such rankings can train models for tasks like research help or trip planning, where success is qualitative.

The authors show a basic obstacle, even with perfect, endless, noise-free rankings. Ordinal feedback alone can block a learner from reaching even near-optimal outcomes. They formalize that impossibility result, which warns against treating preference training as a full substitute for richer outcome measures. The claimed limit is a theoretical one, proved under stated ideal conditions.

Catalog record

CREATOREric Zhao, Jessica Dai, Pranjal Awasthi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19964

Browse the full library · About the Tower of Babel