The Limits of Preference Data for Post-Training
Eric Zhao, Jessica Dai, Pranjal Awasthi · 2025
Read The Limits of Preference Data for Post-Training by Eric Zhao, Jessica Dai, Pranjal Awasthi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19964.txt.
Description
Eric Zhao, Jessica Dai, and Pranjal Awasthi study a limit on learning from human preferences in a 2025 study. Preference data give rankings between outputs rather than scores, and they are easy to collect at scale. Teams hope such rankings can train models for tasks like research help or trip planning, where success is qualitative.
The authors show a basic obstacle, even with perfect, endless, noise-free rankings. Ordinal feedback alone can block a learner from reaching even near-optimal outcomes. They formalize that impossibility result, which warns against treating preference training as a full substitute for richer outcome measures. The claimed limit is a theoretical one, proved under stated ideal conditions.
Catalog record
| CREATOR | Eric Zhao, Jessica Dai, Pranjal Awasthi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19964