Principled Foundations for Preference Optimization * {}^{\mbox{\small*}}
Wenxuan Zhou, Shujian Zhang, Brice Magdalou, John Lambert, Ehsan Amid, Richard Nock, Andrew Hard · 2025
Read Principled Foundations for Preference Optimization * {}^{\mbox{\small*}} by Wenxuan Zhou, Shujian Zhang, Brice Magdalou, John Lambert, Ehsan Amid, Richard Nock, Andrew Hard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07855.txt.
Description
Wenxuan Zhou and co-authors rebuild the theory under a popular method for tuning language models to human taste in a 2025 paper. The method, Direct Preference Optimization, skips building a separate reward model by tying training to a classic model of human choice. That tie was narrow, covering only one choice model.
The paper widens the frame to the method's full normative base. By reworking textbook choice theory to fit machine learning, the authors show any sound learning choice can pair with any choice model. Surprises follow: losses need not be convex, and later extensions of the method, such as margin and length fixes, sit safely inside the frame. A small toy test illustrates the reach.
Catalog record
| CREATOR | Wenxuan Zhou, Shujian Zhang, Brice Magdalou, John Lambert, Ehsan Amid, Richard Nock, Andrew Hard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07855