Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution
Adam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher · 2026
Read Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution by Adam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06239.txt.
Description
Preference training can push a model too far toward pleasing its training data, a failure called over-optimization. Adam Barla and 3 co-authors present PEPO, a preference optimization method built to resist that failure without knowing how the data was generated. It needs no separate reward model.
PEPO trains an ensemble of policies on separate slices of the data, then combines them through a worst-case rule that favors where the models agree. That pessimism is the guardrail. In the tabular setting, its sample guarantees depend only on one policy's coverage, avoiding the harsher all-policy condition that burdens rival methods. Practical tests backed the theory, while the training stayed as simple as the standard approach.
Catalog record
| CREATOR | Adam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 139 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06239