Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

Adam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher · 2026

Read Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution by Adam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06239.txt.

Description

Preference training can push a model too far toward pleasing its training data, a failure called over-optimization. Adam Barla and 3 co-authors present PEPO, a preference optimization method built to resist that failure without knowing how the data was generated. It needs no separate reward model.

PEPO trains an ensemble of policies on separate slices of the data, then combines them through a worst-case rule that favors where the models agree. That pessimism is the guardrail. In the tabular setting, its sample guarantees depend only on one policy's coverage, avoiding the harsher all-policy condition that burdens rival methods. Practical tests backed the theory, while the training stayed as simple as the standard approach.

Catalog record

CREATORAdam Barla, Emanuele Nevali, Luca Viano, Volkan Cevher
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06239

Browse the full library · About the Tower of Babel