Preference Optimization by Estimating the Ratio of the Data Distribution
Yeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon · 2025
Read Preference Optimization by Estimating the Ratio of the Data Distribution by Yeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19601.txt.
Description
Yeongmin Kim and co-authors rethink how language models are tuned to match human preferences in 2025. The standard method, DPO, is prized for being simple and stable. The team studies a broader family of losses based on matching a target policy through likelihood ratios. That view avoids reward models and awkward normalizing terms.
Their framework, called BPO, includes DPO as one special case and gives workable forms for the rest. A scaling trick further steadies the training. On a popular test, a tuned model beat other models of its size on win rate while keeping varied output. The method takes only a few lines of code to add.
Catalog record
| CREATOR | Yeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 127 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19601