Preference Optimization by Estimating the Ratio of the Data Distribution

Yeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon · 2025

Read Preference Optimization by Estimating the Ratio of the Data Distribution by Yeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19601.txt.

Description

Yeongmin Kim and co-authors rethink how language models are tuned to match human preferences in 2025. The standard method, DPO, is prized for being simple and stable. The team studies a broader family of losses based on matching a target policy through likelihood ratios. That view avoids reward models and awkward normalizing terms.

Their framework, called BPO, includes DPO as one special case and gives workable forms for the rest. A scaling trick further steadies the training. On a popular test, a tuned model beat other models of its size on win rate while keeping varied output. The method takes only a few lines of code to add.

Catalog record

CREATORYeongmin Kim Heesun Bae Byeonghu Na Il-Chul Moon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE127 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19601

Browse the full library · About the Tower of Babel