Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator
Zhuotong Chen, Fang Liu, Xuan Zhu, Haozhu Wang, Yanjun Qi, Mohammad Ghavamzadeh · 2025
Read Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator by Zhuotong Chen, Fang Liu, Xuan Zhu, Haozhu Wang, Yanjun Qi, Mohammad Ghavamzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04567.txt.
Description
Zhuotong Chen and co-authors rethink how language models are trained from human preferences. Standard preference training pairs a favored answer with a disfavored one, chosen by rough rules of thumb with no firm theory behind them. The team instead frames the task as fitting a probability model, where the hard part is estimating a normalizing constant.
Their fix borrows a sampling trick called contrastive divergence to draw hard negative examples straight from the reward model. The samples then serve as the disfavored answers in training. A second version extends the method to online training, where fresh examples arrive as the model learns. On popular alignment benchmarks, the new method beat leading baselines, and the online version improved results further.
Catalog record
| CREATOR | Zhuotong Chen, Fang Liu, Xuan Zhu, Haozhu Wang, Yanjun Qi, Mohammad Ghavamzadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04567