Mitigating Mismatch within Reference-based Preference Optimization
Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng Dadong Wang Ivor Tsang Tongliang Liu · 2026
Read Mitigating Mismatch within Reference-based Preference Optimization by Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng Dadong Wang Ivor Tsang Tongliang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11902.txt.
Description
Direct Preference Optimization is the standard for offline alignment tuning, but its reference policy creates tension. Each update is weighed relative to that policy, keeping training stable. For pessimistic pairs, where the reference prefers the rejected response, gradients shrink as soon as the policy margin merely beats the anchor margin, even while the policy is still wrong. Suqin Yuan and six co-authors call this premature satisfaction, a training-inference mismatch.
Anchor-free objectives remove the mismatch by optimizing the absolute margin, but discard the steadying anchor signal. Hybrid-DPO mitigates the tension, applying the anchor by case. It behaves like standard DPO when the anchor is upbeat or neutral, and treats the anchor as neutral when downbeat, flooring the anchor margin at zero. That one-line change strengthens learning signals on such pairs while keeping objective form and cost, improving test-time metrics and pairwise win rates.
Catalog record
| CREATOR | Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng Dadong Wang Ivor Tsang Tongliang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-11902