Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
Inoussa Mouiche · 2026
Read Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models by Inoussa Mouiche (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02626.txt.
Description
Inoussa Mouiche and Abderaouf Bahi diagnose a failure mode in direct preference optimization, a popular way to train language models from ranked answers. Training widens the gap between chosen and rejected answers, but the objective does not say where the freed probability should go. Probability can collapse away from both the good answer and unrelated ones, a squeezing effect.
The fault, they show, is local: once a rejected answer falls into a low-probability valley, further suppression harms the whole distribution through the softmax link. Their fix, Gate-DPO, scales down each rejected answer's update based on its current probability, slowing excess suppression. Across four architectures, two datasets, and several objectives, gating reduced squeezing and raised the likelihood of chosen answers, without tying the method to one objective.
Catalog record
| CREATOR | Inoussa Mouiche |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02626