Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

Inoussa Mouiche · 2026

Read Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models by Inoussa Mouiche (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02626.txt.

Description

Inoussa Mouiche and Abderaouf Bahi diagnose a failure mode in direct preference optimization, a popular way to train language models from ranked answers. Training widens the gap between chosen and rejected answers, but the objective does not say where the freed probability should go. Probability can collapse away from both the good answer and unrelated ones, a squeezing effect.

The fault, they show, is local: once a rejected answer falls into a low-probability valley, further suppression harms the whole distribution through the softmax link. Their fix, Gate-DPO, scales down each rejected answer's update based on its current probability, slowing excess suppression. Across four architectures, two datasets, and several objectives, gating reduced squeezing and raised the likelihood of chosen answers, without tying the method to one objective.

Catalog record

CREATORInoussa Mouiche
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02626

Browse the full library · About the Tower of Babel