AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

Shaolong ChenIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA · 2026

Read AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates by Shaolong ChenIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28440.txt.

Introduction

Shaolong Chen and three colleagues diagnose an uneven gradient in DPO, the alignment method that skips a separate reward model. Its loss pushes disliked replies down far faster than it lifts liked ones, teaching dodging over good generation. Their AdaDPO variant assigns per-pair weights from the policy's own generation odds, making gradient sizes match for liked and disliked replies. On a Llama-3-8B test it beat standard DPO on most settings and reached the best win rate with length taken into account, while keeping the original settings.

Catalog record

CREATORShaolong ChenIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35415
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates Shaolong ChenIncept LabsHouston, TXMadalina CiobanuIncept LabsHouston, TX, Qingqing MaoIncept Labs, Houston, TXTitan Holdings, San Francisco, CA [email protected] Ritankar DasIncept Labs, Houston, TXTitan Holdings, San Francisco, CA Abstract Direct Preference Optimization (DPO) has become a widely adopted alternative to reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences, eliminating the need for a separate reward model or reinforcement learning loop. However, recent theoretical analysis uncovers an asymmetric gradient behavior in DPO: the loss suppresses dispreferred responses substantially faster than it promotes preferred counterparts, causing the model to predominantly learn to avoid bad answers rather than to generate good ones. We propose AdaDPO , a Self- Ada ptive variant of the D irect P reference O ptimization algorithm that introduces per-preference-pair, stop-gradient-based coefficients derived directly from the policy model’s generation probabilities, with the reference model’s probabilities as an optional component. AdaDPO is constructed to enforce equality of gradient magnitudes between preferred and dispreferred probabilities; the practical implementation balances per-token gradients and applies a numerical clipping bound for stability, while retaining DPO’s original hyperparameter structure. On Llama-3-8B-Instruct trained on UltraFeedback under a setup nearly identical to that of SimPO, AdaDPO consistently outperforms DPO on AlpacaEval 2: it achieves higher length-controlled win rates (LC) in 81% of hyperparameter combinations, attains the global best LC (48.3%) and raw win rate (46.1%), and enlarges the LC-over-WR margin in 88% of combinations, indicating effective mitigation of length bias. Additional analyses on KL divergence, reward margin, and reward accuracy confirm that AdaDPO rectifies the gradient imbalance and yields more efficient optimization. Because it operates purely at the loss level, AdaDPO can be dropped into existing preference-based alignment pipelines without changing data collection or model architectures. The method requires only a few lines of code, and the same self-adaptive principle generalizes to a broad family of pairwise contrastive preference losses including SimPO, R-DPO, IPO, CPO, and ORPO. 1 Introduction Aligning large language models (LLMs) with human preferences has become essential for deploying capable, controllable systems. Reinforcement Learning from Human Feedback ( 3 ; 21 ) , typically implemented with Proximal Policy Optimization ( 15 ) , established the dominant paradigm but introduces substantial complexity through its reliance on a separately trained reward model and an online RL loop. Direct Preference Optimization (DPO) ( 13 ) simplified this pipeline dramatically by reparameterizing the reward function and learning the policy directly from a fixed preference dataset via a binary cross-entropy loss, achieving strong empirical performance without reinforcement learning. Despite DPO’s success, recent theoretical analysis ( 6 ) has revealed an asymmetric gradient pathology in its loss function. Using a field-theoretic analysis of DPO’s gradient vector field, 6 show that the loss applies a substantially larger gradient to dispreferred responses than to preferred ones, with the asymmetry growing as the policy becomes more confident. As a consequence, DPO-trained models predominantly learn to suppress bad answers rather than to promote good ones—an effect that becomes especially pronounced when preferred and dispreferred responses are semantically close. This is not an artifact of hyperparameter choice but a structural property of the loss: the same coefficient \beta multiplies both log-ratios, leaving no mechanism to balance the per-pair gradient magnitudes. Several DPO variants address related concerns: SimPO ( 10 ) , R-DPO ( 12 ) , IPO ( 2 ) , CPO ( 18 ) , and ORPO ( 7 ) . However, none directly addresses the per-preference-pair gradient imbalance: the asymmetry persists in any pairwise contrastive loss with a shared coefficient on both terms. We propose AdaDPO, a self-adaptive variant of DPO that resolves this imbalance by replacing the fixed \beta in DPO’s implicit reward margin with per-preference-pair coefficients \beta_{w} and \beta_{l} . Setting \beta_{l}=\beta (preserving DPO’s hyperparameter) and computing \beta_{w} via a stop-gradient-based ratio of generation probabilities, AdaDPO is constructed to enforce equality of gradient magnitudes on preferred and dispreferred probabilities at every optimization step (the length-normalized variant used in our experiments balances gradients per-token within a numerical clipping bound). The construction is theoretically principled, requires only a few lines of code to implement, and introduces no sensitive hyperparameters beyond those already in DPO. The same self-adaptive principle extends as a drop-in modification to any pairwise contrastive preference loss (SimPO, R-DPO, IPO, CPO, ORPO). Contributions. (i) We rigorously derive the condition under which a DPO-style loss produces equal gradient magnitudes on preferred and dispreferred probabilities, and prove that AdaDPO satisfies it exactly through stop-gradient operators, providing a principled correction of the gradient imbalance (Section 3 ). (ii) On Llama-3-8B-Instruct trained on UltraFeedback under a setup nearly identical to SimPO ( 10 ) , AdaDPO consistently outperforms DPO on AlpacaEval 2: higher length-controlled win rates in 81% of hyperparameter combinations, the global best LC (48.3%) and raw WR (46.1%), and an enlarged LC-over-WR margin in 88% of combinations, indicating effective mitigation of length bias (Section 5 ). (iii) The same self-adaptive principle applies as a drop-in modification to any pairwise contrastive preference loss; we provide explicit constructions for SimPO, R-DPO, IPO, CPO, and ORPO that preserve each method’s existing hyperparameters (Section 6 ), highlighting a general recipe for balancing gradients in preference optimization. 2 Background Direct Preference Optimization. We are given a preference dataset \mathcal{D}=\{(x,y_{w},y_{l})\} of triples consisting of a prompt x , a preferred response y_{w} , and a dispreferred response y_{l} , together with a reference policy \pi_{\mathrm{ref}} . DPO ( 13 ) optimizes the policy \pi_{\theta} by minimizing \mathcal{L}_{\mathrm{DPO}}(\pi_{\theta};\pi_{\mathrm{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\!\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l}\mid x)}\right)\right], (1) where \sigma is the sigmoid function and \beta>0 controls the deviation from \pi_{\mathrm{ref}} . The same coefficient \beta multiplies both log-ratios—a structural choice with consequences we examine next. The gradient imbalance pathology. 6 analyze the gradient vector field of \mathcal{L}_{\mathrm{DPO}} and identify a structural asymmetry. Let P_{w}\!=\!\pi_{\theta}(y_{w}\mid x) , P_{l}\!=\!\pi_{\theta}(y_{l}\mid x) , R_{w}\!=\!\pi_{\mathrm{ref}}(y_{w}\mid x) , R_{l}\!=\!\pi_{\mathrm{ref}}(y_{l}\mid x) , and let x_{w}\!=\!P_{w}/R_{w} , x_{l}\!=\!P_{l}/R_{l} . The partial derivatives yield the gradient-magnitude ratios \mathrm{Ratio}_{x}=\left|\frac{\partial\mathcal{L}_{\mathrm{DPO}}/\partial x_{w}}{\partial\mathcal{L}_{\mathrm{DPO}}/\partial x_{l}}\right|=\frac{x_{l}}{x_{w}},\qquad\mathrm{Ratio}_{P}=\left|\frac{\partial\mathcal{L}_{\mathrm{DPO}}/\partial P_{w}}{\partial\mathcal{L}_{\mathrm{DPO}}/\partial P_{l}}\right|=\frac{P_{l}}{P_{w}}. (2) As training progresses and the policy learns to prefer y_{w} , both ratios fall below 1, meaning DPO suppresses y_{l} more aggressively than it promotes y_{w} ( 6 ) . We note that DPO’s gradients with respect to log-probabilities are balanced by construction; the asymmetry analyzed by 6 arises in probability and probability-ratio space, where it reflects the differential rate of change of P_{w} versus P_{l} during training. AdaDPO operates in this same space, deliberately introducing a compensating asymmetry in log-probability space to maintain promotion pressure on y_{w} as the policy becomes confident. This imbalance is structural rather than an artifact of hyperparameter choice: equation ( 1 ) applies the same coefficient \beta to both log-ratio terms. A concrete illustration. Consider a single preference pair where, after some training, the policy assigns P_{w}=0.10 and P_{l}=0.02 (the model has learned to prefer y_{w} ). With a uniform reference R_{w}=R_{l} , the gradient on y_{w} is roughly 0.02/0.10=1/5 of the gradient on y_{l} , so the optimization signal on the dispreferred response is five times stronger than on the preferred one. As P_{w} increases further, the gradient on y_{w} shrinks toward zero and DPO continues to focus on suppressing y_{l} rather than reinforcing y_{w} . AdaDPO eliminates this asymmetry by

Read the complete text