Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
Kaiyang Guo, Yinchuan Li, Zhitang Chen · 2025
Read Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO by Kaiyang Guo, Yinchuan Li, Zhitang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23316.txt.
Introduction
Kaiyang Guo, Yinchuan Li, and Zhitang Chen pull apart a hidden flaw in a training method in a 2025 paper. The method, direct preference optimization, trains language models by setting liked answers against disliked ones. It reads relative taste well, yet it widely pushes down the raw chances of sample answers. Models tuned this way can drift into reward-hacking moves, even when no reward model is in sight.
The authors name the flaw likelihood underdetermination: the raw chances are left too loose. They show that the loss behind the method splits into parts in a principled way. Read that way, the loss stretches to wider kinds of feedback and shows the root cause: the standard form trims a regularizer term down too far. Putting the full term back fixes the flaw. Their proximalized method builds on that fix and handles feedback of many kinds.
Catalog record
| CREATOR | Kaiyang Guo, Yinchuan Li, Zhitang Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO Kaiyang Guo, Yinchuan Li, Zhitang Chen Abstract Direct alignment methods typically optimize large language models (LLMs) by contrasting the likelihoods of preferred versus dispreferred responses. While effective in steering LLMs to match relative preference, these methods are frequently noted for decreasing the absolute likelihoods of example responses. As a result, aligned models tend to generate outputs that deviate from the expected patterns, exhibiting reward-hacking effect even without a reward model. This undesired consequence exposes a fundamental limitation in contrastive alignment, which we characterize as likelihood underdetermination. In this work, we revisit direct preference optimization (DPO)—the seminal direct alignment method—and demonstrate that its loss theoretically admits a decomposed reformulation. The reformulated loss not only broadens applicability to a wider range of feedback types, but also provides novel insights into the underlying cause of likelihood underdetermination. Specifically, the standard DPO implementation implicitly oversimplifies a regularizer in the reformulated loss, and reinstating its complete version effectively resolves the underdetermination issue. Leveraging these findings, we introduce PRoximalized PReference Optimization (PRO), a unified method to align with diverse feeback types, eliminating likelihood underdetermination through an efficient approximation of the complete regularizer. Comprehensive experiments show the superiority of PRO over existing methods in scenarios involving pairwise, binary and scalar feedback. 1 Introduction Learning from feedback provides a powerful paradigm to align large language models (LLMs) with nuanced human preferences. Instead of relying on meticulously crafted gold standards for supervision, it leverages preference feedback to promote desired behaviors and inhibit undesired ones. This shift in learning signal facilitates scalable data generation, yet simultaneously introduces the need for more advanced alignment techniques. Initially, reinforcement learning from human feedback (RLHF) transforms pairwise preference feedback into continuous rewards, which are then used to train LLM via reinforcement learning [ 1 , 2 ] . Although RLHF has shown empirical effectiveness, it presents significant computational challenges, including the need for training an additional reward model and generating on-policy responses with LLM. Moreover, as an imperfect proxy, the reward model may produce unreliable evaluations for responses that fall outside training distribution, making RLHF susceptible to reward hacking [ 3 , 4 , 5 , 6 , 7 ] . Direct preference optimization (DPO) circumvents the need for an explicit reward model and learns directly from offline preference data [ 8 ] . Its core methodology involves constructing a contrastive loss, that maximizes the likelihood differences between preferred and dispreferred responses. Owing to the simplicity, DPO has inspired the development of numerous contrastive approaches [ 9 , 10 , 11 ] . However, an unanticipated phenomenon has been frequently reported in DPO: the likelihoods of both preferred and dispreferred responses decrease after alignment [ 12 , 13 , 14 , 15 ] . This reduction inadvertently encourages the generation of out-of-distribution responses, suggesting persistent reward hacking even without an explicit reward model [ 16 ] . Recent research on this phenomenon predominantly focuses on the training dynamics of DPO and its general variants. The proposed explanations include embedding similarity between paired responses [ 15 ] , asymmetric update ratios for the probabilities of paired responses [ 17 ] , and the “squeezing effect” of softmax when applying gradient ascent to dispreferred responses [ 18 ] . While these studies offer valuable insights, they largely overlook the intrinsic limitation of the loss function itself. In particular, when attention is limited to relative likelihood differences, the contrastive loss becomes insensitive to simultaneous decreases or increases in absolute likelihoods. We characterize this issue as likelihood underdetermination. Several works also noticed and attempted to remedy it, but they incorporate additional supervised or regression signals into DPO [ 12 , 14 ] , which may unintentionally compromise the original intent of alignment. The aforementioned line of research aligns LLMs using pairwise feedback. In addition, parallel studies have explored alternative feedback types for alignment. Considering that response-pair annotation demands more effort than single-response evaluation, Kahneman-Tversky optimization (KTO) aligns with binary feedback by constructing separate utility functions for desired and undesired responses [ 19 ] . Conversely, in scenarios where finer-grained scalar feedback is available, noise contrastive alignment (NCA) formulates a classification task to capture the varying degree of desirability for each labeled response [ 13 ] . Both KTO and NCA depart from the contrastive framework. Although they can either directly or under certain assumption be adapted to pairwise feedback, KTO does not support scalar feedback, and NCA is incompatible with binary feedback. Currently, there is still no unified approach that seamlessly accommodates all these feedback types. For a more detailed discussion of related works, see Appendix A . In this study, we revisit DPO, the seminal method in field of direct alignment. Through a theoretical reformulation of its loss, we highlight DPO’s inherent capacity to accommodate diverse feedback, and reveal new insights into the underlying cause of likelihood underdetermination. Building on these findings, we propose a practical approach that not only addresses the underdetermination issue, but also unifies alignment across various feedback types, thereby achieving the best of both worlds pursued in previous works. Specifically, our contributions are as follows: • DPO Reformulation We show that the DPO loss admits a decomposed reformulation, which consists of separate optimizer and regularizer terms. The optimizer reorganizes pairwise feedback into a pointwise signal, naturally extending applicability to a wider range of feedback types. The regularizer is independent of the preference label, and thus allows for a more flexible development of sample-based variant. • Unveiling Likelihood Underdetermination We find that the standard DPO implementation implicitly oversimplifies the regularizer introduced in the reformulation. Importantly, once the full regularizer is reinstated, any optimal solution to the reformulated loss—if it exists—no longer suffers from likelihood underdetermination. • Practical Approach However, computing the regularizer in full is intractable, and the optimal solution may not always exist. To overcome these challenges, we propose PRoximalized PReference Optimization (PRO), which employs a hyper response mechanism for efficiently approximating the regularizer, and guarantees the existence of an optimal solution whenever its hyperparameter is properly chosen. • Empirical Evaluation The experiments show that PRO mitigates likelihood underdetermination, performs effectively across diverse feedback types, and achieves performance comparable to or better than DPO and several other methods designed for specific feedback types. Remarkably, even in the challenging scenario with extremely imbalanced binary feedback (desired:undesired = 1:100), PRO demonstrates comparable performance to that obtained with fully balanced feedback. 2 Preliminaries In LLM alignment, preference feedback data is commonly collected through two steps: prompting a baseline model with inputs x to generate response pair y_{1},y_{2}\sim\mu(y|x) ; human annotators labeling the generated responses to indicate which is preferred. Let y_{w}\succ y_{l}|x denote the preference feedback, where y_{w} and y_{l} are preferred and dispreferred responses amongst y_{1} and y_{2} . RLHF leverages reward modeling to transform the pairwise feedback data \mathcal{D}=\big\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\big\}_{i=1}^{I} into quantifiable rewards, and subsequently uses these signals to optimize LLM via reinforcement learning. Recently, DPO [ 8 ] circumvents the need for explicit reward modeling, by recognizing that the optimal solution to the RL problem possesses a closed-form relationship with the reward function. In consequence, the reward function can be expressed using the LLM to be optimized: \displaystyle r_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}, where \pi_{\text{ref}} is