Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, Vasant G Honavar · 2024
Read Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment by Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, Vasant G Honavar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14516.txt.
Introduction
Teng Xiao and co-authors improve preference training for language models in a 2024 paper. Training with human likes and dislikes now guides most chat models. A popular method contrasts a liked reply with a disliked one. That contrast, though, ignores the true size of each reward.
Their method, Cal-DPO, calibrates the hidden reward during training. Learned rewards are pulled to match the scale of true rewards. Theory in the paper backs advantages over prior methods. Tests on standard benchmarks show clear gains over off-the-shelf rivals. The fix is simple to add to existing pipelines. Better scale, the authors argue, yields better alignment with human taste. The work targets a subtle flaw in a widely used recipe. Small calibration steps, they show, can bring large practical gains.
Catalog record
| CREATOR | Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, Vasant G Honavar |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14516
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, Vasant G Honavar Abstract We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization ( Cal-DPO ), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods. Code is available at https://github.com/tengxiao1/Cal-DPO . 1 Introduction Aligning the behavior of large language models (LLMs) with human preferences is crucial for ensuring that the responses of a pretrained LLM are aligned with human or societal values and preferences [ 1 , 2 , 3 ] . In recent years, reinforcement learning from human feedback (RLHF) [ 2 , 4 ] has become a standard approach for fine-tuning language models based on human preferences. RLHF involves first fitting a reward signal from human preference data and then using reinforcement learning (RL) algorithms such as PPO [ 5 ] to optimize language models to generate responses with high reward. While RLHF shows impressive capabilities on diverse tasks ranging from programming to creative writing, its training process is unstable and complex [ 6 , 7 ] . This potentially worsens the sample complexity and compromises efficient convergence. To address these issues, offline contrastive preference learning methods, which include DPO [ 7 ] , IPO [ 8 ] , and SLiC [ 9 ] , have been proposed to replace RLHF with supervised learning on the preference data. These methods eliminate the need for explicit reward modeling by directly using the likelihood of the policy to define an implicit reward fitted to the preference data, and achieve notable efficiency and competitive performance [ 10 ] . While various contrastive preference learning methods employ different pairwise ranking losses, they share a common underlying motivation: Maximize the expected relative difference between the implicit rewards associated with the chosen and rejected responses. Because the ranking loss is invariant to various score transformations (e.g., subtracting a constant), these methods tend to ignore the absolute values of the rewards. Hence, while these methods learn to preserve the relative ordering between the likelihoods of the chosen and the rejected responses, they may reduce the likelihood of the chosen response. Figure 1 illustrates this behavior and its implications. In the case of DPO, a representative method of the contrastive methods, the likelihood of the chosen response counter-intuitively continues to decrease despite remaining higher than the likelihood of the rejected response. An undesirable consequence of this behavior is that the learned policy increases the likelihood of unknown out-of-distribution responses, resulting in poor performance. Maximizing the likelihood of the chosen response can be important in many practical applications, e.g., reasoning and mathematical problem solving [ 11 , 12 ] , limiting the applicability of contrastive preference learning. The preceding discussion raises an important question with significant implications for how we align LLMs with human preferences: How can we design a new objective that effectively alleviates this problem while ensuring that the learned policy theoretically converges to an optimal policy? Our answer to this question is Cal-DPO , a simple yet effective framework for preference learning which optimizes the contrastive preference objective to maximize the relative differences between implicit rewards of chosen and rejected responses, while simultaneously ensuring that learned implicit rewards are calibrated to match the actual values of the ground-truth rewards (see Section 4.1 for a formal definition). The key intuition behind Cal-DPO is quite simple: If the implicit reward estimates from preference data are well-calibrated relative to the ground-truth rewards (meaning both lie on the same scale), we can prevent the likelihood (reward) of chosen responses from continually decreasing. Hence, Cal-DPO is designed to learn an implicit reward parameterized by the policy calibrated against the ground-truth reward. This can be achieved through a simple modification to the existing methods. For instance, Cal-DPO can be implemented on top of DPO with just one line of code and without any additional hyperparameters. Although we refer to our method as Cal-DPO , it notably generalizes to other preference optimization methods such as IPO and SLiC (see Section 4.3 ). In addition, we theoretically demonstrate that Cal-DPO possesses several properties that are desirable for fine-tuning LLMs based on preferences, such as mode-seeking behavior, negative preference optimization, or "negative gradient" to push down the likelihood of undesirable responses [ 10 ] . The main contributions of this paper are: (i) We propose Cal-DPO , a simple, effective, and intuitive framework for preference learning that facilitates alignment of language models. Cal-DPO aims to learn implicit reward functions for learning policy that are calibrated with respect to ground-truth rewards. (ii) We theoretically analyze the learning behaviors of Cal-DPO and prove that Cal-DPO is guaranteed to yield an optimal policy for preference learning. (iii) We present results of extensive experiments on a range of benchmark tasks, including controlled text generation [ 13 ] , summarization [ 14 ] , dialogue generation [ 1 ] , and several reasoning tasks [ 15 ] that demonstrate that Cal-DPO consistently outperforms previous alignment methods for preference fine-tuning. Figure 1: The implicit reward dynamics during training of DPO and Cal-DPO on UltraFeedback data with the base model Zephyr-7b-sft reveal that the rewards for rejected data continuously decrease, while the margins between chosen and rejected data keep increasing. However, in DPO , the rewards for chosen data decrease below zero, whereas in our Cal-DPO , they keep increasing and remain positive. Our Cal-DPO significantly outperforms DPO across reasoning benchmarks. More results on other datasets are provided in Section 5 . 2 Related Work Reinforcement Learning from Human Feedback (RLHF) is highly effective in aligning Large Language Models (LLMs) with human preferences [ 2 , 4 ] . In RLHF, a reward model is trained from human preference data to map responses to a scalar reward, which aligns a policy using RL algorithms like PPO [ 5 ] . Although RLHF excels in instruction-following [ 2 ] , safety alignment [ 1 ] , and summarization [ 3 ] , it requires a more complex training pipeline than supervised learning. Recent work proposes simplifying RLHF by directly optimizing language models with contrastive learning on preference data, resulting in contrastive preference learning methods [ 10 ] , such as DPO [ 7 ] , IPO [ 8 ] , and SLiC [ 9 ] , NLHF [ 16 ] , and their variants that incorporate rejection sampling, e.g., RSO [ 17 ] . While each of these methods works with different loss functions, the primary objective is