KL Penalty Control via Perturbation for Direct Preference Optimization
Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, Youngjae Yu, Yonsei University, LG AI Research, University of Michigan, Ann Arbor · 2025
Read KL Penalty Control via Perturbation for Direct Preference Optimization by Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, Youngjae Yu, Yonsei University, LG AI Research, University of Michigan, Ann Arbor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13177.txt.
Introduction
Sangkyu Lee and co-authors improve a method that aligns language models with human choices, called direct preference optimization. A penalty in that method keeps the tuned model from drifting too far from its starting point. In the standard form the penalty stays fixed for every example.
Their version sets the penalty separately for each pair of answers. It checks how the model's confidence shifts when the penalty is nudged, using values the training already computes. On chatbot test sets, this simple rule beat most rival alignment methods. The authors argue it reflects how confused the model is about a pair and gives a better trade-off per example.
Catalog record
| CREATOR | Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, Youngjae Yu, Yonsei University, LG AI Research, University of Michigan, Ann Arbor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13177
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KL Penalty Control via Perturbation for Direct Preference Optimization Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, Youngjae Yu, Yonsei University, LG AI Research, University of Michigan, Ann Arbor Abstract Direct Preference Optimization (DPO) demonstrates the advantage of aligning a large language model with human preference using only an offline dataset. However, DPO has the limitation that the KL penalty, which prevents excessive deviation from the reference model, is static throughout the training process. Several methods claim to change this static KL penalty of DPO into a dynamic one, but no approach can adaptively assign different KL penalties for each preference pair. In this paper, we propose ε \varepsilon -Direct Preference Optimization ( ε \varepsilon -DPO), which allows adaptive control of the KL penalty strength β \beta for each preference pair. Specifically, ε \varepsilon -DPO adaptively controls β \beta for each preference pair based on the monotonicity of logits as a preference model under the perturbation of β \beta during training. This is equivalent to adjusting the KL penalty by checking whether the change in training-time temperature can lead to better preference confidence as preference models by simply reusing the logit of the current policy and the reference policy. Experimental results show that the simple criterion of ε \varepsilon -DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms on general chatbot benchmarks and reveal that this KL penalty control criterion can reflect confusion as a preference model and provide an efficient KL trade-off, highlighting the significance of instance-level adaptive KL penalty control in DPO. 1 1 1 The code is available at github.com/oddqueue/e-dpo. † † \ast Work done during internship at LG AI Research. 1 Introduction Aligning large language models with human preferences for helpfulness and harmless principles [ 2 , 4 , 9 ] is a crucial requirement for general chatbot agents. Reinforcement Learning from Human Feedback (RLHF) [ 44 ] is the pioneering approach that regards the alignment of large language models as a reward maximization problem and solves it by reinforcement learning [ 36 ] . However, the complicated training pipeline of RLHF increases the training complexity and computation cost of the rollout for online reinforcement learning, in addition to the difficulty of collecting human preference datasets. Moreover, introducing a trained reward model as a proxy reward function to replace the intractable ground-truth human preference reward function makes large language models suffer from the side effect of reward over-optimization [ 13 ] inherited from the reward models. Direct Preference Optimization (DPO) [ 31 ] proposes an alternative approach to reform the limitation of RLHF by converting the policy optimization problem into a preference modeling problem and performs alignment using only the offline preference dataset. It shows comparable performance while skipping the reward modeling process required by RLHF and has become an effective alternative approach for alignment. In particular, subsequent studies with various modifications to the DPO loss objective open a new research domain called direct alignment algorithms [ 32 ] , which perform alignment directly from offline preference datasets without training separate reward models. However, DPO assumes that \beta and the reference policy, which define a KL penalty that prevents excessive deviations from the reference model in RLHF, are fixed for exploiting the existence of a closed-form solution derived from the objective function of the RLHF. However, this assumption can lead to suboptimal results, since the KL penalty can be regarded as a Lagrangian relaxation of the constraint optimization defined by the trust region [ 36 ] . In this regard, \beta -DPO [ 39 ] argues that \beta should be adaptively chosen according to the quality of the preference pair but fails to control \beta at the instance-level and proposes a batch-level control method. On the other hand, TR-DPO [ 14 ] claims to periodically update the reference policy to reduce over-optimization [ 32 ] , but it may induce unnecessary KL divergence for improvement since the update is not adaptive. In this paper, we present \bm{\varepsilon} -Direct Preference Optimization ( \bm{\varepsilon} -DPO) , a simple instance-level adaptive KL penalty control for DPO that neither TR-DPO nor \beta -DPO achieves. Specifically, we check the advantage of adjusting \beta for each preference pair by observing the monotonicity of the log-likelihood ratio between the chosen response and the rejected response if the \beta used during training was perturbed, as described in Figure 1 . Here, the criterion for controlling \beta does not require batch-level statistics, and the policy under the perturbed \beta can be estimated by reusing the logits from policy and reference policy. This criterion results in independence from the choice of micro-batch size and no additional computation requirements for model updates, unlike \beta -DPO and TR-DPO. Experimental results demonstrate that the instance-level adaptive criterion of \varepsilon -DPO remarkably improves DPO, better than \beta -DPO and TR-DPO, to outperform most direct alignment algorithms that modify the DPO loss objective [ 41 , 43 , 3 , 40 , 12 , 18 , 30 , 28 ] . This reveals that the static KL penalty of DPO is the major bottleneck to final model performance and highlights the importance of instance-level adaptive KL penalty control. Furthermore, we confirm that the variation of \beta determined by the adaptive criterion in \varepsilon -DPO reflects the confusion as a preference model, which is not addressed in the adaptive \beta control criterion proposed by \beta -DPO. We also find that the adaptive KL penalty control of \varepsilon -DPO is crucial for an efficient KL trade-off compared to TR-DPO, which is not an adaptive KL penalty control because of the periodic update of the reference policy. Our contribution to the alignment of large language models can be summarized as threefold: (1) We present \varepsilon -DPO that changes the static KL penalty of DPO and adaptively controls the KL penalty at the instance-level with a simple criterion, which can outperform most direct alignment algorithms proposed as alternatives to DPO in general chatbot benchmarks. (2) We show that DPO’s perspective, which reparameterizes the policy to a preference model, can be converted to an approach for controlling the KL penalty at the instance-level by estimating the preference confidence changes according to the perturbation of \beta used during training. (3) We demonstrate that this instance-level adaptive KL penalty control distinguishes confusing preference pairs and achieves an efficient KL trade-off, and both are not addressed in existing research that performs KL penalty relaxation. Figure 1 : \varepsilon -DPO adaptively controls \beta corresponding to the KL penalty strength for each preference pair by checking monotonicity of the log-likelihood ratio of the chosen response and the rejected according to perturbation of training-time \beta by estimating the perturbed policies by reusing logits. 2 Preliminaries Reinforcement Learning from Human Feedback To obtain a language model that aligns with human preference, RLHF [ 44 ] introduces reinforcement learning. It is equivalent to approaching preference alignment as a reward maximization problem, where we find a policy \pi that maximizes a ground-truth reward function r^{*} representing human reward for a response y obtained from a corresponding policy for a given prompt x . However, since the ground-truth reward function cannot be accessed, a reward model trained