What Matters in Data for DPO?

Yu Pan 1 Zhongze Cai 2 Guanting Chen 3 Huaiyang Zhong 4 Chonghuan Wang 5 1 University of Sydney 2 Imperial College London 3 University of North Carolina at Chapel Hill 4 Virginia Tech 5 University of Texas at Dallas · 2025

Read What Matters in Data for DPO? by Yu Pan 1 Zhongze Cai 2 Guanting Chen 3 Huaiyang Zhong 4 Chonghuan Wang 5 1 University of Sydney 2 Imperial College London 3 University of North Carolina at Chapel Hill 4 Virginia Tech 5 University of Texas at Dallas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18312.txt.

Introduction

Yu Pan and four co-authors present a 2025 study of what makes training data good for Direct Preference Optimization, or DPO. DPO tunes large language models to match human choices and skips a separate reward model. Yet no one knew which traits of paired preference data matter most.

Theory and tests point the same way: the quality of the chosen replies rules the outcome. The quality of the rejected replies matters far less, since contrast helps mostly by lifting the chosen set. In an online form, the method acts much like plain training on chosen replies. Tests across tasks confirmed the pattern. The authors draw lessons for building data sets that give the most gain per example.

Catalog record

CREATORYu Pan 1 Zhongze Cai 2 Guanting Chen 3 Huaiyang Zhong 4 Chonghuan Wang 5 1 University of Sydney 2 Imperial College London 3 University of North Carolina at Chapel Hill 4 Virginia Tech 5 University of Texas at Dallas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18312

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Matters in Data for DPO? Yu Pan 1 Zhongze Cai 2 Guanting Chen 3 Huaiyang Zhong 4 Chonghuan Wang 5 1 University of Sydney 2 Imperial College London 3 University of North Carolina at Chapel Hill 4 Virginia Tech 5 University of Texas at Dallas Abstract Direct Preference Optimization (DPO) has emerged as a simple and effective approach for aligning large language models (LLMs) with human preferences, bypassing the need for a learned reward model. Despite its growing adoption, a fundamental question remains open: what characteristics of preference data are most critical for DPO performance? In this work, we provide a systematic study of how preference data distribution influences DPO, from both theoretical and empirical perspectives. We show that the quality of chosen responses plays a dominant role in optimizing the DPO objective, while the quality of rejected responses may have relatively limited impact. Our theoretical analysis characterizes the optimal response distribution under DPO and reveals how contrastiveness between responses helps primarily by improving the chosen samples. We further study an online DPO setting and show it effectively reduces to supervised fine-tuning on the chosen responses. Extensive experiments across diverse tasks confirm our findings: improving the quality of chosen responses consistently boosts performance regardless of the quality of the rejected responses. We also investigate the benefit of mixing the on-policy data. Our results interpret the mechanism behind some widely adopted strategies and offer practical insights for constructing high-impact preference datasets for LLM alignment. 1 Introduction The importance of aligning large language models (LLMs) with human preferences cannot be overstated. Two leading paradigms for achieving this alignment are Reinforcement Learning from Human Feedback (RLHF) ( Bai et al. 2022 ; Ouyang et al. 2022 ) and Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) . The key difference lies in whether an explicit reward model is trained (as in RLHF) or whether the model itself is optimized directly using preference data (as in DPO). Significant effort has been devoted to improving the performance of these methods by constructing more effective preference datasets. Common techniques include rejection sampling (i.e., generating multiple responses and selecting the best or worst, see Khaki et al. 2024 ), annotator rewriting/editing, and iterative use of on-policy data ( Tajwar et al. 2024 ) . However, despite the empirical progress, fundamental questions about what properties of preference data actually matter for alignment remain underexplored. For example: Do chosen and rejected responses contribute symmetrically during optimization? How does the contrastiveness between response pairs affect learning? Under what conditions does incorporating on-policy data lead to gains? In this paper, we provide a systematic study of the role of preference data in DPO, combining theoretical analysis with empirical validation. We begin by analyzing how the preference dataset’s coverage of the high quality responses influences the gradient of the DPO loss. We begin by examining how the coverage of high-quality responses in the preference dataset influences the gradient of the DPO loss. Insufficient coverage of such responses can hinder optimization, as the DPO objective lacks an explicit gradient signal to promote high-reward outputs when they are absent from the comparisons in \mathcal{D}_{\text{DPO}} . Then, we analyze the optimal distribution that minimizes the DPO loss and how it is shaped by the distribution of the preference dataset. Our theoretical results show that the quality of the chosen responses plays a dominant role in DPO performance, while the quality of rejected responses has a more limited impact. We further demonstrate that the widely adopted strategy of increasing contrastiveness between responses is effective primarily because it tends to elevate the quality of the chosen responses. Moreover, we examine a simplified online DPO setting in which high-quality chosen responses remain fixed, and rejected responses are generated in an online fashion. We show that this setting essentially reduces to supervised fine-tuning on the chosen responses, further highlighting the central role of the quality of the chosen responses. We empirically validate our theoretical insights across multiple tasks and datasets. When fixing the chosen responses and varying the quality of the rejected ones, we observe little change in DPO performance. In contrast, when fixing the rejected responses and increasing the quality of the chosen ones, DPO performance consistently improves. Additionally, when holding the quality gap between chosen and rejected responses constant, improving the absolute quality of the chosen responses leads to better outcomes. Finally, we investigate how mixing on-policy and offline data affects performance under varying levels of offline data quality. 2 Preliminaries Supervised Fine-tuning (SFT). SFT is is typically the first stage in adapting a pre-trained LLM to downstream tasks. Given a dataset \mathcal{D}_{\text{SFT}} consisting of high-quality instruction-response pairs (\mathbf{x},\mathbf{y}) ( Ouyang et al. 2022 ), the objective is to maximize the log-likelihood of the the demonstration data. Specifically, SFT is minimizing the loss function: \mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{SFT}})=-\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}_{\text{SFT}}}[\log\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})]. (1) Reinforcement Learning from Human Feedback (RLHF). After SFT, fine-tuning with human preference data is widely used to further align the model. RLHF begins by training a reward model r(\mathbf{x},\mathbf{y}) to reflect human preferences based on a preference dataset \{(\mathbf{x}_{i},\mathbf{y}_{i.w},\mathbf{y}_{i,l})_{i\geq 1}\} , where \mathbf{y}_{i,w} and \mathbf{y}_{i,l} denote the preferred and rejected response, respectively. The policy \pi_{\theta} is then optimized, typically using reinforcement learning algorithms such as PPO ( Schulman et al. 2017 , Ouyang et al. 2022 ), by minimizing the following objective: \mathcal{L}_{\text{RLHF}}(\bm{\theta})=-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\mathbf{x}},\mathbf{y}\sim\pi_{\bm{\theta}}(\cdot|\mathbf{x})}[r(\mathbf{x},\mathbf{y})]+\beta\mathbb{D}_{\text{KL}}(\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})||\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})). (2) Directed Preference Optimization (DPO). To simplify the process of RLHF, particularly to get rid of the training of a reward model, Rafailov et al. 2023 realize that the reward function can be represented by the learning policy: r_{\bm{\theta}}(\mathbf{x},\mathbf{y})=\beta[\log\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})-\log\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})]+\beta\log Z_{\bm{\theta}}(\mathbf{x}), (3) where Z(\mathbf{x})=\sum_{\mathbf{y}}\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})\exp{(r_{\bm{\theta}}(\mathbf{x},\mathbf{y})/\beta)} is the partition function. Based on the Bradley-Terry (BT) preference assumption ( Bradley and Terry 1952 ), together with the pairs of chosen and rejected responses, DPO fine-tunes the language model by optimizing the following loss function: \mathcal{L}_{\text{DPO}}(\bm{\theta};\mathcal{D}_{\text{DPO}})=-\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{DPO}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\bm{\theta}}(\mathbf{y}_{w}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}_{w}|\mathbf{x})}-\beta\log\frac{\pi_{\bm{\theta}}(\mathbf{y}_{l}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}_{l}|\mathbf{x})}\right)\right], (4) where \mathbf{y}_{w} and \mathbf{y}_{l} are the chosen response and the rejected response respectively, and \sigma(x)=\frac{1}{1+e^{-x}} . The global minimizer of Eqs. ( 2 ) and ( 4 ) under BT assumption has been well understood to be \displaystyle\pi_{r^{*}}(\mathbf{y}|\mathbf{x})\propto\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})\exp\left(\frac{1}{\beta}r^{*}(\mathbf{x},\mathbf{y})\right), (5) where r^{*} is the true reward model. Rafailov et al. 2023 also derive the derivative of the DPO loss ( 4 ) with respect to the parameters \bm{\theta} : \displaystyle\nabla_{\bm{\theta}}\mathcal{L}_{\text{DPO}}(\bm{\theta};\mathcal{D}_{\text{DPO}})= \displaystyle-\beta\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{DPO}}}\left[\sigma(\hat{r}_{\bm{\theta}}(\mathbf{x},\mathbf{y}_{l})-\hat{r}_{\bm{\theta}}(\mathbf{x},\mathbf{y}_{w}))\left[\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(\mathbf{y}_{w}|\mathbf{x})-\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(\mathbf{y}_{l}|\mathbf{x})\right]\right], (6) where \hat{r}_{\bm{\theta}}(\mathbf{x},\mathbf{y})=\beta\log\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})} denotes the implicit reward induced by the language model \pi_{\bm{\theta}} and the reference model \pi_{\text{ref}} . In this work, we aim to better understand the role of the data distribution \mathcal{D}_{\text{DPO}} , and identify the key factors that contribute to successful DPO training. 3 Related Works RL-based LLM Alignment and DPO. Following SFT, RL policy gradient methods are then employed to align model outputs with human preferences encoded through reward modeling. While early RL approaches like TRPO ( Schulman et al. 2015 ) and PPO ( Schulman et al. 2017 ) established foundational frameworks, their computational intensity motivated the development of resource-efficient alternatives such as RAFT ( Dong et al. 2023 ) , RRHF ( Yuan et al. 2023 ) , SLiC ( Zhao et al. 2023 ) , and ORPO ( Hong et al. 2024 ) . DPO ( Rafailov et al. 2023 ) cleverly reinterprets the RL objective through contrastive loss by eliminating explicit reward value while maintaining

Read the complete text