InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization
Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang LiuBeijing Academy of Artificial IntelligenceBeihang University [email protected] · 2025
Read InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization by Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang LiuBeijing Academy of Artificial IntelligenceBeihang University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15880.txt.
Introduction
Yunan Wang and co-authors blend on- and off-policy data for preference tuning in a 2025 paper. Direct preference optimization aligns models with human choices. Samples from the model itself stay on-distribution but cap quality. External data can be better yet shifts the distribution.
Their method synthesizes preference pairs from both sources. Dynamic adjustment balances shift against quality continuously. The optimal trade-off is found during training. Limitations of each source alone are overcome. Two public alignment benchmarks evaluated the approach. It beat both pure on-policy and pure off-policy training. A top win rate on the hard benchmark was reported. The base method remained ordinary preference optimization. Data synthesis, not a new loss, drives the gain. Alignment teams can reuse existing external corpora safely. Quality ceilings imposed by weak generators loosen.
Catalog record
| CREATOR | Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang LiuBeijing Academy of Artificial IntelligenceBeihang University [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15880
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang LiuBeijing Academy of Artificial IntelligenceBeihang University [email protected] Abstract Direct Preference Optimization (DPO) optimizes language models to align with human preferences. Utilizing on-policy samples, generated directly by the policy model, typically results in better performance due to its distribution consistency with the model compared to off-policy samples. This paper identifies the quality of candidate preference samples as another critical factor. While the quality of on-policy data is inherently constrained by the capabilities of the policy model, off-policy data, which can be derived from diverse sources, offers greater potential for quality despite experiencing distribution shifts. However, current research mostly relies on on-policy data and neglects the value of off-policy data in terms of data quality, due to the challenge posed by distribution shift. In this paper, we propose InCo-DPO, an efficient method for synthesizing preference data by integrating on-policy and off-policy data, allowing dynamic adjustments to balance distribution shifts and data quality, thus finding an optimal trade-off. Consequently, InCo-DPO overcomes the limitations of distribution shifts in off-policy data and the quality constraints of on-policy data. We evaluated InCo-DPO with the Alpaca-Eval 2.0 and Arena-Hard benchmarks. Experimental results demonstrate that our approach not only outperforms both on-policy and off-policy data but also achieves a state-of-the-art win rate of 60.8 on Arena-Hard with the vanilla DPO using Gemma-2 model. † † footnotetext: * Corresponding authors 1 Introduction Aligning Large Language Models (LLMs) with human preferences significantly improves their performance in terms of helpfulness, correctness, coherence, and safety ( Gao et al. 2024 ; Achiam et al. 2023 ) . Among widely adopted alignment methods, Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) directly aligns the language model with human preferences using preference pairs (i.e., responses y_{win}>y_{lose} for a given prompt x ). Preference data, as a critical factor in alignment, has garnered significant attention Cui et al. 2024 ; Zhang et al. 2024 ; D’Oosterlinck et al. 2024 . Preference data can be categorized into on-policy data generated by the current policy model itself, and off-policy data from other models or human annotations. In LLM alignment, distributional shift is a key consideration Yang et al. 2024 ; Thangarasa et al. 2024 ; Zhou et al. 2024 . Off-policy data often exhibits a distributional shift relative to the current policy model, leading to instability and hindering convergence. In contrast, on-policy data benefits from distribution consistency, often resulting in better performance. However, we find that the quality of the candidate preference samples is another critical factor (see Section 3.2 ), and the value of off-policy data in terms of quality is often overlooked. Since on-policy data is generated by the current policy model, its quality is inherently limited by the model’s own capabilities (especially for smaller-scale models). On the other hand, off-policy data can be derived from diverse sources such as existing supervised fine-tuning datasets, preference datasets and manually constructed dialogue data. Among these sources, responses from state-of-the-art (SOTA) open-source models or human annotations tend to outperform on-policy data in terms of quality. Consequently, pure on-policy and off-policy data each have distinct advantages and limitations regarding distributional shifts and data quality. However, current research mostly relies on on-policy data and neglects the value of off-policy data in terms of data quality due to the challenge posed by distribution shift Agarwal et al. 2024 ; Meng et al. 2024 ; Tajwar et al. 2024 ; Yuan et al. 2024 ; Wu et al. 2024b . To address this gap, we propose InCo-DPO (Figure 3 ), an efficient preference data synthesis method that In tegrates on-policy and off-policy data based on Co ntinuation. InCo-DPO supports dynamic adjustment of the trade-off between distribution shift and data quality, thus overcoming the challenges posed by the distribution shift of off-policy data and quality limitations of on-policy data. We quantify the quality of preference data with the reward provided by a reward model, and the degree of distributional shift with the normalized output probability by the policy model (see Section 3.1 ). Inspired by the observation that the reward of partial responses often positively correlates with the reward of full responses Sun et al. , we introduce a simple yet effective continuation-based data synthesis method that significantly improves response reward with only a minor sacrifice in distribution consistency. For a given instruction, we first sample a few tokens with a SOTA open-source model as a prefix and then directly continue the prefix with the target policy model. Through this pipeline, the reward score of the response can be significantly improved compared to on-policy responses even though when the prefix length is less than 10 tokens. To mitigate the cost of reduced distribution consistency, we reduce the sampling temperature, thereby increasing the likelihood of the model outputting the token consistent with its own predictions. We evaluate the effectiveness of InCo-DPO with Alpaca Eval 2.0 and Arena-Hard benchmarks across multiple open-source models, datasets and DPO variants. Experimental results demonstrate that our method achieves superior performance compared to both on-policy and off-policy data. Using UltraFeedback instruction dataset and Gemma-2 model, InCo-DPO achieves a SOTA win rate of 60.8 on Arena-Hard with vanilla DPO. In summary, our contributions are as follows: • Identifying data quality as key factor: While past research limited to the analysis of distribution shift, we uncover data quality as another key factor for preference learning. • Exploiting the value of off-policy data: We highlight the long-overlooked value of off-policy data and efficiently leverage its advantages in reward compared to on-policy data. • Proposing InCo-DPO: We introduce a novel preference data synthesis strategy that integrates off-policy and on-policy data through a continuation-based method. InCo-DPO dynamically balances distribution shift and reward, thus significantly improving the alignment performance of DPO compared to pure on-policy and off-policy settings. • Extensive empirical validation: We conduct comprehensive experiments on multiple open-source datasets, models, and DPO variants, demonstrating that preference data synthesized via InCo-DPO outperforms both on-policy and off-policy data. 2 Related Work LLM preference data synthesis and augmentation . High-quality training data are challenging to collect and often limited in quantity. To address this issue, researchers have proposed various methods for synthesizing and augmenting data with LLMs Gao et al. 2023 ; Gandhi et al. 2024 ; Wang et al. 2023 ; Cui and Wang 2023 ; Ye et al. 2023 . Ultrafeedback employs models of different scales and series to generate diverse responses based on various prompting principles, with preference annotations provided by GPT-4. SDFT Yang et al. 2024 leverages the trained model to rewrite downstream SFT data to mitigate the impact of distribution gap. WPO Zhou et al. 2024 and DNO Rosset et al. 2024 incorporate an off-policy sample from strong model as teacher response to build preference pairs. Unlike this sample-level data augmentation approach, InCo-DPO dynamically adjusts data quality and distribution shift by introducing token-level strong external model response prefixes, achieving more flexible control. On-policy vs. off-policy . When applying RL to the post-training of the LLM, on-policy learning ( Zheng et al. 2023 ; Santacroce et al. 2023 ; Dong et al. 2024 ) samples training data from the policy model (i.e.,