Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

2025

Read Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04149.txt.

Introduction

Xuan Qi, Rongwu Xu, and Zhijing Jin wrote this AI paper in 2025 on training language models to match human taste. Preference data, pairs where one answer beats another, drive that training, but such data are dear. Worse, no good method picked which pairs teach most. The team propose a difficulty test drawn from the training method itself: pairs whose hidden reward gap is small are the hard, telling cases.

Training on those hard pairs beat five strong rival pickers across data sets and tasks. Just ten percent of the original data sufficed for the win. The authors frame the pick as principled, not a hunch, since it reads the model's own reward math. They offer it as a way to align models well when data budgets are tight.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap Abstract Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources. Code and data to reproduce our method can be found at https://github.com/Difficulty-Based-Preference-Data-Select/Difficulty-Based-Preference-Data-Select . Xuan Qi *1 , Rongwu Xu *1,2 , Zhijing Jin 3,4,5 1 Tsinghua University 2 University of Washington 3 MPI for Intelligent Systems, Tübingen, Germany 4 University of Toronto 5 Vector Institute [email protected] , [email protected] , [email protected] † † footnotetext: ∗ represents co-first authors. The junior author is listed first. 1 Introduction Aligning large language models (LLMs) with human preferences has emerged as one of the most critical challenges in recent AI research DBLP:conf/nips/ChristianoLBMLA17 . As LLMs demonstrate increasingly sophisticated capabilities across diverse domains naveed2023comprehensive ; thirunavukarasu2023large ; qiu2025alita , ensuring that their outputs align with human values and expectations becomes paramount for safe and beneficial deployment yudkowsky2016ai ; gabriel2020artificial ; ouyang2022training . Among the various alignment paradigms, Reinforcement Learning from Human Feedback (RLHF) DBLP:conf/nips/ChristianoLBMLA17 ; ouyang2022training has proven instrumental in fine-tuning state-of-the-art models. More recently, Direct Preference Optimization (DPO) rafailov2024direct has gained significant traction as a computationally efficient alternative that bypasses explicit reward modeling while maintaining competitive performance. Central to the success of both algorithms is the quality of preference data that captures nuanced distinctions between desirable (e.g., helpful, honest) and undesirable (e.g., harmful, biased) model behaviors. However, as preference datasets scale to hundreds of thousands of examples (e.g., the widely used SHP dataset has 350K samples DBLP:journals/corr/abs-2112-00861 ), the computational burden and potential inclusion of low-quality or redundant data points gao2024impact necessitate data selection strategies. Effective curation of high-quality preference data not only reduces training costs but also enhances model alignment by focusing learning on the most informative preference signals. Despite the critical importance of preference alignment, existing data selection methodologies for the LLM training pipeline predominantly target instruction fine-tuning (IFT) datasets rather than preference datasets. 1 1 1 While IFT can be viewed as a form of behavior alignment in the broad sense of AI alignment DBLP:conf/nips/ChristianoLBMLA17 , our focus here is on aligning models with explicit preferences. Current approaches, including difficulty-based methods (filtering examples based on challenge level) swayamdipta2020dataset ; pleiss2020identifying ; yuan2023self , diversity-based techniques (selecting maximally heterogeneous subsets) agarwal2021openbook ; sorscher2022beyond ; xie2023data , and importance-based strategies (leveraging metrics to prioritize influential data points) paul2021deep ; mindermann2022prioritized ; marion2023less , are fundamentally designed for data in IFT. However, preference datasets have a fundamentally different structure: Each data point comprises an instruction paired with two responses, one chosen and one rejected, creating a comparative learning signal that requires specialized treatment. This structural distinction renders many IFT-oriented data selection algorithms inapplicable or suboptimal for working with preference datasets. Despite recent advances in preference alignment data filtering, such as SDPO DBLP:journals/corr/abs-2502-09650 , these approaches have not adequately addressed the challenge of identifying high-quality preference data subsets, nor have they demonstrated sufficiently robust performance improvements. Consequently, the field currently lacks effective, theoretically grounded algorithms specifically designed for preference data selection, representing a significant gap in the LLM alignment toolkit . In response to this gap, we propose a novel difficulty-based data selection method specifically designed for preference datasets. Our approach leverages the implicit reward mechanism inherent in the DPO algorithm rafailov2024direct to quantify the difficulty of preference examples through the DPO implicit reward gap , which is the difference between implicit rewards assigned to chosen and rejected responses. The core insight underlying our method is that preference examples with smaller reward gaps present greater learning challenges, as they represent boundary cases where the model exhibits uncertainty in distinguishing between preferred and rejected responses. This uncertainty manifests as higher gradient magnitudes during optimization, indicating greater learning potential due to amplified training signals at decision boundaries. Building on this theoretical foundation, we develop a systematic three-stage selection strategy: (1) computing DPO implicit reward gaps for all preference pairs using an aligned policy and its corresponding reference policy, (2) ranking examples by ascending reward gaps, and (3) selecting a subset where data points’ reward gaps are under a certain threshold for downstream preference learning tasks. This principled approach ensures that selected examples provide maximum learning signal while maintaining computational efficiency. To verify the effectiveness of our method, we carry out comprehensive empirical validations of our method across four preference datasets of diverse data types, including both human-annotated preferences (SHP DBLP:journals/corr/abs-2112-00861 ) and synthetic datasets (Skywork DBLP:journals/corr/abs-2410-18451 , UltraFeedback DBLP:journals/corr/abs-2310-01377 , RLHFlow RLHFlow2024 ). Our evaluation covers two prevalent alignment tasks, reward model training and policy fine-tuning via DPO. Our approach is then benchmarked against five strong baselines. Results show that it consistently outperforms other data selection methods using the same amount of data. Furthermore, it even surpasses the models trained on the full dataset in over 67.5% of cases, achieving comparable or better performance while consuming only 10% of the data. Additional analyses reveal: (1) Our method works robustly across different models for difficulty calculation; (2) The optimal data selection ratio falls between 10-15%, and (3) Our approach remains effective even without length normalization. In total, these results establish our approach as both theoretically principled and practically effective for preference data selection of LLM alignment. To summarize, our main contributions are as follows: 1. We propose a novel yet simple data selection method tailored for preference datasets, grounded in the theoretical framework of the DPO implicit reward mechanism to quantify sample difficulty. 2. We provide a theoretical justification for our difficulty metric via gradient analysis, showing that smaller DPO implicit reward gaps correspond to larger gradient magnitudes, indicating higher learning potential. 3. We perform extensive experiments on four diverse preference datasets and two alignment tasks, consistently achieving superior performance using only 10% of the training data, outperforming five strong baselines and matching the performance of full-dataset training. 4. We perform a comprehensive analysis of the method’s robustness under various difficulty computation models, data scaling regimes, and length normalization strategies, further identifying optimal selection ratios and demonstrating the method’s robustness across different settings. 2 Related Work 2.1 Aligning LLM with Human Preferences Achieving alignment between LLMs and human preferences is a fundamental endeavor. A major advancement in this domain has been Reinforcement Learning from Human Feedback (RLHF) DBLP:conf/nips/ChristianoLBMLA17 ; ouyang2022training ; bai2022training , which has played a pivotal role in the fine-tuning of leading LLMs such as GPT-4 openai2023gpt4 , Claude anthropic2023claude , and Gemini team2023gemini series models. The conventional RLHF approach involves training a reward model to evaluate the language

Read the complete text