Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

Nirav Diwan, Tolga Ergen, Dongsub Shim, Honglak Lee · 2025

Read Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning by Nirav Diwan, Tolga Ergen, Dongsub Shim, Honglak Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15109.txt.

Introduction

Four authors find a cheap way to pick the best training pairs for preference learning in this 2025 paper. Direct preference optimization aligns language models with human choices. It works best when the chosen and rejected answers differ clearly in quality, yet finding such pairs usually costs extra models or effort.

The insight is that the reference model's own probability space already flags clear pairs. Their sampling strategy, guided by that signal, improved benchmark scores by small but consistent margins while using only thirty to fifty percent of the training data. Gains were larger on coding, math, and reasoning tasks, reaching nearly a full point across models and settings.

Catalog record

CREATORNirav Diwan, Tolga Ergen, Dongsub Shim, Honglak Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15109

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning Nirav Diwan, Tolga Ergen, Dongsub Shim, Honglak Lee Abstract Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO’s effectiveness relies on training data quality. In particular, clear quality differences between preferred and rejected responses enhance learning performance. Current methods for identifying and obtaining such high-quality samples demand additional resources or external models. We discover that reference model probability space naturally detects high-quality training samples. Using this insight, we present a sampling strategy that achieves consistent improvements ( + 0.1 +0.1 to + 0.4 +0.4 ) on MT-Bench while using less than half ( 30 30 - 50 % 50\% ) of the training data. We observe substantial improvements ( + 0.4 +0.4 to + 0.98 +0.98 ) for technical tasks (coding, math, and reasoning) across multiple models and hyperparameter settings. Introduction Preference learning aims to align Large Language Models (LLMs) with human preferences. It is applied after pre-training and supervised fine-tuning (SFT) to teach models to generate responses that better align with human expectations while preserving knowledge acquired in earlier training stages. This approach has demonstrated practical impact in improving user experience ( Bai et al. 2024 ) , implementing safety filters ( Liu, Sun, and Zheng 2024 ; Huang et al. 2024 ) , and moderating content ( Ma et al. 2023 ) . Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) is a a supervised off-policy method that has recently emerged as a leading approach to preference learning. Unlike on-policy methods, DPO directly optimizes the policy using paired preference data, where each pair contains an aligned (or preferred ) and misaligned (or rejected ) response. By training the model to assign higher probabilities to preferred responses, DPO effectively shapes the model’s output distribution without requiring a separate reward model. This simplicity, combined with strong empirical results, has made DPO increasingly popular for language model alignment. Models learn better when there is a clear distinction between preferred and rejected responses, while noisy or ambiguous preferences hinder learning ( Ivison et al. 2024 ) . This distinction is quantified using preference clarity , measured using ground truth scalar values that indicate each response’s alignment or instruction-following ability. Prior work has explored various approaches to address this challenge: modifying alignment objectives ( Gao, Alon, and Metzler 2024 ) , filtering training data ( Morimura et al. 2024 ) , and improving preference data collection ( Morimura et al. 2024 ) . Figure 1: Relationship between sampling threshold ( \delta ) and preference clarity for Ultrafeedback using the fine-tuned LLAMA-3 8B as the reference model. The solid blue line is the preference clarity between preferred and rejected responses calculated using the difference of Ultrafeedback’s preference scores. The dashed orange line (log scale) shows the available training pairs at each threshold. Preference pairs at a higher sampling threshold show clearer preferences, indicating that the reference model can identify high-quality preference pairs even when it incorrectly attributes the correct response. Creating high-quality preference pairs requires extensive annotation resources. Current datasets like Ultrafeedback ( Cui et al. 2023 ) use GPT-4 for evaluation, but scaling this approach to new alignment datasets increases costs and creates dependencies on external models. The better way to ensure high-quality annotation is to perform manual annotation. However, this demands considerable effort — requiring multiple rounds of review, quality checks, and careful measurement of annotator agreement. These resource limitations drive the need for methods that can identify high-quality preference pairs without expensive labeling processes. We find that the probability space of the reference model serves as a natural detector of preference clarity. When there is strong gap between the probabilities of the two responses — regardless of which it favors — the pair represents a clearer preference signal in the alignment data. This reveals an intriguing property: the reference model can identify high quality preference pairs, even when it may not know which response is better . Using this insight, we propose a reference model-guided sampling method that achieves better performance than the full dataset of sampling method. We validate our findings through extensive experimentation across model architectures and hyperparameter configurations. The improvements remain consistent across these variations, suggesting our method captures fundamental preference signals rather than exploiting model-specific patterns. Using only 30 - 50\% of the original training data, we achieve higher MT-Bench performance ( +0.1 to +0.4 ), with particularly strong gains on technical tasks ( +0.4 to +0.98 for coding, reasoning, and math). Through ablation studies, we identify key factors for effective adoption of this sampling strategy. Methodology Background Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) is a supervised off-policy method used to optimize models based on user preferences without relying on a separate reward model. Instead, DPO directly aligns the policy with preference data by leveraging a ranking-based approach. The objective of DPO is to use the reference policy, typically a SFT model, to guide the optimization process. The preference optimization is defined using pairs of responses, where one response is preferred over the other. By focusing on these response pairs, the goal is to minimize the DPO loss function defined as: \displaystyle\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\Bigg[ \displaystyle\log\sigma\Bigg(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)} \displaystyle-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\Bigg)\Bigg] where \pi_{\theta} is the policy model, \pi_{\text{ref}} is the reference model, \sigma is the sigmoid function, and (x,y_{w},y_{l}) , \beta is the hyperparameter controlling the deviation from the base reference policy, are preference pairs comprising a prompt x , a preferred response y_{w} , and a rejected response y_{l} , drawn from the dataset \mathcal{D} . Preference Clarity It quantifies the degree of quality difference between two responses in a preference pair. Given responses r_{1} and r_{2} , we calculate preference clarity using ground truth quality scores s_{1} and s_{2} assigned to each response. These scores can be obtained either through human annotation or using LLM-as-a-judge frameworks with models like GPT-4. Formally, for a preference pair (r_{1},r_{2}) , the preference clarity is measured as: clarity(r_{1},r_{2})=|s_{1}-s_{2}| (1) where s_{1},s_{2} are the ground truth quality scores. Reference-model based sampling Method. Our sampling strategy leverages normalized reference policy probabilities to identify clear preference signals. For input x with responses y_{w} (preferred) and y_{l} (rejected), we compute the difference between length-normalized reference probabilities. We sample pairs where this difference exceeds threshold \delta : \left|\frac{\log(\pi_{\text{ref}}(y_{w}|x))}{|y_{w}|}-\frac{\log(\pi_{\text{ref}}(y_{l}|x))}{|y_{l}|}\right|\geq\delta (2) where \pi_{\text{ref}}(y|x) is the reference probability and |y| is the sequence length. This normalization enables comparison between responses of different lengths, and focuses training on pairs with larger probability gaps. This simple approach prioritizes training examples with clearer preference signals while downweighting pairs where reference model estimates that the responses are similar. Reference Model as Quality Detector. Our analysis reveals that the sampling strategy effectively identifies high-quality preference pairs. Using LLAMA-3-8B as the reference model, we compute probability differences between preferred and rejected responses in the Ultrafeedback dataset (Figure 1 ). Samples selected at higher thresholds ( \delta ) in Equation 2 consistently demonstrate greater preference clarity. This suggests reference models naturally detect strong training examples without requiring additional annotation. Dataset MT-Bench Performance Model \delta Percentage Score \Delta vs Full Dataset † Mistral 7B SFT – 6.33 -0.63 + \delta\geq 0 100\% 6.96

Read the complete text