Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model

Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, Yue Zhang · 2025

Read Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model by Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, Yue Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15843.txt.

Introduction

Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang published this paper in 2025. It improves Direct Preference Optimization, a popular way to tune language models on human preferences without training a separate reward model. Their insight concerns the reference model inside the method, which most recipes set identical to the model being trained.

That twin setup wastes data: the reference acts as a weighting dial, and starting both models equal mis-sets the dial. Their method, Pre-DPO, uses a guiding reference that already glimpses the best reachable policy. Training then leans harder on suitable examples and softer on poor fits. On two public benchmarks, the approach lifted both parent methods, with no extra data and no outside models. The authors pitch it as a drop-in change for preference tuning pipelines.

Catalog record

CREATORJunshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, Yue Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, Yue Zhang Abstract Direct Preference Optimization (DPO) simplifies reinforcement learning from human feedback (RLHF) for large language models (LLMs) by directly optimizing human preferences without an explicit reward model. We find that during DPO training, the reference model plays the role of a data weight adjuster. However, the common practice of initializing the policy and reference models identically in DPO can lead to inefficient data utilization and impose a performance ceiling. Meanwhile, the lack of a reference model in Simple Preference Optimization (SimPO) reduces training robustness and necessitates stricter conditions to prevent catastrophic forgetting. In this work, we propose Pre-DPO , a simple yet effective DPO-based training paradigm that enhances preference optimization performance by leveraging a guiding reference model . This reference model provides foresight into the optimal policy state achievable through the training preference data, serving as a guiding mechanism that adaptively assigns higher weights to samples more suitable for the model and lower weights to those less suitable. Extensive experiments on AlpacaEval 2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently improves the performance of both DPO and SimPO, without relying on external models or additional data. 1 1 1 Our code and data can be found at https://github.com/DtYXs/Pre-DPO . 1 Introduction Reinforcement learning from human feedback (RLHF) has become a crucial paradigm in aligning large language models (LLMs) with human preferences and values ( Christiano et al. 2017 ; Ouyang et al. 2022 ; Achiam et al. 2023 ) . This approach involves training an explicit reward model to provide reward signals and performing reinforcement learning under a KL constraint by a reference model instantiated identically to the initial policy. Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) , as a representative of reference-based preference optimization methods ( Rafailov et al. 2023 ; Ethayarajh et al. 2024 ; Azar et al. 2024 ; Gorbatovski et al. 2024 ) , transforms the objective of RLHF into direct preference training of the language model, enabling offline preference learning under the constraint of the reference model. Recently, it has been shown that a reference model is not necessary for achieving effective preference optimization. Simple Preference Optimization (SimPO) ( Meng et al. 2024 ) , as a representative of reference-free preference optimization methods ( Xu et al. 2023 ; Xu et al. 2024 ; Hong et al. 2024 ; Meng et al. 2024 ), eliminates the need for a reference model and yields better performance and efficiency, though at the cost of an increased risk of catastrophic forgetting ( Meng et al. 2024 ) . Meanwhile, other studies have empirically demonstrated that DPO can benefit either from relaxing the constraints imposed by the reference model ( Gorbatovski et al. 2024 ) or from stronger external reference models ( Liu et al. 2024 ) . However, decreasing reliance on a reference model imposes stricter practical requirements to ensure effective learning ( Meng et al. 2024 ) , and clear methodologies for obtaining an appropriate reference model are still lacking. Despite the empirical efforts, the role of the reference model in DPO and its impact on training dynamics remain insufficiently explored. We find that during DPO training, the reference model plays the role of a data weight adjuster (see Section 4.1 ). It adaptively tends to assign higher weights to data aligned with itself while reducing weights for conflicting data. However, due to the common practice in DPO of initializing the policy and reference models identically ( Rafailov et al. 2023 ) , as training progresses, the reference model increasingly constrains the policy model by penalizing deviations, potentially introducing a performance ceiling. Moreover, identical initialization of policy and reference models results in the nearly uniform weighting of training examples during the early training stages, which is in contrast to prior studies showing that assigning non-uniform weights to training data can lead to improved learning and performance ( Lin et al. 2017 ; Ren et al. 2018 ; Shu et al. 2019 ) . In light of the limitations of conventional reference models, we hypothesize that an effective reference model for DPO should (1) differ from the initial policy model and (2) provide insights into potential directions for policy improvement based on the preference data. We define this type of reference model as a guiding reference model , which can better support learning by transforming the role of the reference model from a static constraint into a guide with foresight (see Section 4.2 ). Building on this insight, we propose Pre-DPO, a simple yet effective training paradigm that enhances data utilization and improves the performance of existing preference optimization methods without relying on external models or additional data. Pre-DPO first optimizes the initial policy using a standard preference optimization method (e.g., DPO or SimPO). The resulting optimized policy is then employed as the guiding reference model. Finally, the initial policy is re-optimized using DPO with this guiding reference model, yielding a better-optimized policy through more effective data reweighting. The guiding reference model in Pre-DPO essentially serves as an adaptive guiding mechanism that dynamically assigns higher weights to samples more suitable for the model and lower weights to those less suitable. Specifically, these suitable cases typically correspond to examples that are easier to learn, allowing the model to leverage data that aligns well with its learning trajectory efficiently. Figure 1 : Pre-DPO further enhances the performance of DPO and SimPO by leveraging a guiding reference model, consistently across different models and scales. Our extensive experiments results conducted on the Llama3.2 ( Grattafiori et al. 2024 ) and Qwen2.5 ( Yang et al. 2024 ) model series demonstrate that Pre-DPO can consistently improve the performance of both DPO and SimPO on the AlpacaEval 2 ( Li et al. 2023 ; Dubois et al. 2024 ) and Arena-Hard v0.1 ( Li et al. 2024 ) benchmarks, achieving average improvements of 2.5 points in the AlpacaEval 2 length-controlled win rate (LC) and 2.6 points in the Arena-Hard v0.1 win rate (WR). By introducing the guiding reference model, Pre-DPO can further improve the performance of existing well-tuned preference optimization methods (as illustrated in Figure 1 ), effectively overcoming the performance ceiling caused by inefficient data utilization under traditional reference model settings. Notably, Pre-DPO does not rely on external models or additional data, making it highly flexible and easy to deploy. 2 Related work RLHF is currently an effective technique for aligning LLMs with human values and preferences ( Ouyang et al. 2022 ; Achiam et al. 2023 ; Grattafiori et al. 2024 ; Yang et al. 2024 ; Shen et al. 2025 ) . Typically, pretrained LLMs first undergo SFT to acquire the ability to follow human instructions. Subsequently, preference data is annotated by human or AI annotators to train a reward model. Finally, policy optimization is performed using reinforcement learning methods, such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) , Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , REINFORCE++ ( Hu 2025 ) , etc. Although RLHF has demonstrated superior performance, the entire pipeline is complex and

Read the complete text