PILAF: Optimal Human Preference Sampling for Reward Modeling

Yunzhen Feng · 2025

Read PILAF: Optimal Human Preference Sampling for Reward Modeling by Yunzhen Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04270.txt.

Introduction

Yunzhen Feng and co-authors propose a smarter way to sample answers for human preference training in this 2025 paper. Reward models learn from labeled comparisons, yet which responses get labeled is usually left to chance. Their method, PILAF, picks samples to serve the final goal of matching hidden human values.

The strategy is grounded in theory from both optimization and statistics, where it is shown to be optimal. It works by interpolating between policies when drawing responses for labeling. In repeated and online training settings, it performed strongly. Feedback curation matters most exactly there, the authors note. The method is simple to implement on top of existing pipelines. They offer it as a principled upgrade for aligning language models.

Catalog record

CREATORYunzhen Feng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE141 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04270

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PILAF: Optimal Human Preference Sampling for Reward Modeling Yunzhen Feng Abstract Abstract As large language models increasingly drive real-world applications, aligning them with human values becomes paramount. Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible. In practice, RLHF mostly relies on approximate reward models, which may not consistently guide the policy toward maximizing the underlying human values. We propose Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for preference labeling that explicitly aligns preference learning with maximizing the underlying oracle reward. PILAF is theoretically grounded, demonstrating optimality from both an optimization and a statistical perspective. The method is straightforward to implement and demonstrates strong performance in iterative and online RLHF settings where feedback curation is critical. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) has revolutionized large language models (LLMs) by incorporating human preferences, enabling significant progress in applications such as conversational AI ( Achiam et al., 2023 ) , personalized tutoring ( Limo et al., 2023 ) , and content curation ( Yue et al., 2024 ) . At the core of RLHF is reward modeling , a critical process that translates human feedback—such as pairwise comparisons or rankings—into a measurable objective for model training. By formalizing human preferences, reward models then guide LLMs towards alignment through policy optimization . † † footnotetext: † Correspondence to: Yunzhen Feng ( [email protected] ). † † footnotetext: ∗ Joint second authors. † † footnotetext: ⋄ Joint senior authors While numerous studies have focused on improving language models (LMs) by optimizing fixed reward functions ( Dong et al., 2023 ; Liu et al., 2024c ) or leveraging pre-existing preference datasets ( Ethayarajh et al., 2024 ; Azar et al., 2024 ; Xu et al., 2024 ) , comparatively less attention has been paid to the critical challenge of collecting effective data for human-labeling in RLHF, to maximize its utility. This is an important problem, as the quality of preference data directly impacts the effectiveness of reward modeling and, consequently, the overall success of fine-tuning. This challenge is further compounded by the high cost of expert preference labeling ( Lightman et al., 2023 ) . Preference data is usually generated by sampling response pairs (\vec{\boldsymbol{y}}\!\,^{a}_{i},\vec{\boldsymbol{y}}\!\,^{b}_{i}) to a prompt x_{i} from a policy, and presenting them to human labelers for preference annotation. It is commonly assumed that the annotation follows the Bradley-Terry (BT) model, under an oracle reward . Next, we use maximum likelihood estimation (MLE) on these preference data to train a reward model, which then serves as a measurable objective to optimize the policy (i.e. LLM) while staying close to a reference policy. In Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , this pipeline is simplified by optimizing the policy with implicit reward modeling. However, all these pipelines give rise to a misalignment of objectives: RLHF (or DPO) should, in principle, train its policy to maximize the (inaccessible) oracle objective which combines the oracle reward from the BT model with reference regularization. In practice, RLHF relies on preference data through the MLE objective in reward modeling or through methods like DPO, which are not designed to guide policy optimization towards maximizing oracle rewards. Thus, reward optimization (either directly or implicitly via DPO) and (optimal) policy optimization are not inherently aligned, potentially leading to inefficiencies (Sec. 2 ). In this work, we study this misalignment by examining the sampling scheme that generates response pairs (\vec{\boldsymbol{y}}\!\,^{a}_{i},\vec{\boldsymbol{y}}\!\,^{b}_{i}) for preference labeling, which is especially important when additional preference data is collected mid-RLHF training to mitigate the off-policy distributional shift, as is empirically standard ( Touvron et al., 2023 ; Bai et al., 2022 ) . We show that uniform sampling from the current policy, as is common, leads to misaligned gradients of the two objectives (reward model loss and true oracle objective). Figure 1: Overview of our approach . (a) We consider a full RLHF training setup, where a language model (LM) policy is iteratively refined through active data collection. Our goal is to develop an optimal response sampling method for preference labeling. (b) We introduce PILAF, which generates responses by interpolating between the current policy and a reference policy, balancing exploration and exploitation. (c) Our theoretical analysis shows that T-PILAF aligns the parameter gradient with the steepest direction for maximizing human values and achieves more favorable convergence in regions of high sensitivity. To tackle this issue, we present Theoretically Grounded Policy-Interpolated Learning for Aligned Feedback (T-PILAF), a novel sampling method that aligns reward modeling with value optimization. Specfically, T-PILAF generates responses by interpolating the policy model and the reference model for a balanced exploration and exploitation. We provide rigorous theoretical analysis to show that for preference data generated with T-PILAF, the gradient of the MLE loss with respect to the policy network’s parameters is aligned with the policy gradient of the oracle objective in a first-order sense. This alignment enables the policy to optimize directly for the oracle value, achieving both alignment and efficiency. Furthermore, we separately show from a statistical perspective that T-PILAF aligns optimization with the steepest directions of the oracle objective. It thus makes the sampled preference pairs more informative, reducing variance and improving training stability. We then present PILAF, a simple modification of our theoretical sampling scheme T-PILAF, which naturally lends itself to practical implementation. For clarity of exposition, we present our method in the context of DPO; however, PILAF can be adapted to a wide class of preference optimization methods. 1 1 1 See Appendix F for the extension to PPO. See Figure 1 for an illustration of our setup, method, and the optimization and statistical principles underlying PILAF. We conduct extensive experiments to validate PILAF’s effectiveness and robustness. As a stand-in for expensive human annotators, we use a well-trained reward model—Skywork-Llama-3.1-8B ( Liu et al., 2024a ) —as a proxy for the oracle reward. Throughout training, we query this model exclusively for preference labels, simulating human feedback. We then align the Llama-3.1-8B base model ( Dubey et al., 2024 ) using these proxy-labeled preference data in two settings: iterative DPO ( Xiong et al., 2024 ) and online DPO ( Guo et al., 2024 ) . In both scenarios, preference data is collected on-the-fly, either after each full training epoch in the iterative setting or after every training step in the online setting. Across all configurations, PILAF outperforms all the baselines, producing a policy with higher reward (as measured by the proxy) and a lower KL divergence from the reference model, reducing annotation and computation costs by over 40% in iterative DPO. Our key contributions are as follows: • (Practical sampling algorithm) We propose PILAF ( Section 5 ), an efficient sampling algorithm for generating response pairs in the RLHF pipeline for improved sample efficiency and performance, derived from its theoretically grounded variant T-PILAF ( Section 3 ). • (Theoretical optimality) We provide theoretical guarantees for the efficiency of our approach from both optimization and statistical perspectives ( Section 4 ). • (Empirical validation) We validate PILAF in both iterative and online DPO settings ( Section 6 ) and observe that it

Read the complete text