Policy-labeled Preference Learning: Is Preference Enough for RLHF?

Taehyun Cho, Seokhun Ju, Seungyub Han, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee · 2025

Read Policy-labeled Preference Learning: Is Preference Enough for RLHF? by Taehyun Cho, Seokhun Ju, Seungyub Han, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06273.txt.

Introduction

Taehyun Cho and co-authors question a common step in training language models from human feedback, in a 2025 paper. The usual method learns a reward score from human choices between answers, then trains the model to chase that score.

The authors note that standard methods treat the compared answers as if an ideal policy had made them, which skews the math. Their method, policy-labeled preference learning, folds in regret, a measure tied to the policy that truly made each answer. A matching penalty term keeps learning stable over long action chains. Tests on hard control tasks showed clear gains in the offline setting. The method also helped when learning online.

Catalog record

CREATORTaehyun Cho, Seokhun Ju, Seungyub Han, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy-labeled Preference Learning: Is Preference Enough for RLHF? Taehyun Cho, Seokhun Ju, Seungyub Han, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee Abstract To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. Inspired by Direct Preference Optimization framework which directly learns optimal policy without explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret , which reflects behavior policy information. We also provide a contrastive KL regularization, derived from regret-based principles, to enhance RLHF in sequential decision making. Experiments in high-dimensional continuous control tasks demonstrate PPL’s significant improvements in offline RLHF performance and its effectiveness in online settings. † † affiliationnotice: Equal contribution 1 Introduction Preference-based reinforcement learning (PbRL), a branch of RLHF, focuses on learning optimal policies directly from human preferences, avoiding the needs for explicit, handcrafted rewards. Unlike traditional RLHF, which infers numerical rewards, PbRL derives reward signals from preference comparisons between trajectory pairs. As RLHF applications expand into complex domains such as large language models (LLMs) ( Achiam et al., 2023 ; Touvron et al., 2023 ; Ye et al., 2024 ; Meng et al., 2024 ; Xiao et al., 2024 ) and robotic manipulation ( Christiano et al., 2017 ; Hwang et al., 2023 ) , aligning agents with human intentions has become increasingly important. Early RLHF research ( Lee et al., 2021 ; Park et al., 2022 ) assumed humans prefer trajectories with higher cumulative rewards, leading to a two-step learning process: (1) training a reward model to align with human preferences and (2) applying a RL algorithm to optimize the policy using the learned reward. Recently, Rafailov et al. (2024b) introduced Direct Preference Optimization (DPO), which bypasses the need for an explicit reward function and directly optimizes the policy based on preferences. This simplification reduces computational complexity and dependency on potentially imperfect reward models, improving training stability. DPO has demonstrated superior performance over reward-based RLHF methods, particularly in fine-tuning LLMs ( Bai et al., 2022 ; Stiennon et al., 2020 ; Ziegler et al., 2019 ) and offline RL benchmarks ( Hejna & Sadigh, 2024 ) . While DPO has shown strong performance in LLM fine-tuning and offline RL benchmarks, its assumptions are largely shaped by the structure of LLM training. Many RLHF studies assume contextual bandits or deterministic Markov decision processes (MDPs), where the next state is determined solely by the previous state (prompt) and action (response), leading to a simplified transition model. However, in standard RL settings, state transitions involve environmental stochasticity , introducing additional uncertainty that complicates both policy optimization and MDP estimation ( Yang et al., 2022 ) . Since outcomes depend on external stochasticity beyond the agent’s control, observed transitions may not always accurately provide the agent’s policy performance. Consequently, inferring optimal behavior from observed sequences becomes more challenging. For example, an agent executing a suboptimal policy might transition to a preferred state with low probability, whereas an optimal policy could occasionally lead to an undesirable state due to environmental randomness. This contrast with the deterministic nature of LLM training, where responses are directly mapped from prompts, highlights a key limitation when applying DPO to general RL problems. This challenge becomes more critical in offline RL, where learning relies on pre-collected datasets rather than direct interaction with environment. When data comes from diverse policies but lacks explicit information about the behavior policies, distinguishing whether outcome quality stems from policy suboptimality or environmental stochasticity becomes difficult. Given this ambiguity, a key question arises: Can preference data generated by diverse policies sufficiently guide sequential decision-making, or is additional information required? To address this question, we propose a novel RLHF framework, Policy-labeled Preference Learning (PPL), which leverages regret-based preference modeling while explicitly labeling the behavior policy. Unlike conventional approaches that rely solely on preferences between trajectory pairs, PPL incorporates policy information directly into the learning process, disentangling the effects of environmental stochasticity and the suboptimality of behavior policies. To provide theoretical insights, we define a reward equivalence class —a set of reward functions that induce the same optimal policy—and derive a bijective mapping that allows regret to be expressed as a function of the optimal policy. We show that, unlike the partial sum of rewards, regret is uniquely defined with respect to a given optimal policy, making it a well-structured metric that mitigates issues related to reward sparsity and enhances the stability of learning. Furthermore, we introduce contrastive KL regularization , which sequentially aligns policies with preferred trajectories while explicitly contrasting them against less preferred ones. Empirically, to consider the fact that real-world offline data often consists of rollouts from diverse policies, we construct homogeneous and heterogeneous datasets in the MetaWorld environment and evaluate performance across various offline datasets. 2 Preliminaries Maximum Entropy Framework. We define the MDP as \mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},r,\gamma) characterized by state space \mathcal{S} , action space \mathcal{A} , transition kernels \mathbb{P} which represents the probability of the next state s^{\prime} given the current state s\in\mathcal{S} and action a\in\mathcal{A} , underlying reward r\in[r_{\text{min}},r_{\text{max}}] , and discount factor \gamma . For notational simplicity, we denote the expectation over trajectories \tau=(s_{0},a_{0},s_{1},a_{1},\cdots) generated by a policy \pi and the transition kernel \mathbb{P} as \mathbb{E}_{\tau\sim\mathbb{P}^{\pi}}[\cdot] . The MaxEnt framework provides an optimal policy which not only maximizes the expected cumulative return, but also the entropy for each visited state: \displaystyle\pi^{*}_{\text{MaxEnt}}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\mathbb{P}^{\pi}}\Big[\sum_{t\geq 0}\gamma^{t}(r(s_{t},a_{t})+\alpha\mathcal{H}^{\pi}(\cdot|s_{t}))\Big], where \mathcal{H}^{\pi}(\cdot|s)=-\mathbb{E}_{\pi}[\log\pi(\cdot|s)] is the entropy of policy \pi at state s . Here, \alpha is a temperature hyperparameter that determine the relative importance of entropy and reward. For clarity, we say \pi^{*}_{\text{MaxEnt}} as \alpha -optimal. In addition, soft Q -function Q^{\pi}(s,a) is defined as the expected cumulative return augmented by an entropy terms, expressed as; \displaystyle Q^{\pi}(s,a)=r(s,a)+\mathbb{E}_{\tau\sim\mathbb{P}^{\pi}}\Big[\sum_{t>0}\gamma^{t}(r(s_{t},a_{t})+\alpha\mathcal{H}^{\pi}(\cdot|s_{t}))\Big]. Analogously, we can derive soft value function V^{\pi}(s) and soft Bellman equation as follows: \displaystyle V^{\pi}(s)=\mathbb{E}_{a\sim\pi}[Q^{\pi}(s,a)-\alpha\log\pi(a|s)], \displaystyle Q^{\pi}(s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim\mathbb{P}}\Big[V^{\pi}(s^{\prime})]\Big] for all state-action pairs (s,a)\in\mathcal{S}\times\mathcal{A} . Note that the interpretation of the value function is modified by involving the entropy term in the MaxEntRL, i.e., V^{\pi}(s)\neq\mathbb{E}_{\pi}[Q^{\pi}(s,a)] . For an \alpha -optimal policy \pi^{*} , Ziebart (2010) derived the relationship between the optimal policy and optimal soft Q -function Q^{\pi^{*}} : \displaystyle\pi^{*}(a|s) \displaystyle=\exp\left(\alpha^{-1}(Q^{\pi^{*}}(s,a)-V^{\pi^{*}}(s))\right), \displaystyle V^{\pi^{*}}(s) \displaystyle=\alpha\log\int_{a\in\mathcal{A}}\exp\left(\alpha^{-1}Q^{\pi^{*}}(s,a)da\right). 2.1 Preference-based Reinforcement Learning Table 1 : Comparison for different preference models under PbRL framework. Algorithm Score Function Direct Preference Optimization Likelihood Matching PEBBLE ( Lee et al., 2021 ) r_{\psi}(s_{t},a_{t}) ✗ ✗ DPO ( Rafailov et al., 2024b ) \log{\pi_{\psi}(y|s)}/{\pi_{\text{ref}}(y|s)} ✓ ✗ DPPO ( An et al., 2023 ) -\mathbb{E}_{a\sim\pi_{\psi}(\cdot|s_{t})}[\|a-a_{t}\|_{2}] ✓ ✗ CPL ( Hejna et al., 2023 ) Q^{\pi_{\psi}}(s_{t},a_{t})-V^{\pi_{\psi}}(s_{t}) ✓ ✗ PPL [Ours] -(V^{\pi_{\psi}}(s_{t})-Q^{{\color[rgb]{1,0,0}\pi}}(s_{t},a_{t})) ✓ ✓ Designing a reward function that accurately aligns with human behaviors is inherently challenging. To address this, PbRL focuses on learning the optimal policy directly from human preferences rather than relying on predefined rewards. In this context, we adopt a reward-free MDP \mathcal{M}\setminus r within the MaxEnt framework. We define a segment \zeta=(s_{0},a_{0},\dots,s_{k},a_{k}) as a sequence sampled from a dataset \mathcal{D} . Specifically, human annotators or AI systems are tasked with comparing pairs of

Read the complete text