Value-Free Policy Optimization via Reward Partitioning

Bilal FAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected] · 2025

Read Value-Free Policy Optimization via Reward Partitioning by Bilal FAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13702.txt.

Introduction

Bilal Faye and co-authors introduce a 2025 method for tuning language models with simple scores. Each reply to a prompt gets a single reward score, rather than a pair to compare. Older methods in this line must also learn a value model. That extra step adds noise and makes training less steady.

The new method, called RPO in the paper, skips the value model. It scales scores with a rule drawn from the scores for each prompt. The result is a plain training target with no extra models. Tests on several model types beat strong rivals, with replies judged better and less harmful. Stability checks also favored the new method.

Catalog record

CREATORBilal FAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13702

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Value-Free Policy Optimization via Reward Partitioning Bilal FAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected] Abstract Single-trajectory reinforcement learning (RL) methods aim to optimize policies from datasets consisting of (prompt, response, reward) triplets, where scalar rewards are directly available. This supervision format is highly practical, as it mirrors real-world human feedback, such as thumbs-up/down signals, and avoids the need for structured preference annotations. In contrast, pairwise preference-based methods like Direct Preference Optimization (DPO) rely on datasets with both preferred and dispreferred responses, which are harder to construct and less natural to collect. Among single-trajectory approaches, Direct Reward Optimization (DRO) has shown strong empirical performance due to its simplicity and stability. However, DRO requires approximating a value function, which introduces several limitations: high off-policy variance, coupling between policy and value learning, and a lack of absolute supervision on the policy itself. We introduce Reward Partitioning Optimization (RPO) , a new method that resolves these limitations by removing the need to model the value function. Instead, RPO normalizes observed rewards using a partitioning approach estimated directly from data. This leads to a straightforward supervised learning objective on the policy, with no auxiliary models and no joint optimization. RPO provides direct and stable supervision on the policy, making it robust and easy to implement in practice. We validate RPO on scalar-feedback language modeling tasks using Flan-T5 encoder-decoder models. Our results demonstrate that RPO outperforms existing single-trajectory baselines such as DRO and Kahneman-Tversky Optimization (KTO). These findings confirm that RPO is a simple, effective, and theoretically grounded method for single-trajectory policy optimization. Our code is publicly available at https://github.com/b-faye/rpo . I Introduction The dominant approach for aligning large language models (LLMs), whether through reinforcement learning from human feedback (RLHF) [ 1 , 2 , 3 ] or direct preference optimization, is to learn from preference or reward data [ 4 , 5 , 6 , 7 ] . In RLHF, models are first trained via supervised fine-tuning (SFT) [ 8 ] , followed by learning a reward model (RM) from human-labeled preference pairs (e.g., two responses to the same prompt ranked by quality), and finally aligned with reinforcement learning (e.g., PPO [ 9 ] , TRPO [ 10 ] ) to maximize the RM’s score. This pipeline is central to the training of models such as InstructGPT [ 11 ] , GPT-4 [ 12 ] , and Claude 3 [ 13 ] . While effective, RLHF presents key limitations: collecting preference data and training reward models is costly, and the RL step is sample-inefficient and brittle. To reduce reliance on human annotation, recent work has proposed Reinforcement Learning from AI Feedback (RLAIF), where feedback is generated by models rather than humans. RLAIF includes strategies such as distilling AI feedback into a reward model [ 14 ] , using an LLM directly as a reward function [ 15 , 16 ] , or leveraging self-evaluation [ 17 , 18 ] . These techniques offer scalability but still depend on large models as evaluators, reintroducing cost and inference complexity. A parallel line of work avoids reward models and RL altogether, by directly optimizing policies from preference data via supervised learning. These methods—such as DPO [ 4 ] , GPO [ 5 ] , and SLiC [ 19 ] —contrast response pairs to a prompt and define a preference-based loss. This strategy enables simple and stable training, and has been adopted in models like LLaMA 3 [ 20 ] , Qwen 2 [ 21 ] , and Nemotron-4 [ 22 ] . However, collecting high-quality pairwise preference data is increasingly difficult. As LLMs improve, distinguishing between good responses becomes subtler, demanding more qualified annotators and driving up labeling costs. This makes scaling preference-based methods impractical at the frontier. To address this, recent work proposes single-trajectory learning, where data consists of a prompt, a response, and a scalar reward (e.g., thumbs-up/down). This format is simpler and more natural for annotation. Methods such as KTO [ 7 ] and DRO [ 6 ] leverage this setup to train policies directly from reward-labeled completions. Among them, DRO has shown strong empirical results by regressing the optimality condition from KL-regularized RL. However, DRO suffers from several limitations: it requires learning a value function from off-policy data, the joint optimization over policy and value is unstable, and the policy is only supervised relatively to a reference model, not directly by reward. We introduce Reward Partitioning Optimization (RPO), a new method for single-trajectory policy optimization. RPO eliminates the need to estimate a value function by computing an empirical normalization over observed rewards. This enables a direct supervised loss on the policy, using only prompt-response-reward triplets and a reference model. RPO is simple to implement, avoids instability from value approximation, and provides absolute reward supervision. It is fully offline-compatible and scalable to large datasets. Our contributions are: • We analyze theoretical and practical limitations of DRO for single-trajectory learning. • We propose RPO, a new method that replaces value estimation with empirical reward normalization. • We provide a formal derivation showing RPO approximates the optimal policy structure. • We validate RPO on instruction-tuning tasks using T5 encoder-decoder models [ 23 ] , demonstrating strong performance against DRO and KTO baselines. II Background In this section, we review existing paradigms for aligning large language models (LLMs) with human feedback, focusing on three main categories: reinforcement learning from human feedback (RLHF), direct preference optimization (DPO-style), and direct reward optimization (DRO-style). We examine their assumptions, limitations, and motivation for a more general framework that can directly leverage scalar reward data. II-A Reinforcement Learning from Human Feedback (RLHF) RLHF has become the dominant paradigm for aligning large language models (LLMs) with human preferences. It proceeds in three main stages: • Supervised Fine-Tuning (SFT): A reference policy \pi_{\text{ref}} is trained using standard supervised learning to mimic high-quality human-written responses. • Reward Model (RM) Training: Given a dataset \mathcal{D} of pairwise preferences (x,y^{\text{win}},y^{\text{lose}}) —where x is a prompt, and y^{\text{win}}\succ y^{\text{lose}} indicates a preferred response—the goal is to learn a scalar reward function r_{\phi}(x,y) that reflects human preferences. This is typically modeled using a Bradley-Terry [ 24 ] formulation: \footnotesize\mathbb{P}[y^{\text{win}}\succ y^{\text{lose}}\mid x]=\sigma(r_{\phi}(x,y^{\text{win}})-r_{\phi}(x,y^{\text{lose}})), (1) where \sigma is the logistic sigmoid. The reward model is trained by minimizing the negative log-likelihood: \footnotesize\mathcal{L}_{\text{RM}}(r_{\phi})=\mathbb{E}_{(x,y^{\text{win}},y^{\text{lose}})\sim\mathcal{D}}[-\log\sigma(r_{\phi}(x,y^{\text{win}})-r_{\phi}(x,y^{\text{lose}}))].\\ (2) • Policy Optimization: A policy \pi_{\theta} is then optimized to generate outputs that maximize the learned reward r_{\phi}(x,y) while staying close to the reference policy \pi_{\text{ref}} . This is formalized as the following Kullback-Leibler (KL)-regularized objective: \footnotesize\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot|x)}\left[r_{\phi}(x,y)-\tau\cdot\text{KL}(\pi_{\theta}(y|x)\parallel\pi_{\text{ref}}(y|x))\right], (3) where \tau>0 is a regularization coefficient that balances reward maximization and policy deviation. Since this objective is not differentiable with respect to \pi_{\theta} , reinforcement learning algorithms such as Proximal Policy Optimization (PPO) [ 9 ] are used to perform the optimization. Despite its success, RLHF is resource-intensive and challenging to implement. Training the reward model demands significant human supervision and is susceptible to out-of-distribution generalization errors that can mislead the policy. Furthermore, reinforcement learning algorithms add substantial computational overhead and are sensitive to hyperparameter tuning. These drawbacks have spurred interest in alternative alignment strategies that bypass explicit reinforcement learning. II-B Direct Preference Optimization (DPO-style) To address the high cost and complexity of full reinforcement learning, a family of recent approaches—such

Read the complete text