Capturing Individual Human Preferences with Reward Features

André Barreto, Vincent Dumoulin, Yiran Mao, Mark Rowland, Nicolas Perez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, Hugo Larochelle · 2025

Read Capturing Individual Human Preferences with Reward Features by André Barreto, Vincent Dumoulin, Yiran Mao, Mark Rowland, Nicolas Perez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, Hugo Larochelle (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17338.txt.

Introduction

André Barreto and colleagues argue that one reward model for all humanity is a design error, and build the alternative. Feedback-trained systems usually compress everyone's preferences into a single reward function. Where disagreement runs deep, as in language model training, that average can represent nobody. Their framework learns reward features: general components that combine linearly into each person's own reward.

A probably approximately correct bound ties the approximation error to both the number of examples and the number of human raters. Theory then guides how pairwise preference data should be collected. Experiments with language models confirm the analysis: the adaptive model's edge grows with more raters and more disagreement. Fast personalization follows, even for people whose tastes never appeared in training. The architecture turns pluralism from noise into structure.

Catalog record

CREATORAndré Barreto, Vincent Dumoulin, Yiran Mao, Mark Rowland, Nicolas Perez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, Hugo Larochelle
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Capturing Individual Human Preferences with Reward Features André Barreto, Vincent Dumoulin, Yiran Mao, Mark Rowland, Nicolas Perez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, Hugo Larochelle Abstract Reinforcement learning from human feedback usually models preferences using a reward function that does not distinguish between people. We argue that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We formalise and analyse the problem of learning a reward model that can be specialised to a user. Using the principle of empirical risk minimisation, we derive a probably approximately correct (PAC) bound showing the dependency of the approximation error on the number of training examples, as usual, and also on the number of human raters who provided feedback on them. Based on our theoretical findings, we discuss how to best collect pairwise preference data and argue that adaptive reward models should be beneficial when there is considerable disagreement among users. We also propose a concrete architecture for an adaptive reward model. Our approach leverages the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models illustrating our theoretical results and comparing the proposed architecture with a non-adaptive baseline. Consistent with our analysis, the benefits provided by our model increase with the number of raters and the heterogeneity of their preferences. We also show that our model compares favourably to adaptive counterparts, including those performing in-context personalisation. 1 Introduction Reinforcement learning from human feedback (RLHF) can be useful when we do not have an explicit reward function but can distinguish between good and bad behaviour [ 15 ] . This is often the case in the training of large language models (LLMs), in which RLHF has been applied with great success [ 46 , 48 , 41 , 58 ] . The most common way to collect human feedback is to ask humans to rank examples of an agent’s behaviour. Usually, all the feedback is integrated to derive a single reward function that reflects as well as possible the preferences of the population of interest. Although this is a sensible strategy when preferences are homogeneous across the population, it may be less effective when there is considerable disagreement. This is likely the case in the training of LLMs. As an illustration, suppose that, given a pair of alternative responses to a subjective question, 51\% of the target audience prefer the first option while the remaining 49\% prefer the second. If we do not distinguish between users, we are left with two options: either we pick the preferred answer and leave 49\% of the users unhappy 100\% of the time, or we sample the answers proportionally to how often they are preferred and leave 100\% of the users unhappy approximately half of the time. Both solutions are clearly unsatisfactory. We need reward models that can be specialised to users. Crucially, we want the model to be able to adapt to people outside of the group who provided feedback for training. In this paper we formalise and analyse the problem of learning a reward model with this ability. Using the principle of empirical risk minimisation, we derive a probably approximately correct (PAC) bound that shows how the approximation error depends not only on the number of training examples, but also on the number of human raters who provided feedback on them. This is an interesting problem from a theoretical standpoint because the data involved is not identically and independently distributed (i.i.d.). The resulting analysis provides a formal framework to discuss strategies for pairwise preference data collection and to assess the trade-offs associated with the use of an adaptive reward model. We then turn our attention to the question of how to adapt a reward model to a new user in practice. Generalising to unseen users is challenging because it requires one to capture the subjective criteria that underlie human preferences—a non-trivial endeavour, as humans often cannot fully articulate the reasons why they prefer one behaviour over the other. A possible approach is to leverage precisely the type of pairwise preference data used in RLHF, since it allows the preference criteria to be indirectly elicited rather than explicitly spelled out. Building on our theoretical findings, we propose a simple, principled method to unveil the preference criteria underlying the RLHF data. Our model leverages the fact that individual preferences can be captured as a linear combination of a set of general reward features . These features are unknown (in some cases to the person themself), so we need a systematic way to extract them from the data. We show how this can be accomplished using a very simple architecture and training. During the regular RLHF process, we use data coming from different individuals to learn common features that capture the preferences of the group. When the reward model is being specialised to an unknown user, the features are frozen, and only the coefficients of the linear combination must be learned. This results in a simple classification problem that can be reliably solved with a few training examples provided by the user. 2 Background Our goal is to use human preference data to learn a reward function that can be used in RL to learn a policy or LLM [ 15 , 56 ] . We will focus on and adopt the terminology from LLMs, although our analysis and proposed approach are more general. Let \mathcal{T} be a finite set of symbols called tokens , let \mathcal{X}\coloneq\cup_{i=0}^{l_{x}}\mathcal{T}^{i} be the context space , where l_{x} is the maximum context length, and let \mathcal{Y}\coloneq\cup_{i=0}^{l_{y}}\mathcal{T}^{i} be the response space . An LLM is a mapping from \mathcal{X} to a distribution over \mathcal{Y} . Let \mathcal{H} be the set of human users we are interested in. Given a context x\in\mathcal{X} and two responses (y,y^{\prime})\in\mathcal{Y}^{2} , we can ask a user h\in\mathcal{H} for their preferred option. We use y\succ y^{\prime}|x,h to indicate that h prefers y over y^{\prime} given x [ 47 , 2 ] . We encode this outcome with the value z=1 , and encode the opposite preference as z=0 . We want to model humans’ preferences as accurately as possible. Define \mathcal{Z}\coloneq\{0,1\} and let \mathcal{D} be a distribution over \mathcal{H}\times\mathcal{X}\times\mathcal{Y}^{2}\times\mathcal{Z} . We will use subscripts to refer to the marginal distributions of \mathcal{D} and superscripts to indicate conditioning. For example, \mathcal{D}^{h}_{\mathcal{X}} is the marginal distribution over \mathcal{X} conditioned on H=h . We will use numeric superscripts to indicate the joint distribution induced by independent draws from the same distribution; for instance, (\mathcal{D}^{h}_{\mathcal{X}})^{n} is the distribution over \mathcal{X}^{n} resulting from n independent draws from \mathcal{D}^{h}_{\mathcal{X}} . We will formalise the problem through the lens of empirical risk minimisation [ 51 ] . Let \mathcal{C}_{0}\coloneq\{\mathcal{X}\times\mathcal{Y}^{2}\rightarrow[0,1]\} be a classifier set (or a hypothesis class ) and let \ell:[0,1]\times\mathcal{Z}\rightarrow\mathbb{R} be an appropriately defined loss function . Our goal is to minimise the generalisation loss defined as L_{\mathcal{D}}(c)\coloneq\mathbb{E}_{H,X,Y,Y^{\prime},Z\sim\mathcal{D}}[\ell(c(X,Y,Y^{\prime}),Z)] . To do so, we will

Read the complete text