The Representation-Rationalizability Tradeoff in Reward Learning
Jing Dong, Yaoliang Yu, Pascal Poupart · 2026
Read The Representation-Rationalizability Tradeoff in Reward Learning by Jing Dong, Yaoliang Yu, Pascal Poupart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00291.txt.
Introduction
Dong, Yu, and Pourpart show a trade-off at the heart of reward learning from picks. In RLHF, folk pick the best of two replies to a prompt. One reward score must then grade all replies. Old choice math says pooled picks can loop in rings. No one score can rank all pairs true in such rings.
Fresh pipes score replies through a learned code first. That code sets which replies count as distinct picks. A rich code shrinks one loss term, the shade term. It swells a next term, by baring more pairs no score can rank true. The split holds for a kin route, DPO, as well. Joint training of code and score can not pledge the sweet spot. Tests on fake and real pick sets back the math.
Catalog record
| CREATOR | Jing Dong, Yaoliang Yu, Pascal Poupart |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00291
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Representation-Rationalizability Tradeoff in Reward Learning Jing Dong, Yaoliang Yu, Pascal Poupart Abstract In RLHF, each training example contains a prompt x x and two candidate responses y , y ′ y,y^{\prime} , and annotators provide pairwise preferences between these responses. The learning problem is to convert these heterogeneous pairwise judgments into a single scalar reward r ( x , y ) r(x,y) that measures response quality for each prompt. Classical social choice implies an impossibility because heterogeneous annotator samples can induce pooled preferences with Condorcet cycles, so no scalar reward can evaluate all compared response pairs consistently. A growing literature analyzes RLHF as a social-choice problem, but usually assumes a fixed finite set of alternatives, i.e., a pre-enumerated finite set of candidate responses for each prompt. Modern pipelines instead score responses through a learned representation ϕ ( x , y ) \phi(x,y) before a scalar head, so ϕ \phi determines which responses are treated as distinguishable alternatives and which comparisons are visible to the reward model. Once this embedding is part of the problem, the impossibility results from social choice theory become a tradeoff. We show that the excess cross-entropy loss of any reward built on ϕ \phi decomposes exactly into a representational term, which a richer ϕ \phi shrinks, and an aggregation term, which a richer ϕ \phi enlarges by exposing more comparisons that no scalar can rank consistently. The same results extend to direct preference optimization (DPO), and jointly training the embedding and the reward cannot guarantee to recover the sweet spot of this tradeoff. Experiments on synthetic data and real preference datasets corroborate our results. 1 Introduction Should reward models use richer representations to better fit human preferences? In modern RLHF ( 20 ) , the aggregation task is to fit a single scalar reward to pooled pairwise preferences from many annotators. A popular architecture uses a pretrained language-model encoder that maps a prompt-response pair (x,y) to features \phi(x,y) , followed by a thin score head g , so r(x,y)=g(\phi(x,y)) . A common intuition is that increasing representational richness can only help because it gives the reward model more signals to fit. We show that this intuition is incomplete. Richer representations can improve fit in one sense while hurting it in another. The reason is that representation changes the aggregation problem itself. The embedding determines which response pairs are distinguishable to the reward fitter, so refining \phi turns previously collapsed responses into distinct alternatives and adds new pairwise comparison that a single scalar reward must rationalize. Classical social choice theory already warns that no scalar reward can consistently aggregate pooled judgments from heterogeneous annotators when their combined comparisons contain cycles 1 ; 11 ; 25 , and a recent body of work has translated these social choice results into the RLHF setting 5 ; 7 ; 30 ; 16 . Most of these works, however, treat the response space as a fixed finite set of alternatives. Once the embedding is allowed to vary, the social choice impossibility result becomes tunable because changing \phi changes which annotator disagreements are visible in the pooled comparisons. Our main result makes this trade-off precise. Under a random-utility model with arbitrary annotator heterogeneity, the excess cross-entropy loss of any embedding-based reward decomposes into two parts: an embedding loss measuring information that \phi discards by collapsing distinguishable preferences, and an agreement cost that measures how far the pooled preferences, as seen through \phi , are from being rationalizable by any scalar reward. Under a mild assumption on the representation, the lower bounds of the two terms move in opposite directions as \phi refines. As a consequence, both the smallest and the largest representations can be suboptimal, and the optimal embedding dimension is dataset-dependent. We further show that this lower bound is tight under a constructive reward. Our framework extends beyond reward modeling. We show that Direct Preference Optimization (DPO) ( 23 ) admits a similar decomposition and excess-loss lower bound, demonstrating that the same tradeoff governs preference-based policy learning even without an explicit reward model. Based on the excess loss lower bound results, one might hope that jointly training the embedding and reward can naturally converge to an optimal embedding. However, we show that the joint embedding-reward training is not guaranteed to decrease either tradeoff term per round and admits only fixed points where the reward is constant on the data. We complement the theory with experiments on synthetic data and three real preference datasets (Jester, Sushi, and MT-Bench) using two pretrained sentence encoders. Across settings, the embedding loss decreases and the agreement cost increases with embedding resolution, and total excess loss is often minimized at an intermediate dimension, consistent with our bounds. 2 Related Work Alignment with Social Choice Social choice theory offers a rich framework for analyzing how RLHF aggregates diverse human preferences. The optimal solution to the Bradley-Terry loss in RLHF induces a ranking equivalent to the Borda count from social choice theory ( 26 ) . Under the assumption that each pairwise comparison is labeled by a single evaluator, RLHF satisfies pairwise majority and Condorcet consistency ( 30 ) . More fundamentally, as the annotator pool grows, the probability that the preference profile contains a Condorcet cycle approaches one exponentially fast ( 16 ) , so reward-based alignment is statistically fragile under general preferences. In the case of a linear social choice model, where voters are linear directions in fixed embedding space, 10 analyze the ranking induced by an optimal linear reward with respect to Pareto optimality (PO) and Pairwise Majority Consistency (PMC). 13 show that an appropriate reweighting of the Bradley-Terry loss can enforce PO and PMC, and further analyze clone robustness. RLHF Reward Modeling, and Failure Modes Reward models are central to RLHF yet remain difficult to train reliably. 4 give a survey summarizing limitations of RLHF, including reward hacking, reward misgeneralization, distribution shift, and the difficulty of representing diverse human values with a single reward. 29 identify noisy preference pairs and out-of-distribution generalization as practical challenges. A number of works identify reward overfitting and overoptimization in reward-model training and propose alternative training methods to standard RLHF ( 22 ; 34 ; 31 ; 6 ) . 9 quantify how proxy reward scores diverge from true rewards as optimization pressure increases, following smooth scaling laws. On the theoretical side, 35 show that with a linear reward model and a Bradley-Terry (BT) model, MLE can converge. A number of works propose alternatives to the Bradley-Terry model as scalar rewards cannot capture intransitive preferences ( 19 ; 2 ; 32 ) . 26 show that implicitly aggregating over hidden contexts during reward modeling can produce counter-intuitive results very different from aggregation via expected utility. 28 establish the asymptotic convergence rate of BT reward models based on deep neural networks using embeddings. 3 RLHF Pipeline We now introduce the setup used throughout the paper. We consider the standard RLHF pipeline with n human annotators, denoted as \{1,\dots,n\} , a prompt space {\mathcal{X}} , and a response space {\mathcal{Y}} . Preference Model Fix a prompt x\in{\mathcal{X}} and an annotator i\in\{1,\dots,n\} . Annotator i has a latent utility u_{i}(x,y)\in\mathbb{R} for each response y\in{\mathcal{Y}} . Given two candidate responses y,y^{\prime}\in{\mathcal{Y}}