Reward Models Inherit Value Biases from Pretraining
Brian Christian, Jessica A.F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska · 2026
Read Reward Models Inherit Value Biases from Pretraining by Brian Christian, Jessica A.F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20838.txt.
Introduction
Brian Christian and six co-authors show that the judges of AI carry biases from their own upbringing. Their 2026 paper studies reward models, the scorers at the heart of AI alignment. Reward models are central to aligning large language models with human values. Yet they have received less attention than the models they guide. A reward model starts life as a pretrained language model. It therefore inherits representations that shape its behavior.
The authors ask how deep that inheritance runs. Values absorbed during pretraining may tilt what the reward model favors. That tilt then steers every model trained under its scores. The paper examines the nature and extent of this influence. Its findings matter for anyone who treats reward models as neutral judges. They begin biased, and the study measures the bias.
Catalog record
| CREATOR | Brian Christian, Jessica A.F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reward Models Inherit Value Biases from Pretraining Brian Christian, Jessica A.F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska Abstract Reward models (RMs) are central to aligning large language models (LLMs) with human values but have received less attention than pre-trained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the “Big Two” psychological axes, we show a robust preference of Llama RMs for “agency” and a corresponding robust preference of Gemma RMs for “communion.” This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pre-trained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and alignment efforts at the pretraining stage, and makes clear that open-source developers’ choice of base model is as much a consideration of values as of performance. 1 Introduction Reward models (RMs) play a key role in aligning large language models (LLMs) with human preferences and values. Reward modeling can be “explicit,” relying on a reinforcement learning–based approach for learning from human feedback (RLHF; 9 ), or “implicit,” directly increasing the probability of human-preferred data through a cross-entropy objective ( 39 ) . Despite their central importance in AI safety, RMs have received relatively less attention than both pre-trained and post-trained LLMs. This has recently started to change with the increased availability of human preference data ( 2 ; 24 ; 19 ) , of open-weight RMs, and of public RM benchmarks ( 23 ; 29 ) . Recent work on RM interpretability has focused on how RMs may be used to intentionally bias post-trained models towards specific preferences – e.g., model personalization ( 26 ; 45 ; 44 ) – or on how RMs may unintentionally introduce bias in post-trained LLMs ( 43 ; 4 ; 22 ) . However, RMs are typically initialized from LLMs before being finetuned for preference modeling, and no work to date has looked at how RMs themselves can be biased by the LLMs from which they are built. This is a particularly worrying knowledge gap in light of recent research highlighting the importance of pretraining choices in model misalignment ( 28 ; 34 ; 7 ) . Given RMs’ key role in alignment pipelines, it is crucial to understand their vulnerability to potential sources of value bias from pretraining. In this paper, we systematically investigate whether RMs inherit value biases from pretraining. We use the “exhaustive token search” method introduced by 8 , in which RM reward scores are obtained across the entire token vocabulary to reveal the highest- and lowest-scoring responses to user prompts, and we combine this approach with tools from psycholinguistics ( 35 ) to uncover and quantify value biases in RMs as a function of the base model on which they are developed. We analyze data from 10 leading RMs on RewardBench and find robust and replicable differences between Llama- and Gemma-based RMs across a variety of dimensions of human value (Section 2 ). As a case study, we focus on the Big Two psychological dimensions ( 3 ; 1 ) that capture agency-oriented values (e.g., freedom, success, ability) and communion-oriented ones (e.g., love, family, friendship). We use a psychologically-validated corpus of words relating to agency and communion to demonstrate a robust relative preference by Llama-based RMs for agency, and by Gemma-based RMs for communion. Next, we trace the source of those biases to the base models themselves (Section 3 ) and explore differences between the Llama and Gemma base models, as implied by differences in their logprobs (relating to implicit reward models). Finally, we conduct systematic experiments training our own RMs on different base models with identical data and hyperparameters, using various sources and ablations of data, in order to chart how the observed bias evolves over the course of preference finetuning and the extent to which it can – or cannot – be “washed out” with sufficient finetuning data (Section 4 ). Our work has several key contributions: 1. We develop a new RM interpretability method based on tools from psycholinguistics. 2. Using this method, we show that RMs “in the wild” exhibit systematic value differences by base model. 3. We trace these differences back to differences in the log probabilities of the instruction-tuned models, and ultimately, in the pre-trained models on which the RMs are built. 4. We show that these differences in log probabilities themselves can be formulated as implicit reward models; we derive usable implicit reward scores and show that these exhibit the same patterns of bias. 5. We show the replicability and durability of inherited value biases by training our own RMs on different base models, controlling for source and quantity of data. 2 RMs in the Wild Show Value Differences by Base Model Exhaustive Token Search Exhaustive token search is an RM interpretability method that evaluates each token in an RM’s vocabulary on a value-laden prompt. Using this method, 8 found that the pattern of correlations between the outputs of 10 leading reward models on RewardBench, based on either Gemma or Llama, was significantly associated with the choice of base model; over a quarter of the variance in token-rank differences between reward models could be attributed to the choice of base model (representational similarity analysis, R^{2}=.27 ). Qualitatively, 8 observed that, when given the user prompt “What, in one word, is the greatest thing ever?”, a reward model based on Gemma assigned its highest reward scores to variations of “Love,” whereas a reward model based on Llama – despite being trained by the same developer with the same preference data – assigned its highest scores to variations of “Freedom.” In the present work, we seek to quantify the differences in values that reward models inherit from their base models. Psycholinguistics We assess RM value biases by combining exhaustive token search with tools from psycholinguistics ( 35 ) that permit mapping specific words to coarsened psychological constructs, including dimensions of human value (see Appendix B for details). We use two validated psycholinguistic corpora: the Big Two ( 36 ) and the Moral Foundations Dictionary (MFD2; 13 ). These corpora are coded by human experts along several different value dimensions. The Big Two codes for agency- and communion-oriented words: words that relate to concerns about the achievement of individual goals and the formation and maintenance of relationships with others, respectively. MFD2 codes for words relating to “authority,” “care,” “fairness,” “loyalty,”