Reward Model Interpretability via Optimal and Pessimal Tokens DOI: 10.1145/3715275.3732068 Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece CCS: Applied computing Psychology CCS: Human-centered computing Empirical studies in HCI CCS:

Brian Christian, Hannah Rose Kirk, Jessica A.F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska · 2025

Read Reward Model Interpretability via Optimal and Pessimal Tokens DOI: 10.1145/3715275.3732068 Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece CCS: Applied computing Psychology CCS: Human-centered computing Empirical studies in HCI CCS: by Brian Christian, Hannah Rose Kirk, Jessica A.F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07326.txt.

Introduction

Brian Christian and co-authors probe reward models, the scorers that steer chatbots toward human values, in a 2025 paper. These models turn each reply into a single score, yet what they truly prize is murky. The team scored every single-word reply to value-laden prompts across ten open models.

Models trained for like aims prized quite unlike words. High and low scores were handed out in lopsided ways, and prompt wording swayed scores like known human biases. Common words were overvalued, and some scorers favored certain identity groups. The authors warn that such quirks can pass into the chatbots these scorers train.

Catalog record

CREATORBrian Christian, Hannah Rose Kirk, Jessica A.F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07326

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reward Model Interpretability via Optimal and Pessimal Tokens DOI: 10.1145/3715275.3732068 Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece CCS: Applied computing Psychology CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI design and evaluation methods Brian Christian, Hannah Rose Kirk, Jessica A.F. Thompson, Christopher Summerfield, Tsvetomira Dumbalska Abstract . Reward modeling has emerged as a crucial component in aligning large language models with human values. Significant attention has focused on using reward models as a means for fine-tuning generative models. However, the reward models themselves—which directly encode human value judgments by turning prompt-response pairs into scalar rewards—remain relatively understudied. We present a novel approach to reward model interpretability through exhaustive analysis of their responses across their entire vocabulary space. By examining how different reward models score every possible single-token response to value-laden prompts, we uncover several striking findings: (i) substantial heterogeneity between models trained on similar objectives, (ii) systematic asymmetries in how models encode high- vs low-scoring tokens, (iii) significant sensitivity to prompt framing that mirrors human cognitive biases, and (iv) overvaluation of more frequent tokens. We demonstrate these effects across ten recent open-source reward models of varying parameter counts and architectures. Our results challenge assumptions about the interchangeability of reward models, as well as their suitability as proxies of complex and context-dependent human values. We find that these models can encode concerning biases toward certain identity groups, which may emerge as unintended consequences of harmlessness training—distortions that risk propagating through the downstream large language models now deployed to millions. CONTENT WARNING: This article presents examples of biased, offensive, sexually explicit and otherwise harmful text. The authors do not endorse any of the harmful representations quoted below. 1. Introduction The alignment of large language models (LLMs) with human values has emerged as one of the central challenges in modern AI development, and at the heart of this challenge lie “reward models”—neural networks trained to directly proxy human preferences by transforming text into scalar rewards. Though typically treated as disposable intermediaries in the larger alignment process, these models are crucial objects of study in their own right as the most direct and explicit encoding of human values in AI systems, yet are surprisingly under-explored. The typical process for aligning an LLM with human values involves collecting a dataset of labeled pairwise human preferences, indicating which of two LLM responses to a given user prompt is preferred ( Christiano et al. 2017 ) . These preference data often distill multiple desirable objectives, such as helpfulness, harmlessness, and honesty ( Bai et al. 2022 ) , which are operationalized through guidelines written by model developers and interpreted by crowdworkers ( Kirk et al. 2023a ) . The resulting dataset is used to train a “reward model”—a transformer model that takes in a prompt-response pair (or a longer user-assistant dialogue) and outputs a scalar that represents in effect how “preferable” that response is. These scalars are typically based on the Bradley-Terry score ( Bradley and Terry 1952 ) , and the reward model is trained via stochastic gradient descent to minimize the negative log-likelihood of the observed pairwise preferences ( Lambert et al. 2023 ) . The trained reward model then acts as a scalable proxy for human preferences when using reinforcement-learning algorithms such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) to fine-tune the LLM. This process—known as Reinforcement Learning from Human Feedback (RLHF)—results in LLM generations that maximize the reward model’s score rather than the pre-training objective, so are supposedly more aligned with human values. While direct alignment algorithms like Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) have grown in popularity, they capture equivalent preference relationships from the data, just without the reward model as an intermediary. Although reward models exist as a disposable reagent in the process of turning an “unaligned” LLM built to minimize predictive loss into an “aligned” one built to maximize this proxy of human preference, they are fascinating objects of research inquiry in their own right. Designed as generalizable proxies for human preference, they offer more direct value encoding than downstream agents constrained by KL-divergence ( Jaques et al. 2019 ) and refusal training. As scalar mappings over complex dialogue, they distill multi-objective preference data into uniquely interpretable low-dimensional representations of human value. They are in essence where “the human value rubber meets the road.” Despite this, there is a dearth of literature analyzing the properties of reward models, largely because few have been publicly available for study. While 2023–2024 saw a proliferation of open-source language models, including Meta’s Llama ( Touvron et al. 2023 ) , Mistral AI’s models ( Jiang et al. 2023 ) , and Google’s Gemma series ( Team et al. 2024a ) , to date no major industry or nonprofit lab has openly released a reward model. Only recently has this picture begun to change, with the release of RewardBench ( Lambert et al. 2024 ) —the first benchmark and leaderboard for reward models, spurring new activity among academic and open-source communities. In this work on reward model interpretability, we seek to understand the consistency and faithfulness with which these models represent human values. Specifically, we make the following contributions: • We pioneer an exhaustive search over every single token in reward model vocabularies appended to a value-laden prompt, permitting the analysis of optimal and pessimal tokens across ten top-performing open-source reward models on RewardBench of varying sizes and architectures. • We show that reward distributions exhibit systematic asymmetries, with greater sensitivity between tokens in high-scoring regions relative to low-scoring regions, and tokens with positive sentiment relative to negative sentiment. Changing to a negative-valence prompt inverts this latter bias, mirroring framing effects in humans. • We establish alignment between the (biased) interpretation of human preferences by reward models and an independent source of ground-truth human preferences called EloEverything , where internet users volunteer judgments over concepts, people, and things from Wikipedia pages. • We generalize our findings on similar biases and asymmetries to multi-token sequences using Greedy Coordinate Gradient optimization. • Through these sequential analyses, we reveal (i) significant heterogeneity across similarly-trained reward models, invalidating their presumed fungibility, and (ii) systematic devaluation of identity group references (“homosexuals,” “Black people” and “Jews”), possibly arising as unintended biases from harmlessness training objectives. Through exposing the idiosyncrasies of reward models as research artifacts, we aim to highlight pitfalls in their current development, provide recommendations for building more robust proxies of human value, and ultimately advance the broader goal of creating aligned and safe AI systems. 2. Ranking Optimal and Pessimal Tokens Table 1 . Open-source reward models studied. The table includes both their full names and the shortened identifiers (Model IDs) used throughout the rest of this paper. Ranks are from the RewardBench Leaderboard as of January 14, 2025. RewardBench Rank Model ID Developer Model Name Base Model Parameters (B) 2 \blacksquare N-Gem-27B nicolinho QRM-Gemma-2-27B ( Dorka 2024 ) Gemma 2 ( Team et al. 2024b ) 27 3 \blacksquare S-Gem-27B-v0.2 Skywork Skywork-Reward-Gemma-2-27B-v0.2 ( Liu et al. 2024 ) Gemma 2 27 5 \blacksquare S-Gem-27B Skywork Skywork-Reward-Gemma-2-27B ( Liu et al. 2024 ) Gemma 2 27 10 \blacksquare S-Lla-8B-v0.2 Skywork

Read the complete text