One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber · 2026
Read One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models by Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03291.txt.
Introduction
Daniel Fein and four colleagues examine the reward models that steer language models toward human preferences. Such models can be gamed: a policy learns to please the scorer rather than do what people actually want. The 2026 study measures bias carefully in five high-quality reward models, including the best currently available.
Known problems persist: longer answers, flattering answers, and overconfident answers still score too well. The team also finds new biases, favoring one model's personal style and favoring answers by their order. Their fix, called mechanistic reward shaping, adjusts the model after training using its own internal signals. It cuts the simpler biases with very little labeled data, keeps overall quality, and carries over to cases outside the tuning set.
Catalog record
| CREATOR | Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65949 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03291
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer, Nick Haber Abstract Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking , whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including the state-of-the-art, we find that issues persist despite prior work with respect to length, sycophancy, and overconfidence. We also discover new issues related to bias toward model-specific “styles” and answer-order. We categorize RM failures by complexity and propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations. Our proposed mechanistic reward shaping reduces targeted biases without degrading reward quality and while using minimal labeled data. The method is extensible to new biases, model-internal, and generalizes out-of-distribution. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning from Human Feedback (RLHF) remains a popular method to control the behavior of language models ( 43 ; 62 ; 28 ) . However, RLHF is vulnerable to reward hacking , whereby optimizing an imperfect proxy reward function leads to poor performance according to a true reward function ( 56 ; 26 ; 5 ) . Reward hacking during RLHF is a potential source of observed biases in language model behavior including length bias ( 50 ; 54 ) , position bias ( 69 ) , overconfidence ( 8 ) , and sycophantic user agreement ( 51 ; 63 ) . Though a significant body of work has addressed the susceptibility of RLHF to reward hacking, much prior work frames reward hacking as arising from reward misgeneralization based on linear spurious correlation, with trained policies learning to exploit flawed reward functions ( 53 ; 41 ; 6 ) . However, this perspective diminishes the role of reward hacking resultant from the non-linear artifacts caused by reward misspecification. Here, we distinguish low-complexity, correlational reward errata from complex, non-linear biases. Motivated by the Linear Representation Hypothesis ( 44 ) , we hypothesize that low-complexity shortcut (spurious-feature) signals used by reward models (RM)s correspond to approximately linear directions in representation space. We find that isolating and removing such directions via a null-space projection mitigates length-, uncertainty-, and position-related biases. In contrast, other RM failures appear more complex and are not well-approximated by a single linear direction, limiting the effectiveness of linear probe interventions. Addressing biases at the RM level is particularly impactful as RMs serve as the foundation for multiple alignment techniques beyond RLHF, including automated red-teaming, best-of-N sampling, and iterative data filtering. Moreover, model-internal interventions on RMs enable pointwise reward correction without requiring access to or modification of the policy optimization procedure, making them broadly applicable across deployment contexts. Through systematic evaluation of five RMs (including state-of-the-art) across multiple dimensions, we identify both persistent and novel failure modes, introduce a data-efficient probe-nulling technique for tractable biases, and categorize biases as linear or complex, characterizing the limitations of linear interventions for complex artifacts. We make five core contributions: • We demonstrate that previously discovered biases persist in state-of-the-art RMs, using labeled datasets to detect biases based on length, overconfidence, and sycophancy across five contemporary RMs. • We identify two novel biases in RMs: (1) position bias (both in free-form text and multiple-choice settings), and (2) model-style reward sensitivity , where RMs systematically reward or penalize completions based on their distributional similarity to specific language model writing styles. • We categorize RM biases by complexity, distinguishing between low-complexity biases amenable to linear probe intervention (length, uncertainty, position) and high-complexity biases resistant to simple debiasing (sycophancy, model-style sensitivity). • We introduce a mechanistic reward shaping approach using linear activation probes to mitigate low-complexity biases. This method is data-efficient, model-internal (enabling use beyond regular RLHF), and significantly reduces targeted biases. • We show that the proposed intervention does not significantly degrade accuracy on RewardBench-2, and that probes constructed from synthetic datasets generalize out-of-distribution. We provide extensive empirical analysis across four diverse benchmarks (PlausibleQA, BIG-bench, GSM8K-MC, MMLU), demonstrating both the effectiveness of our intervention for tractable biases and the limitations of linear interventions for complex artifacts. The code and generated data are publicly available on GitHub 1 1 1 github.com/drfein/OneBiasAfterAnother (MIT license). 2 Related Work A significant body of work addresses reward hacking in RLHF . Existing methods toward this problem span data augmentation techniques ( 25 ; 33 ) , preference data analysis ( 39 ) , RM token-level ranking analysis ( 11 ) , robust RM training methods ( 37 ; 6 ; 61 ; 41 ) , and post-hoc reward shaping ( 15 ; 24 ; 13 ) . Some work found RMs rank completions from their own base-model lineage slightly higher on average using an inverse-rank analysis ( 34 ) , but not in a continuous, model-conditioned style sensitivity. In contrast, we identify two previously uncharacterized RM biases and intervene directly on RM representations, enabling targeted removal of specific spurious features without retraining, while explicitly distinguishing low-complexity correlational biases from entangled, context-dependent artifacts. Post-hoc methods avoid costly retraining but are limited in generality. Prior length-bias corrections rely on explicit modeling of the length–reward relationship ( 45 ; 24 ; 67 ) , e.g., with length penalties, but restricting applicability to biases that can be precisely measured. In contrast, our approach removes length-correlated directions in the RM latent space, enabling correction even when the functional form of the bias is unknown. Other post-hoc methods, such as reward shaping via bounded transformations ( 15 ) or RM ensembles ( 13 ) , regularize reward behavior globally but cannot isolate and selectively remove individual spurious features. Linear Probes: The linear representation hypothesis posits that high-level concepts are represented linearly as directions in some representation space ( 44 ) . It has proven instrumentally useful within the subfield of interpretability and 64 show that difference-of-mean probes (which we adopt) constructed from language model activations can reliably detect concepts. Significant work has shown how these concept vectors can be used to steer generation behavior of language models ( 60 ; 30 ; 49 ; 55 ; 7 ) . A smaller body of work attempts to use the complement of a known activation subspace to steer model behavior. 48 null subspaces from an early tweet-classification model to remove unwanted biases. 4 null out directions obtained through sparse autoencoders to steer language-model finetuning away from misaligned behavior. Compared to prior debiasing methods, our intervention is lightweight and easily extensible to new biases for already-trained RMs, enabling reward correction without modifying downstream policy optimization algorithms. More broadly, existing RM debiasing and probing approaches do not distinguish between biases that are well-approximated by simple linear features and those arising from complex, entangled representations. 3 Background 3.1 Reward Bias Complexity We categorize RM biases into two complexity classes based on the extent to which they manifest as isolated linear directions in the RM-head representation space: Low-complexity biases , which correspond to relatively simple features (e.g., length, linguistic markers, positional information) that can be captured by linear directions in the model’s representation space, and high-complexity biases , which arise from entangled, context-dependent factors resistant to simple