Detecting Prefix Bias in LLM-based Reward Models

Ashwin Kumar, Yuzi He, Aram H. Markosyan, Bobbie Chern, Imanol Arrieta-Ibarra · 2025

Read Detecting Prefix Bias in LLM-based Reward Models by Ashwin Kumar, Yuzi He, Aram H. Markosyan, Bobbie Chern, Imanol Arrieta-Ibarra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13487.txt.

Introduction

Ashwin Kumar and co-authors hunt for prefix bias in the reward models that steer language model training, in a 2025 paper. Small changes at the start of a prompt can tilt which answer a reward model prefers, even when the content is the same. The team built measures to expose that tilt across race and gender lines.

Across open preference datasets and model designs, the bias showed up everywhere, whatever the architecture. A data augmentation fix cut the effect in their tests. The authors urge bias aware design of preference data and routine checks of reward models before they shape deployed systems.

Catalog record

CREATORAshwin Kumar, Yuzi He, Aram H. Markosyan, Bobbie Chern, Imanol Arrieta-Ibarra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13487

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting Prefix Bias in LLM-based Reward Models Ashwin Kumar, Yuzi He, Aram H. Markosyan, Bobbie Chern, Imanol Arrieta-Ibarra Abstract . Reinforcement Learning with Human Feedback (RLHF) has emerged as a key paradigm for task-specific fine-tuning of language models using human preference data. While numerous publicly available preference datasets provide pairwise comparisons of responses, the potential for biases in the resulting reward models remains underexplored. In this work, we introduce novel methods to detect and evaluate prefix bias—a systematic shift in model preferences triggered by minor variations in query prefixes—in LLM-based reward models trained on such datasets. We leverage these metrics to reveal significant biases in preference models across racial and gender dimensions. Our comprehensive evaluation spans diverse open-source preference datasets and reward model architectures, demonstrating susceptibility to this kind of bias regardless of the underlying model architecture. Furthermore, we propose a data augmentation strategy to mitigate these biases, showing its effectiveness in reducing the impact of prefix bias. Our findings highlight the critical need for bias-aware dataset design and evaluation in developing fair and reliable reward models, contributing to the broader discourse on fairness in AI. DOI: 10.1145/3715275.3732204 Conference: ACM Conference on Fairness, Accountability and Transparency; June 23–26, 2025; Athens, Greece ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Natural language processing CCS: Computing methodologies Learning from demonstrations 1. Introduction Large Language Models (LLMs) have seen a large growth in research and usage ( Achiam et al., 2023 ; Touvron et al., 2023b ) . Their ability to understand context and generate and analyze text has proven their utility in applications from code completion ( Roziere et al., 2023 ) to medical diagnosis ( Singhal et al., 2023 ) . This is thanks to the large amounts of pre-training that models undergo, which deployers can leverage for different use cases. However, to efficiently use LLMs for different tasks, we need to fine-tune them on task-specific information ( Bai et al., 2022 ; Ouyang et al., 2022 ; Stiennon et al., 2020 ) . This finetuning can be done in multiple ways, the simplest being Supervised Fine Tuning (SFT), where domain data is used to train the model with next-token prediction. Reinforcement Learning with Human Feedback (RLHF) is a paradigm popularized by the InstructGPT paper ( Ouyang et al., 2022 ) , and further used by Anthropic ( Bai et al., 2022 ) , Llama ( Touvron et al., 2023b ; Touvron et al., 2023a ) and other popular language models. When using RLHF, a preference model or Reward Model (RM) is trained to discriminate between pairwise responses to an input prompt to give higher scores for responses that are preferred by human reviewers ( Ouyang et al., 2022 ) . These reward models are used for reinforcement learning to guide fine-tuning, using algorithms like proximal policy optimization (PPO) ( Schulman et al., 2017 ) . There have been studies ( Gallegos et al., 2023 ; Sheng et al., 2021 ; Bender et al., 2021 ) looking at bias in language models. However, to the best of our knowledge, no one has studied bias in reward models in the context of RLHF fine-tuning. Further, there has been limited study of prefix bias , where changing the prefix to a prompt can change the model’s behavior. RMs are typically used to evaluate prompt-response pairs and assign a value to them, which then trains the finetuned model to prefer the response that receives a higher reward. If the reward model’s preferences are biased, it may poison the downstream model and cause undesirable learned behavior. Thus, it is important to create ways to identify and mitigate bias in the reward model stage. In this paper, we study bias in reward models by explicitly conditioning candidate responses on different demographic identifiers using short textual prefixes. Our contributions are as follows: (1) We design auto-influence and cross-influence , two methods for evaluating the prefix-bias in a learned reward model. We also present winrate deviation and accuracy deviation , metrics for quantifying auto-influence and cross -influence respectively. (2) We train and evaluate reward models on a variety of open datasets and show the presence of racial and gender bias using these metrics. We also use different model architectures and show that this bias is present for all of them, suggesting that the bias is learned from the datasets. (3) We conduct experiments to identify the source of the bias and uncover patterns based on the training dataset used, and present a solution using data augmentation to fix this issue. 2. Related Work Bias in LLMs: The study of bias in LLMs has become a pressing and active field of research ( Kotek et al., 2023 ; Gallegos et al., 2023 ; Haim et al., 2024 ; Omiye et al., 2023 ) , driven by their widespread use in sensitive domains. Early work focused on how training data can encode systemic biases ( Dodge et al., 2021 ; Bender et al., 2021 ) , while more recent research has highlighted specific downstream harms, such as racial disparities in clinical advice ( Omiye et al., 2023 ) and gendered patterns in name-based completions ( Haim et al., 2024 ; Kotek et al., 2023 ) . A recent survey ( Gallegos et al., 2023 ) provides a broad overview of these challenges. Prefix bias has also been studied in recent work; for instance, Chaudhary et al. (2024) use prefix perturbations to measure LLM robustness in a certification setting, but they do not consider the reward modeling stage. Several recent studies have taken an identity-grounded approach to evaluating bias in LLMs and reward models. Kantharuban et al. (2024) find that chatbots often reflect racial stereotypes based on inferred user identity, blurring the line between personalization and harm. Eloundou et al. (2024) introduce the concept of first-person fairness , showing how name-based probes can surface demographic biases in chatbot behavior. These works underscore the importance of evaluating LLM behavior through demographic context. However, in this paper, we focus on developing metrics to evaluate and quantify prefix bias in Reward Models. We explore the biases that may exist and appear in the Reward Model stage, which can introduce biases to the fine-tuned model due to an inaccurate learned reward function. As such, this paper is orthogonal to the rich field of analyzing bias in pre-trained and deployed LLMs. Language Model Fine-tuning: Trained on vast corpora, language models also possess vast information. However, to constrain their outputs to suit a particular use case, and to ensure their outputs are aligned with our expectations, LLMs need to be fine-tuned on curated data ( Bai et al., 2022 ; Ouyang et al., 2022 ; Stiennon et al., 2020 ) . Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Bai et al., 2022 ) is a popular technique for achieving LLM alignment. RLHF aims to collect human preferences in terms of pairwise rankings, which are then used to learn a preference model or Reward Model (RM). The RM is then used to fine-tune the LLM using Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , which is a popular reinforcement learning algorithm. Reward models may also be used for best-of-n (BoN) sampling of the

Read the complete text