Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models

Kefan Song, Jin Yao, Runnan Jiang, Rohan Chandra, Shangtong Zhang · 2025

Read Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models by Kefan Song, Jin Yao, Runnan Jiang, Rohan Chandra, Shangtong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07806.txt.

Introduction

Song and co-authors benchmark in 2025 group fairness inside the reward models used to fine tune large language models. Fairness checks usually judge only final output, and they force the same prompt questions on every group, which real use breaks. Bias can enter at pretraining or at fine tuning, and the reward model is one entry point.

Expert written texts from arXiv let them test reward models without identical prompts across groups. Results show all evaluated reward models display group fairness gaps in their setup. Locating bias at this component level should help target fixes, rather than treating the whole pipeline as one black box.

Catalog record

CREATORKefan Song, Jin Yao, Runnan Jiang, Rohan Chandra, Shangtong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07806

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models Kefan Song, Jin Yao, Runnan Jiang, Rohan Chandra, Shangtong Zhang Abstract As Large Language Models (LLMs) become increasingly powerful and accessible to human users, ensuring fairness across diverse demographic groups, i.e., group fairness, is a critical ethical concern. However, current fairness and bias research in LLMs is limited in two aspects. First, compared to traditional group fairness in machine learning classification, it requires that the non-sensitive attributes, in this case, the prompt questions, be the same across different groups. In many practical scenarios, different groups, however, may prefer different prompt questions and this requirement becomes impractical. Second, it evaluates group fairness only for the LLM’s final output without identifying the source of possible bias. Namely, the bias in LLM’s output can result from both the pretraining and the finetuning. For finetuning, the bias can result from both the RLHF procedure and the learned reward model. Arguably, evaluating the group fairness of each component in the LLM pipeline could help develop better methods to mitigate the possible bias. Recognizing those two limitations, this work benchmarks the group fairness of learned reward models. By using expert-written text from arXiv, we are able to benchmark the group fairness of reward models without requiring the same prompt questions across different demographic groups. Surprisingly, our results demonstrate that all the evaluated reward models (e.g., Nemotron-4-340B-Reward, ArmoRM-Llama3-8B-v0.1, and GRM-llama3-8B-sftreg) exhibit statistically significant group unfairness. We also observed that top-performing reward models (w.r.t. canonical performance metrics) tend to demonstrate better group fairness. 1 Introduction Large Language Models (LLMs) have demonstrated impressive capabilities and are assisting a growing user base ( Hu 2023 ) . Yet, ensuring these benefits are equitably distributed across diverse demographic groups remains a critical challenge ( Goellner et al. 2024 ; OpenAI 2024 ) . This concern can be formalized as the group fairness problem in LLMs. While existing research on bias in LLMs has reduced stereotypical language toward certain groups ( Nangia et al. 2020 ; Webster et al. 2021 ; Wang & Cho 2019 ; Wang et al. 2025 ) , it assumes that users from different groups pose identical prompts or include explicit group attributes (e.g., “he,” “she”). In practice, demographic information is often unstated, and users may ask distinct questions that originate from their everyday interests and experiences shaped by their demographic groups ( Weber & Castillo 2010 ) . As a result, current methods fall short of measuring potential group unfairness in scenarios where prompts differ across demographic groups. Moreover, fairness assessment in LLMs typically focuses on the final generated text rather than examining the training pipeline itself. Bias can arise from multiple stages of LLM training—including reward modeling and Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al. 2022 ) —making it crucial to pinpoint where biases originate. Recognizing the above limitations, in this work, we aim to benchmark demographic parity, a common group fairness metric in reward models, and our contribution is the following: Figure 1 : Conceptual Comparison of Counterfactual Bias Evaluation and Group Fairness in LLM Evaluation. First, we introduce a novel group fairness problem in reward models from RLHF. We recognize that successful evaluation and mitigation of this problem in reward models could lead to LLMs that are fairer with respect to the demographic parity definition. Second, we propose using arXiv metadata to evaluate group fairness in reward models. Curating datasets for this purpose faces several challenges: (1) there are no publicly available user prompt datasets with demographic data from sources like OpenAI or Anthropic; (2) constructing consistent, expert-written responses is costly, and LLM-generated responses cannot be used due to potential existing group biases; (3) assessing response quality requires alignment with the preferences of specific demographic groups, necessitating additional human annotators. The arXiv dataset overcomes these challenges by providing expert-written and reviewed texts from eight categories (e.g., physics, economics, computer science) that correspond to occupational demographic groups. We curated 2000 query-response pairs per category to serve as a benchmark for evaluating eight top-performing reward models from the RewardBench leaderboard ( Lambert et al. 2024 ) . Last, we analyze the results of this benchmark to make the following novel observations: (1) group unfairness truly exists in all of the evaluated reward models, as the differences in group means are statistically significant from the ANOVA test; (2) good reward models are also fairer ones, as the top 2 reward models from the RewardBench also have the lowest Normalized Maximum Group Difference. (3) In each reward model, the unfairness is pervasive across the demographic groups, as a minimum of 23 out of 28 pairs of groups are shown to be different by the Tukey HSD Test; (4) most reward models share the same pattern of unfairness, as the average rewards from 5 out of 7 models has a Pearson correlation larger than 0.8 with that of the Nemotron-340B model. 2 Related Works Reducing Harmful Language in LLM Outputs. Most research on fairness in LLMs has focused on reducing harm and risk in LLM generation through bias mitigation techniques. Techniques such as counterfactual data augmentation ( Lu et al. 2020 ) , data filtering and selection ( Garimella et al. 2022 ) , machine unlearning methods ( Yao et al. 2024 ; Dige et al. 2024 ) , designing specific prompting triggers ( Venkit et al. 2023 ) and incorporating the notion group fairness in constructing a bias evaluation dataset ( Bi et al. 2023 ) , have proven effective to reduce stereotypical or harmful language targeted at various demographic groups. Debiasing, however, is not sufficient for fairness, as these approaches primarily measure fairness in terms of harmfulness reduction. A perfectly harmless LLM may still provide unfair answers to the different prompts provided by various demographic groups. Aligning LLMs with Diverse Human Preferences. Recent work in fair RLHF, such as MaxMin RLHF ( Chakraborty et al. 2024 ) and preference matching RLHF methods ( Xiao et al. 2024 ) , finetune the LLM to align with a diversity of human preferences. However, the fairness notion of these methods comes from social choice theory, which is different from the algorithm fairness, more specifically, group fairness that we aim to address here. In addition, these methods assume that people from various groups ask the same questions and, therefore, do not address the issue of diversity in informational needs. Group Fairness in LLM Decisions. Recent studies have explored the issue of group fairness when prompting LLMs to perform high-stakes ML classification decisions ( Li et al. 2024 ; Atwood et al. 2024 ) . While these works focus on the specific task of prompting LLMs to play the role of a classifier, we instead focus on the general domain text generation user scenario of LLMs. 3 Group Fairness in LLM We start to consider a particular definition of group fairness, demographic parity (alternatively known as statistical parity) in the context of LLMs. First, we provide the group fairness of reward model definition for reward models. Second, we highlight the unique challenges in addressing group fairness compared to counterfactual bias mitigation. In addition, we outline the RLHF training pipeline and emphasize the importance of addressing

Read the complete text