Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation

2025

Read Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23921.txt.

Introduction

Kaveh Eskandari Miandoab and five co-authors show how little it takes to push language models back into stereotyped answers. Their augmentation framework adds context to fairness test questions in three plug-and-play steps, then re-asks them. The test bed is the BBQ bias benchmark for question answering, applied to leading open and closed models alike.

With the contextual nudges, models grew more likely to answer along stereotype lines, even though the added text carried no evidence. The effect was strongest where the target group is a community the fairness literature has studied least. The authors read this as a warning: current bias alignment is brittle, and passing a benchmark does not mean a model will stay fair under mild variation. Expanding fairness research to more communities is part of the fix. The framework is general and can stress-test other fairness benchmarks the same way.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23921

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation Abstract Large Language Models have been shown to demonstrate stereotypical biases in their representations and behavior due to the discriminative nature of the data that they have been trained on. Despite significant progress in the development of methods and models that refrain from using stereotypical information in their decision-making, recent work has shown that approaches used for bias alignment are brittle. In this work, we introduce a novel and general augmentation framework that involves three plug-and-play steps and is applicable to a number of fairness evaluation benchmarks. Through application of augmentation to a fairness evaluation dataset (Bias Benchmark for Question Answering (BBQ)), we find that Large Language Models (LLMs), including state-of-the-art open and closed weight models, are susceptible to perturbations to their inputs, showcasing a higher likelihood to behave stereotypically. Furthermore, we find that such models are more likely to have biased behavior in cases where the target demographic belongs to a community less studied by the literature, underlining the need to expand the fairness and safety research to include more diverse communities. Keywords: Bias and Fairness, Safety, Augmentation, Evaluation of Large Language Models Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation Kaveh Eskandari Miandoab 1,5,† , Mahammed Kamruzzaman 2,5,† , Arshia Gharooni 3,5 Gene Louis Kim 2 , Vasanth Sarathy 1 , Ninareh Mehrabi 4,5 1 Tufts University 2 University of South Florida 3 Sharif University of Technology 4 Meta 5 Resolution [email protected], [email protected] † Equal contribution Abstract content 1. Introduction While recent advancements in LLMs have demonstrated promising reductions in social biases, such as those related to gender, race, and religion, a critical question remains: Do these improvements truly showcase a fair and ethical model or are they a byproduct of overfitting to well-known bias benchmarks? Given the rapid proliferation of LLMs and the public availability of widely-used bias datasets (e.g., BBQ Parrish et al. 2022 , StereoSet Nadeem et al. 2021 , CrowS-Pairs Nangia et al. 2020 , DiFair Zakizadeh et al. 2023 ) upon which modern LLMs can be built given their ease of access, it is plausible that modern models may have implicitly memorized patterns from these benchmarks rather than internalizing fairness principles, such that perturbation in the direction of these patterns collapses the model fairness. Consequently, evaluating their robustness through variations of the input becomes essential. Furthermore, although some bias categories, such as gender and race, have been extensively studied, other bias categories like age, disability, socio-economic status (SES), and appearance—remain significantly underexplored in fairness evaluations. We base our grouping, as demonstrated by Table 1 , on the findings provided by Gallegos et al. 2024 , as well as our own observations of the literature. We believe that this imbalance raises important questions about the generalizability of LLM fairness across diverse social dimensions. Figure 1: Response from oLMo2-13B for both the original BBQ settings and our augmented settings. In the augmented setting, the scenario is lengthy, so we represent it with “[…]”. However, the question is not answerable based on the information provided in the scenario, yet the model still responds with “Keisha.” To bridge this gap and better understand a model’s robustness to shortcut learning with respect to fairness and equality, we propose a novel, automatically generated, and curated augmentation method that only relies on the presence of target demographic information (such as gender information, racial information, or age) to perform the augmentation with respect to that demographic. Our augmentation is designed such that the original data instance is semantically preserved, while the perturbation sufficiently changes it so that LLMs cannot respond using purely the knowledge acquired by being trained on that instance. More concretely, we utilize a state-of-the-art LLM to generate multiple instances consistent with a given demographic and with respect to an initial instance, such that the original meaning of that instance is preserved, further variety is added to the demographic by generating multiple examples, and more grounded examples are made as a part of the augmentation process. As part of our testing, we perform the augmentation process on the BBQ dataset Parrish et al. 2022 , while emphasizing that the proposed method can be easily extended to other datasets and can even be used to generate an original dataset via the generation capabilities of modern LLMs. We systematically evaluate the fairness performance of a number of state-of-the-art LLMs in cases where the original input is provided, versus the cases where the input is perturbed to generate semantically equivalent instances in the direction of the demographic. We provide one example response of oLMo2-13B for the original input of BBQ and our augmented version in Figure 1 . We find that such a perturbation can significantly degrade the model’s fairness performance, showcasing that it is indeed possible for models to learn representations that are fair only at a superficial level during the training and alignment process, falling back toward a more stereotypical decision process. Our findings are in line with the previous work showing that fair-tuning and alignment of LLMs only ensure superficial fairness Qi et al. 2023 ; Wolf et al. 2024 . However, we do not rely on fine-tuning Qi et al. 2023 to compromise the model safety, and our work naturally follows the theoretical observations by Wolf et al. 2024 , practucally showcasing that it is possible to exploit the latent stereotypical knowledge of LLMs through semantic perturbation. Furthermore, we perform the augmentation in the direction of various demographic groups based on Table 1 and following the grouping of the BBQ dataset. We find that models consistently perform better on the well-studied groups, with less-studied groups observing a performance drop of up to 14% compared to the gender and race dimensions. This finding showcases the research community’s overwhelming focus on only several ‘highly publicized’ demographic groups, shedding further light on the necessity of expanding the fairness endeavors to include a larger number of demographics and underrepresented individuals. Well-Studied Biases Gender Bias Race Bias Religion Bias Nationality Bias Less-Studied Biases Age Appearance Bias Disability Socioeconomic Status Table 1: Different bias types, as grouped into ‘Well-Studied’, and ‘Less-Studied’ categories throughout this work. We make the following contributions through this work: 1) We investigate whether meaning-preserving augmentations to benchmark datasets “jailbreak” current LLMs into revealing latent biases by introducing a novel augmentation framework that only relies on the presence of demographic data. 2) We analyze the model biases with respect to their prevalence in the literature, finding that LLMs are more likely to showcase stereotypical behavior when exposed to inputs containing less-studied biases (e.g., age, disability, SES). 3) We publicly make available the dataset obtained by applying our framework to the BBQ Parrish et al. 2022 dataset, with the emphasis that our framework is applicable to other evaluation benchmarks 1 1 1 The corresponding data will be made publicly available upon the publication of the paper . 2. Background and Related Work Behavioral testing and robustness. Standard held-out accuracy can mask systematic failures. Ribeiro et al. 2020 reframe evaluation as capability-oriented behavioral tests (Minimum Functionality, invariance, directionally expected changes), revealing actionable gaps in morphology, negation, and temporal reasoning despite similar aggregate scores. Complementing this, Jin et al. 2019 show that

Read the complete text