DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMs

Lake Yin, Fan Huang · 2025

Read DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMs by Lake Yin, Fan Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10013.txt.

Introduction

Lake Yin and Fan Huang present DIF in a 2025 paper, a frame to benchmark hidden bias in large language models. Models can shift answers when social context changes, a flaw that is both an ethics and a skill problem. No standard gauge for this one bias type existed, the authors argue.

DIF wraps logic and math problems from known sets in varied social personas and grades the shift. A null model check adds a robustness test to the score. The frame confirms hidden bias in model behavior where it is applied. A fresh trend also shows up, with accuracy and bias moving in opposite ways. That link backs the view of bias as a skill gap, not just a values gap.

Catalog record

CREATORLake Yin, Fan Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMs Lake Yin, Fan Huang Abstract As Large Language Models (LLMs) have risen in prominence over the past few years, there has been concern over the potential biases in LLMs inherited from the training data. Previous studies have examined how LLMs exhibit implicit bias, such as when response generation changes when different social contexts are introduced. We argue that this implicit bias is not only an ethical, but also a technical issue, as it reveals an inability of LLMs to accommodate extraneous information. However, unlike other measures of LLM intelligence, there are no standard methods to benchmark this specific subset of LLM bias. To bridge this gap, we developed a method for calculating an easily interpretable benchmark, DIF (Demographic Implicit Fairness), by evaluating preexisting LLM logic and math problem datasets with sociodemographic personas. We demonstrate that this method can statistically validate the presence of implicit bias in LLM behavior and find an inverse trend between question answering accuracy and implicit bias, supporting our argument. 1 Introduction Large Language Models (LLMs) have become increasingly prominent in artificial intelligence research and applications, demonstrating impressive capabilities in tasks such as text generation, summarization, translation, and code synthesis OpenAI et al. 2024 ; Grattafiori et al. 2024 ; DeepSeek-AI et al. 2025 . LLMs’ outstanding capability to understand nuanced context stems from the massive and diverse corpora of pre-training datasets, which allow them to learn patterns and relationships in language at scales previously unattainable. Despite these advances, concerns about embedded biases in LLM have grown, leading to investigations into how these models might perpetuate stereotypes or exhibit discriminatory behavior reflected from biases present in the training data Gallegos et al. 2024 ; Dai et al. 2024 ; Ferrara 2023 . LLMs do not always maintain objectivity, sometimes letting sociodemographic context or ’personas’ skew their problem-solving process in subtle but detectable ways. Implicit bias can manifest in different forms, such as when LLM behavior changes when a different, but logically irrelevant, social context is introduced Xu et al. 2024 . This also represents a reasoning flaw since an LLM should be able to ignore this irrelevant context. In real-world demographic information simulation cases, such as finance or healthcare, ethical concerns arise about implicit bias even when the simulation does not exhibit explicit bias Bai et al. 2024 . This could potentially introduce harmful bias when personas are introduced in agent-based LLM systems, which have seen use in a variety of circumstances Li et al. 2023 ; Sun et al. 2024 ; Choi et al. 2025 . Measuring this bias systematically remains challenging. Existing LLM performance benchmarks typically focus on knowledge retrieval, language understanding, creativity, or general reasoning, paying limited attention to observed interactions between sociodemographic cues and problem-solving skills Gupta et al. 2024 . In this paper, we have the contributions of: (1) We conduct comprehensive and rigorous investigations comparing LLM bias in complex math problems across sociodemographic personas, elucidating trends in bias across different LLMs, and quantitatively validating the influence of implicit bias in LLM responses. (2) Our approach integrates established math and logical reasoning datasets with experimental prompts incorporating identity-based variables, allowing us to isolate implicit biases that emerge under different persona settings. (3) We propose a metric to capture the implicit ’fairness’ of a model, complementing existing intelligence or reasoning benchmarks and enabling straightforward cross-model comparisons. 2 Literature Review 2.1 Bias Benchmarks Many existing benchmarks focus on measuring the bias exhibited when answering word questions. Parrish et al. 2022 created the Bias Benchmark for QA (BBQ), which uses a dataset of hand-written questions designed to test social bias. Wang et al. 2024 uses a dataset based on BBQ to evaluate models by testing bias recognition, judgment, and continuation. Esiobu et al. 2023 created the AdvPromptSet and HolisticBiasR datasets for benchmarking LLM bias in the context of excessive toxicity or negativity in response generation. Similarly, Marchiori Manerba et al. 2024 created the SoFa bias benchmarking framework and dataset, but with a focus on disparate treatment between groups. Although not benchmarks, other bias evaluation methods have used word embeddings, such as Cao et al. 2022 , which use word embeddings within a social psychology context, or Xie et al. 2024 , which uses the Word Embedding Association Test ( May et al. 2019 ) to measure LLM bias in a healthcare context. Deng et al. 2024 and Liu et al. 2024 use the probability distributions of responses to measure bias. 2.2 Prompt Bias in LLMs Some studies have explored how prompts and personas in prompts can result in an LLM exhibiting bias. Gupta et al. 2024 shows that even when an LLM explicitly expresses unbiased responses when directly queried, it can still exhibit bias when assigned a persona and asked to solve math problems, serving as the main inspiration for this paper. Hida et al. 2024 and Yeh et al. 2023 also find that LLMs can exhibit bias based on prompt variations and then make attempts to de-bias with modified prompts. 3 Methodology In order to quantify the implicit bias of LLMs, such that different models can be compared, this benchmark focuses on measuring differences in LLM problem-solving performance on math problems when prompted with varied sociodemographic personas. We decided to focus on the first 1000 multiple-choice math problems from the GSM-MC test dataset for this paper due to its demonstrated correlation with open-ended math problems as a benchmark and its ease of interpretation in quantifying LLM intelligence ( Zhang et al. 2024 ) . We collected 22 different sociodemographic groups in different categories considered protected groups in the United States. This was chosen because, logically, these should have no causal influence on an LLM’s abstract problem-solving. As protected groups, there is a political consensus that these should have no influence on an individual’s ability to perform intellectual tasks. Starting with a blank persona prompt inspired by Gupta et al. 2024 , each demographic is used to create a corresponding prompt by inserting the demographic into the blank prompt as shown in Table 1 . Using each persona. Changing a single token between each prompt minimizes the confounding influence of superfluous prompt variations while focusing only on the demographic within the prompt ( Sclar et al. 2024 ) . Baseline You are a helpful American assistant. Make sure that your responses closely embody your identity. Race You are a helpful Asian American assistant. Make sure that your responses closely embody your identity. Gender You are a helpful Female American assistant. Make sure that your responses closely embody your identity. Table 1: Example system prompts with some different personas. Since these demographics were selected from an American perspective, every prompt follows the "X American" format, with the only exception being "American Indian", which was specifically chosen because of its official use in the US census. Models Llama-3 3.1 3.1 3.2 3.3 Mistral Phi Gemma Model Parameters 8B 8B 70B 3B 70B 7B 3.8B 9B Baseline Persona 350 356 602 302 597 258 362 472 American Indian 346 372 596 271 593 232 356 474 Asian 349 374 599 210 592 224 362 463 Black 349 369 595 264 598 224 359 475 Hispanic 350 370

Read the complete text