Race, Ethnicity and Their Implication on Bias in Large Language Models

Shiyue Hu, Ruizhe Li, Yanjun Gao · 2026

Read Race, Ethnicity and Their Implication on Bias in Large Language Models by Shiyue Hu, Ruizhe Li, Yanjun Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12868.txt.

Introduction

Shiyue Hu, Ruizhe Li, and Yanjun Gao study bias tied to race and ethnicity in large language models. These models now operate in high stakes settings, including healthcare and medicine. Demographic traits may be stated outright in text, or inferred by the model itself. Past studies mostly document disparities in final outcomes.

Outcome level evidence gives limited insight into internal mechanisms, though. The team aim to look inside at how such effects arise. Knowing the mechanism matters for any serious fix. A patch on outputs may fail if the inner bias stays. Their work serves fair deployment of models in medicine and beyond.

Catalog record

CREATORShiyue Hu, Ruizhe Li, Yanjun Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Race, Ethnicity and Their Implication on Bias in Large Language Models Shiyue Hu, Ruizhe Li, Yanjun Gao Abstract Large language models (LLMs) increasingly operate in high-stakes settings including healthcare and medicine, where demographic attributes such as race and ethnicity may be explicitly stated or implicitly inferred from text. However, existing studies primarily document outcome-level disparities, offering limited insight into internal mechanisms underlying these effects. We present a mechanistic study of how race and ethnicity are represented and operationalized within LLMs. Using two publicly available datasets spanning toxicity-related generation and clinical narrative understanding tasks, we analyze three open-source models with a reproducible interpretability pipeline combining probing, neuron-level attribution, and targeted intervention. We find that demographic information is distributed across internal units with substantial cross-model variation. Although some units encode sensitive or stereotype-related associations from pretraining, identical demographic cues can induce qualitatively different behaviors. Interventions suppressing such neurons reduce bias but leave substantial residual effects, suggesting behavioral rather than representational change and motivating more systematic mitigation. [ BoldFont = texgyretermes-bold.otf, ItalicFont = texgyretermes-italic.otf, BoldItalicFont = texgyretermes-bolditalic.otf ] \setCJKmainfont FandolSong-Regular.otf[ BoldFont = FandolSong-Bold.otf ] \setCJKsansfont FandolHei-Regular.otf[ BoldFont = FandolHei-Bold.otf ] 1 Introduction Large language models (LLMs) are increasingly used in high-stakes domains such as healthcare, where demographic attributes (e.g., race, ethnicity, gender) may be explicitly stated or implicitly inferred from text. Prior work shows that LLMs can condition their outputs on demographic information even when it is not task-relevant 24 ; 13 ; 5 ; 26 , therefore can induce misattribution on model output with undesirable or biased behavior 4 ; 14 ; 24 . Most prior studies on demographic bias focus on outcome-level effects, evaluating disparities in generated responses, accuracy, calibration, or toxicity scores across demographic groups 21 ; 10 ; 9 ; 22 . While these analyses are essential for documenting harm, they treat LLMs as black boxes, offering limited insight into whether demographic attributes are encoded as high-level semantic features, task-relevant representations, or spurious shortcuts during prediction. In parallel, recent works in mechanistic interpretability demonstrated how LLMs encode demographic information and manipulated internal LLMs’ states to ensure fairness 23 ; 2 ; 12 , yet these tools have rarely been applied to demographic bias in a systematic and task-diverse manner. A central challenge is that demographic attributes interact with language in complex ways. In many real-world settings, demographic information may be explicitly stated (e.g., “a Black patient,” “a Hispanic speaker”) or implicitly conveyed through linguistic, cultural, or geographical cues, i.e. the “proxy” cues. Moreover, the same demographic signal can have qualitatively different effects depending on the task: it may alter predicted medical risk in a clinical scenario, while simultaneously modulating perceived toxicity, credibility, or intent in open-ended generation. Existing evaluation typically isolate a single task or domain 10 ; 24 ; 14 , making it difficult to assess whether demographic sensitivity reflects general representational mechanisms or task-specific heuristics. In this work, we investigate how demographic attributes influence LLM behavior, with a focus on mechanistic explanations rather than surface-level disparities. We examine race and ethnicity as commonly occurring coarse-grained categories (e.g. White, Black, Asian, Hispanic and Latino) as they appear in the studied datasets, rather than attempting to model the full sociological complexity of these constructs. Using two publicly available datasets, we study: 1) toxicity-related generation tasks 10 , where the same attributes may alter the likelihood, tone, or framing of model outputs, and 2) clinical narrative tasks 3 , where the same attributes appear through explicit or indirect cues in medical text and modulate model behavior despite identical clinical evidence. We adopt a mechanistic interpretability (MI) framework to study how lexical cues of race and ethnicity are encoded and propagated within three open-source LLMs that are widely used: Qwen2.5-7B 19 , Mistral-7B 11 , and Llama-3.1-8B 8 . Our contributions are threefold: • a reproducible MI pipeline that combines multi-class probing, neuron-level attribution, and targeted intervention to identify internal units associated with demographic attributes and to examine their functional relevance across tasks. The proposed framework is applicable to other social variables beyond race and ethnicity. • a fine-grained characterization of race and ethnicity representations across LLMs, revealing the distributed nature of demographic information and model-specific emphasis on semantic facets such as geography, language, culture, or historical context. • a mechanistic analysis of how demographic representation influences model behaviors. Although internal features encode sensitive or harmful stereotype-related concepts present in pretraining data, these representations are unevenly activated by direct and indirect demographic cues. Our findings show that while race- and ethnicity-associated representations can be identified and analyzed at the neuron level, their associations with biased model behavior persist even when highly active neurons are surpressed. This indicates that biased behavior in LLMs cannot be fully explained or controlled by manipulating a small set of identifiable neurons alone. 2 Related Work Figure 1: With MLP, we locate neurons relevant to race information and inspect them via Logit Lens. For the higher activation score for target race, we adjust its value to steer model’s behavior. Mechanistic Interpretability of Bias in LLMs. Recent works in mechanistic interpretability have begun to locate where demographic information is encoded. Our approach of using probe-based neuron localization aligns with emerging research in this space. 23 utilized neuron editing to understand and mitigate gender bias, identifying specific “gender neurons” within the MLP layers. In the medical domain, 1 investigated the mechanisms of demographic bias specifically for healthcare tasks, suggesting that certain internal representations are disproportionately sensitive to racial identifiers. More recently, 2 explored the use of Sparse Autoencoders (SAEs) to reveal clinical racial biases. Our work extends these findings by demonstrating that race-specific neurons are not only present in general datasets but are consistently activated and influential on domain-specific text. Internal Bias Mitigation and Steering within LLMs. Beyond identification, recent work focuses on manipulating internal model states to ensure fairness. 27 proposed the UniBias framework, which mitigates bias by manipulating attention heads and MLP components. For real-time applications, 15 introduced FairSteer , a dynamic activation steering method that adjusts model behavior during inference. 12 further demonstrated that interpretability-based interventions can improve fairness more robustly than traditional fine-tuning in realistic settings. Our methodology contributes to this line of work by providing a targeted intervention strategy, specifically sign-flipping and amplification, to suppress biased pathways. This builds upon the “context-aware” fairness frameworks suggested by 16 , ensuring that mitigation is grounded in the semantic understanding of the racial directions we extract. Racial Bias in Clinical LLMs. Extensive research has documented that LLMs inherit and propagate racial biases when applied to clinical decision support. 25 demonstrated that ChatGPT exhibits disparate treatment recommendations for acute coronary syndrome based on racial and gender cues. Similarly, 24 evaluated GPT-4, finding that model frequently perpetuates harmful stereotypes that could lead to inequitable health outcomes. 18 further expanded this analysis across various clinical decision-support tasks, highlighting that bias patterns are not idiosyncratic but systematic across model families. While these studies establish the existence of bias, they largely treat model as a black box. Our work seeks to uncover the internal mechanisms driving these disparate outputs. 3 Background MLP Layers and Neuron Activation. Modern Transformer-based LLMs process information through a residual stream . In this framework, the residual stream

Read the complete text