Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare

Hiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, Byron C. Wallace · 2025

Read Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare by Hiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, Byron C. Wallace (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13319.txt.

Introduction

Large language models can treat patients unfairly in medical tasks, and Hiba Ahsan and co-authors set out in 2025 to find where that bias sits inside the model. They used tools from a field that probes the inner workings of such models, asking whether inner signals carry facts like a patient's gender or race. Their test bed was health care, where biased output can change real care.

Gender signals turned out to sit in a small set of layers and could be flipped while the model ran. Flipping them changed the case stories the model wrote and shifted risk scores tied to gender, such as for depression. Race signals were spread more widely but could still be nudged. The work shows that bias in these models has a physical home that can, in part, be adjusted.

Catalog record

CREATORHiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, Byron C. Wallace
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13319

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare Hiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, Byron C. Wallace Abstract We know from prior work that LLMs encode social biases, and that this manifests in clinical tasks ( Gerszberg, 2024 ; Zack et al., 2024 ; Zhang et al., 2020 ) . In this work we adopt tools from mechanistic interpretability to unveil sociodemographic representations and biases within LLMs in the context of healthcare. Specifically, we ask: Can we identify activations within LLMs that encode sociodemographic information (e.g., gender, race)? We find that, in three open weight LLMs, gender information is highly localized in MLP layers and can be reliably manipulated at inference time via patching. Such interventions can surgically alter generated clinical vignettes for specific conditions, and also influence downstream clinical predictions which correlate with gender, e.g., patient risk of depression. We find that representation of patient race is somewhat more distributed, but can also be intervened upon, to a degree. To our knowledge, this is the first application of mechanistic interpretability methods to LLMs for healthcare 1 1 1 Our code is available at https://github.com/hibaahsan/interp-healthcare-bias/ . 1 Introduction LLMs are poised to transform the practice of healthcare in many ways ( Nori et al., 2023 ; Dash et al., 2023 ; Singhal et al., 2023 ) , given the volume of unstructured health data and limited provider bandwidth ( Zhou et al., 2023 ) . Such models are capable of a wide range of tasks related to processing and making sense of healthcare data ( Thirunavukarasu et al., 2023 ) , from summarizing published medical literature ( Shaib et al., 2023 ) to extracting key information from the notes within patient electronic health record (EHR) data ( Agrawal et al., 2022 ; Ahsan et al., 2024 ) . Indeed, excitement around such uses is driving fast adoption: Epic—a major vendor of EHR software—has hastily integrated GPT-4 into its platform, making it directly accessible to caregivers ( Epic Systems Corporation, 2023 ) . But enthusiasm around the uptake of LLMs in this space has been tempered by concerns over fairness and the opaque nature of large generative models ( Haltaufderheide and Ranisch, 2024 ) . One salient concern—which preliminary work suggests is very much warranted—is that such models might exacerbate existing biases in healthcare. For instance, recent work by Zack et al. (2024) found that GPT-4 exaggerates associations between conditions and sociodemographic groups. Specifically, when asked to generate clinical vignettes of patients with particular conditions, GPT-4 will nearly exclusively assume certain demographics (e.g., race, gender). For example, asked to generate vignettes for patients with rheumatoid arthritis , GPT-4 generates cases featuring female patients 97% of the time (the actual percent of individuals with rheumatoid arthritis who are female is about 66%; Linos et al. 1980 ). Similarly, GPT-4 associates sarcoidosis with Black patients and hepatitis B with Asian patients more strongly than actual population-wide correlations. Figure 1: We show that we can localize patient gender information in LLM representations for clinical tasks. In this work, we ask: Is the internal LLM encoding of patient demographics like gender and race localized? And, can we intervene upon this? To answer this we perform activation patching ( Heimersheim and Nanda, 2024 ) in the context of clinical vignette generation. The idea is to identify a small set of internal activations which code for patient characteristics like gender, and then verify these by intervention. We offer the following contributions: (1) We find that gender 2 2 2 Following prior work Zack et al. (2024) , we use “gender” instead of “sex” because of the ambiguity in how LLMs use the terms “male” and “female” (biological vs sociocultural). Most ( \sim 60\% ) vignettes generated by OLMo state “gender” followed by “male” or “female”. Our analysis is also limited to binary gender categories, owing to limitations of the data. information is highly localized in MLP layers. In two of the four models, patching MLP activations of a single layer consistently alters patient gender in generated texts. Gender information can also be localized in conditions such as prostate cancer (exclusive to males) and preeclampsia (females). (2) Race representations are more complicated: Multiple token activations in early and middle MLP layers correspond to patient race. We are able to intervene to “alter” race to a degree. (3) We use two downstream clinical tasks to show how patching demographic information can be used to study implicit biases encoded in LLMs. To our knowledge, this is the first investigation of mechanistic interpretability methods for healthcare. 2 Localizing patient gender 2.1 Vignette Generation Zack et al. (2024) found that GPT-4 exaggerates differences between demographic groups with respect to clinical conditions. Specifically, when asked to generate clinical vignettes of patients with particular conditions, GPT-4 will nearly exclusively assume certain demographics (e.g., race, gender). For instance, asked to generate vignettes for patients with rheumatoid arthritis , GPT-4 generates cases featuring female patients 97% of the time. 3 3 3 The actual percent of individuals with rheumatoid arthritis who are female is about 66% Linos et al. (1980) . Is the encoding of patient gender localized or distributed in the LLM? To answer this we use activation patching Heimersheim and Nanda (2024) in the context of vignette generation. We aim to find a small set of internal activations that encode patient gender: Using these we ought to be able to “change” the gender in a generated vignette by manipulating these activations. This requires access to model internals, so we adopt open-source models in place of the opaque GPT-4. We focus on the OLMo-7B-Instruct model throughout this paper, but show that our findings generalize across models and scale with three additional models: Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 ) , Gemma-2-9B-it ( Riviere et al., 2024 ) , and OLMo-2-0325-32B-Instruct ( OLMo et al., 2024 ) . Condition M F Sarcoidosis 0.02 0.98 Multiple Sclerosis 0.06 0.94 Table 1: Proportion of Male (M) and Female (F) patients in OLMo-7B-Instruct -generated vignettes. Prompt Compose a brief presentation of a patient presenting with Multiple Sclerosis. Please include complete demographic information and past medical history. Before intervention After activation patching Ms. Sarah Johnson Mr. John Smith Gender: Female; Age: 42 Gender: Male; Age: 45 Race: White Birthplace: New York, USA … … Past medical history Past medical history Multiple Sclerosis (MS) Multiple Sclerosis (MS) Hypertension: Diagnosed at 40 Hypertension (high blood pressure) Diabetes Mellitus Type 2 Diabetes Mellitus Type 2 Table 2: Patient vignettes generated by OLMo-7B-Instruct for a patient with Multiple Sclerosis before (left) and after (right) patching in the “male” activation pattern. This intervention alters patient gender, but not other attributes. To find gender-encoding activations, we first prompt the LLM to generate a vignette for a condition (strongly) associated with females (or males). Specifically, following prior work Zack et al. (2024) , we prompt the LLM to provide a succinct description of the patient—including symptoms, medical history, and demographic information—using the same 10 prompts introduced in this prior study. One of the prompts, e.g., is: Compose a brief presentation of a patient presenting with [CONDITION] . Please include complete demographic information and past medical history. We run each prompt for every condition

Read the complete text