A Counterfactual LLM Framework for Detecting Human Biases: A Case Study of Sex/Gender in Emergency Triage

Ariel Guerra-Adames, Marta Avalos-Fernandez, Océane Dorémus, Leo Anthony Celi, Cédric Gil-Jardiné, Emmanuel Lagarde · 2025

Read A Counterfactual LLM Framework for Detecting Human Biases: A Case Study of Sex/Gender in Emergency Triage by Ariel Guerra-Adames, Marta Avalos-Fernandez, Océane Dorémus, Leo Anthony Celi, Cédric Gil-Jardiné, Emmanuel Lagarde (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17124.txt.

Introduction

Ariel Guerra-Adames and five co-authors audit emergency triage records for sex and gender bias using large language models as measuring instruments. Their method trains a model to imitate recorded triage decisions, then tests it on paired cases that are identical except for the patient's sex. Applied to more than 140,000 admissions at Bordeaux University Hospital, and checked for portability on the MIMIC-IV dataset from a different country and system, the audit can scale where manual review cannot.

Identical presentations were slightly more likely to receive a lower severity score when marked female than male, by roughly one to two percentage points depending on the cohort. Retraining on sex-neutralized records erased the gap, pointing to explicit sex markers as the channel. The authors stress the effects are small and describe documentation, not proven bedside behavior.

Catalog record

CREATORAriel Guerra-Adames, Marta Avalos-Fernandez, Océane Dorémus, Leo Anthony Celi, Cédric Gil-Jardiné, Emmanuel Lagarde
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Counterfactual LLM Framework for Detecting Human Biases: A Case Study of Sex/Gender in Emergency Triage Ariel Guerra-Adames, Marta Avalos-Fernandez, Océane Dorémus, Leo Anthony Celi, Cédric Gil-Jardiné, Emmanuel Lagarde Abstract We present a novel, domain-agnostic counterfactual approach that uses Large Language Models (LLMs) to quantify gender disparities in human clinical decision-making. The method trains an LLM to emulate observed decisions, then evaluates counterfactual pairs in which only gender is flipped, estimating directional disparities while holding all other clinical factors constant. We study emergency triage, validating the approach on more than 150,000 admissions to the Bordeaux University Hospital (France) and replicating results on a subset of MIMIC-IV across a different language, population, and healthcare system. In the Bordeaux cohort, otherwise identical presentations were approximately 2.1% more likely to receive a lower-severity triage score when presented as female rather than male; scaled to national emergency volumes in France, this corresponds to more than 200,000 lower-severity assignments per year. Modality-specific analyses indicate that both explicit tabular gender indicators and implicit textual gender cues contribute to the disparity. Beyond emergency care, the approach supports bias audits in other settings (e.g., hiring, academic, and justice decisions), providing a scalable tool to detect and address inequities in real-world decision-making. 1 Introduction Emergency triage represents one of the most critical decision points in healthcare delivery, where trained professionals must rapidly assess patient acuity and allocate limited resources under significant time pressure. Modern emergency departments employ standardized triage systems, such as the 5-level Emergency Severity Index (ESI) in the United States or the French Emergency Nurses Classification in Hospitals (FRENCH), to stratify patients based on clinical urgency zachariasse2019performance . These systems aim to ensure that patients with the most critical conditions receive immediate care, with triage levels ranging from 1 (requiring immediate resuscitation) to 5 (non-urgent). The stakes of these decisions are profound: undertriage can delay critical interventions and increase mortality risk, while overtriage can overwhelm emergency resources and compromise care for other patients hinson2019triage . Despite standardization efforts, mounting evidence suggests that emergency triage decisions are susceptible to systematic biases related to patient demographics. Multiple studies have documented disparities in triage assignment based on sex (biological and physiological characteristics of males and females) or gender (socially constructed characteristics) coisy2023triageappearance ; han2023suicidebias ; banco2022chestpain ; onal2022throughput ; lopez2021poisoning ; preciado2021acs ; vigil2016patient ; chen2008gender ; croskerry2001emergency ; arslanian-engoren2000triage , race schrader2013racial ; lopez2010racial ; lin2022prioritization ; peitzman2023subjective ; morel2019france ; wu2023emsStroke , age platts2010accuracy , and socioeconomic status kangovi2013understanding ; turner2022inequality ; verma2023emsDelays . Gender bias, in particular, has been observed across diverse clinical presentations. Women presenting with acute coronary syndrome are more likely to receive lower triage acuity scores compared to men mnatzaganian2016sex . Similar patterns have been documented for stroke and abdominal pain amy2019sex , where women experience longer wait times and higher rates of undertriage. These disparities persist even after controlling for clinical factors, suggesting the influence of implicit biases in clinical decision-making fitzgerald2017implicit . It is important to emphasize that, to date, there is no systematic “gold standard” for evaluating triage decisions that would allow one to determine, for each patient, whether they have been over- or under-triaged. While several studies illustrate the usefulness of audit in improving triage quality ouellet2025triage ; zaboli2023audit and provide examples of how such audits can be conducted pilleron2024moral , they do not establish universal reference standards. Patient outcomes may offer some indication of the appropriateness of a triage decision and the association with patient’s sensitive attributes liu2024prognostic , but they cannot account for the contextual information available to the triage nurse at the time of assessment. The emergence of artificial intelligence, particularly Large Language Models (LLMs), offers unprecedented chances to both understand and address these biases. Crucially, while LLMs have demonstrated remarkable capabilities in clinical tasks thirunavukarasu2023large ; singhal2023large ; patel2023chatgpt ; shen2021multimodal , they also notoriously inherit and reproduce many unintended human behaviors, including the systematic biases present in their training data mahajan2025cognitive ; yang2024unmasking ; omiye2023large . This tendency, traditionally viewed as a significant limitation, presents a unique opportunity: LLMs can serve as “bias mirrors,” providing a scalable tool for auditing human decisions. Rather than deploying LLMs to replace human decision-makers, we propose leveraging their ability to faithfully reproduce human behavior—including its biases—as an analytical tool to detect and quantify these disparities in clinical practice. Their ability to process both structured and unstructured clinical data makes them particularly well-suited for emergency triage, where decisions integrate vital signs, chief complaints, and clinical narratives. By training an LLM to accurately emulate existing human triage decisions, we aim to create a high-fidelity model that captures both the clinical reasoning and the biases embedded in current practice. In a previous study, we developed a proof-of-concept of a counterfactual framework based on LLMs to detect bias in emergency triage decisions guerra2025uncovering . This approach (illustrated in Figure 1 ) consists of (1) fine-tuning an LLM to accurately reproduce human triage decisions, (2) generating counterfactual patient pairs where only gender-related information is altered while maintaining clinical factors constant and (3) comparing triage predictions between original and counterfactual presentations using the fine-tuned triage LLM. This initial work raised several key questions: How can systematic disparities be quantified through appropriate bias metrics? How can we isolate the causal effect of patient gender or sex on triage outcomes while controlling for clinically relevant variables? Once bias is isolated and quantified, what is its potential impact on patient care? Finally, are the observed results reproducible across different datasets, languages, countries and healthcare systems? In this new study, we address these questions by developing appropriate metrics and applying the framework to two distinct datasets: a French dataset protected under European (GDPR) and national (CNIL) regulations, and a semi-publicly accessible U.S. dataset. Figure 1: Overview of the counterfactual framework for gender bias detection in emergency triage: (1) LLM fine-tuning for the task of emergency triage, (2) generation of counterfactual pairs according to a sensitive variable like sex/gender, (3) comparison of triage predictions by the fine-tuned LLM on counterfactual pairs. 2 Conceptual Framework Our proposed framework consists of three interconnected components designed to detect and quantify biases from sensitive attributes in emergency triage: (1) a fine-tuned LLM-based triage model that replicates human triage decision-making, (2) an LLM-based counterfactual sample generator that systematically modifies sensitive attributes while preserving clinical content, and (3) a set of triage-specific inequality metrics for bias quantification. The framework takes as input a dataset of patient admissions from an emergency department (ED), where each record contains an ordinal triage score assigned by trained clinical personnel, free-text triage notes documenting the clinical assessment and circumstances, and structured variables capturing vital parameters and contextual factors (e.g., patient demographics, nurse identification, temporal factors). The dataset is partitioned into training and test sets of equal size (50%), assuring that the distribution of triage scores is the same on both partitions. 2.1 Fine-tuning LLMs for predicting emergency triage We frame the task of learning emergency triage from patient data as an ordinal regression problem. The goal is to train a model that predicts the correct triage category. Input Space Let \mathcal{X} be the input space of patient records. Each patient record x\in\mathcal{X} is a composite of two modalities: structured tabular data

Read the complete text