Investigating LLMs in Clinical Triage: Promising Capabilities, Persistent Intersectional Biases
Joseph Lee, Tianqi Shang, Jae Young Baik, Duy Duong-Tran, Shu Yang, Lingyao Li, Li Shen · 2025
Read Investigating LLMs in Clinical Triage: Promising Capabilities, Persistent Intersectional Biases by Joseph Lee, Tianqi Shang, Jae Young Baik, Duy Duong-Tran, Shu Yang, Lingyao Li, Li Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16273.txt.
Introduction
Joseph Lee and six co-authors put language models to work in the emergency department's hardest sorting job, triage, and found skill shadowed by bias. They tested several model strategies, from continued pretraining to in-context learning, against ordinary machine learning. They checked two things: how well the systems survive shifted and missing data, and whether their decisions tilt by a patient's sex and race.
The language models proved the most robust when the data got rough. But the fairness tests found preferences hiding at the intersections. Sex-based gaps appeared broadly, and they grew strongest inside particular racial groups. The bias cannot be seen by checking either trait alone. The authors warn that a model can look even-handed on average while treating specific combinations of patients differently. They argue triage tools must be audited intersectionally before any hospital trusts them with the front door.
Catalog record
| CREATOR | Joseph Lee, Tianqi Shang, Jae Young Baik, Duy Duong-Tran, Shu Yang, Lingyao Li, Li Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating LLMs in Clinical Triage: Promising Capabilities, Persistent Intersectional Biases Joseph Lee, Tianqi Shang, Jae Young Baik, Duy Duong-Tran, Shu Yang, Lingyao Li, Li Shen Abstract Large Language Models (LLMs) have shown promise in clinical decision support, yet their application to triage remains underexplored. We systematically investigate the capabilities of LLMs in emergency department triage through two key dimensions: (1) robustness to distribution shifts and missing data, and (2) counterfactual analysis of intersectional biases across sex and race. We assess multiple LLM-based approaches, ranging from continued pre-training to in-context learning, as well as machine learning approaches. Our results indicate that LLMs exhibit superior robustness, and we investigate the key factors contributing to the promising LLM-based approaches. Furthermore, in this setting, we identify gaps in LLM preferences that emerge in particular intersections of sex and race. LLMs generally exhibit sex-based differences, but they are most pronounced in certain racial groups. These findings suggest that LLMs encode demographic preferences that may emerge in specific clinical contexts or particular combinations of characteristics. 1 Introduction Large language models (LLMs) have shown promise for various clinical settings ( Cascella et al. 2023 ; Park et al. 2024 ; Li et al. 2024c ; Jiang et al. 2023 ) , applications ranging from clinical decision support ( Liu et al. 2023 ) to conversational systems ( Montagna et al. 2023 ) . Amongst these applications, emergency departments (EDs) are among the most dynamic and high-stakes clinical environments, serving as the initial point of contact for patients requiring urgent medical attention ( Franc et al. 2024 ) . A critical component of ED operations is the triage process, where patients are prioritized based on clinical urgency to ensure timely and appropriate care. Overtriage misallocates scarce resources in a time of department overcrowding and undertriage endangers time-sensitive patients ( Chmielewski and Moretz 2022 ) . In an era of historic ED overcrowding, these issues are a growing concern. While triage is an imperfect practice due to limited information ( Hinson et al. 2018 ) , triage practices performed by humans are further prone to challenges such as variability in decision-making, lack of training programs, cognitive biases, and decreased quality due to overwhelming patient arrivals ( Joseph et al. 2023 ; Chmielewski and Moretz 2022 ) . Figure 1: A. represents our evaluation of various models B. denotes our counterfactual analysis, where we perturb the dataset by treating demographics as treatment variables to examine intersectional bias. Traditional machine learning (ML) algorithms have performed well ( Hong, Haimovich, and Taylor 2018 ; Goto et al. 2019 ; Miles et al. 2020 ) , but they are limited by their reliance on structured input data and sensitivity to distributional shifts. Health data in these settings predominantly consists of unstructured text, such as clinical notes, and significantly different triage distributions have been observed across hospitals ( Chmielewski and Moretz 2022 ) which can present major challenges to ML-based methods ( Li et al. 2024c ) . In contrast, the clinical knowledge ( Singhal et al. 2023 ) of LLMs and their reasoning abilities pose them as a robust model, able to adapt to diverse settings. Moreover, LLMs can provide explanations that act as a second opinion for practitioners, mitigating potential mispractice ( Goh et al. 2023 ) . LLMs offer a promising approach to make triage in the ED more efficient, accurate, and equitable ( Preiksaitis et al. 2024 ) . Hence, the demand for artificial intelligence (AI) assistance in this process is substantial ( Friedman, Delgado, and Weissman 2024 ) , and it is already being deployed ( Johns Hopkins Medicine 2022 ) . Despite their potential, the application of LLMs to triage and their robustness remains under-studied. Only a few recent studies have examined their capabilities ( Williams et al. 2024b ; Masanneck et al. 2024 ; Lu et al. 2024 ) . A significant opportunity lies in exploring LLM-based approaches to improve both prediction and clinical utility. Existing studies have predominantly focused on basic prompting techniques without testing advanced methodologies. In addition, we find that most studies have not replicated realistic triage scenarios, as noted by ( Franc et al. 2024 ) , as it is usually limited to a brief description of their symptoms, pain, and vital signs. There are even fewer studies, if any exist, on AI in alternative triage settings, such as telephone triage which involves assessing a patient’s health concerns over the phone. Furthermore, a concern comes from the potential biases and inconsistencies in LLMs when applied to triage decisions, where unfair prioritization leads to adverse outcomes against certain demographics. While research has documented various forms of bias in LLMs across different healthcare contexts ( Guevara et al. 2024 ; Omiye et al. 2023 ; Zack et al. 2024 ; Yang et al. 2024 ) , the implications for hospital triage remain unclear. This knowledge gap is critical given that triage can affect vulnerable patient populations. The high-stakes nature of triage decisions necessitates an extensive assessment of LLMs’ performance across diverse patient populations and clinical scenarios before any potential deployment in real-world settings. Addressing these research gaps is important for advancing the integration of LLMs into clinical environments, and achieving this requires stronger collaboration between the medical and natural language processing (NLP) communities. Toward these goals, our paper makes the following key contributions: 1. We comprehensively evaluate across distribution shifts, missing data, and datasets ranging from real hospital records to curated case studies. We assess various LLM-based approaches, ranging from continued pre-training to in-context learning, providing insights to for future research directions. 2. We propose a novel counterfactual analysis framework to systematically investigate potential biases in LLM predictions, with particular attention to intersections of sex and race. To the best of our knowledge, we are the first to look at the intersectional bias of LLMs, particularly in the clinical setting. 2 Related Works LLMs in Biomedical and Clinical Applications. LLMs have demonstrated impressive abilities across diverse scenarios in the biomedical and clinical domain, including fine-tuning general LLMs with biomedical-specific corpora ( Yunxiang et al. 2023 ; Shu et al. 2023 ; Chen et al. 2023 ; Zhang et al. 2023 ; Labrak et al. 2024 ) and equipping frozen LLMs with techniques like chain-of-thought (CoT) reasoning and retrieval-augmented generation (RAG) ( Jeong et al. 2024 ; Toufiq et al. 2023 ; Lee et al. 2024 ; Li et al. 2024a ; Zakka et al. 2024 ; Lozano et al. 2023 ; Shang et al. 2024 ) . In our study, we focus on a few of these recent approaches that have yet to be applied to triage. For example, studies have shown that careful selection of examples can significantly impact model performance ( Rubin, Herzig, and Berant 2021 ) and few-shot prompting techniques, such as retrieving k-nearest neighbors (KNN), have been proposed to enhance LLM performance by retrieving relevant examples ( Xu et al. 2023 ) . Furthermore, recent studies have explored how LLMs acquire knowledge and expertise ( Ovadia et al. 2023 ; Zhou et al. 2023 ) and we extend these investigations to our specific setting. LLMs in the Emergency Department. Recent studies have explored LLMs in triage tasks. For example, a study