First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows
Sihao XingSchool of Enterprise Computing and Digital TransformationTechnological University Dublin [email protected] Zaur GoulievSchool of Information and Communication StudiesUniversity College Dublin [email protected] · 2026
Read First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows by Sihao XingSchool of Enterprise Computing and Digital TransformationTechnological University Dublin [email protected] Zaur GoulievSchool of Information and Communication StudiesUniversity College Dublin [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18038.txt.
Introduction
Xing and Gouliev evaluate racial bias in language models used clinically, viewing five models through the EU AI Act across synthetic case generation and diagnosis ranking. Every model departed from observed racial distributions in synthetic cases, with GPT-4.1 showing the smallest deviation, while DeepSeek V3 led diagnosis. Embedding DeepSeek V3 in an agentic workflow improved mean and median p-values and mean difference over the standalone model, though not on every metric. The findings support multi-metric bias evaluation and suggest agentic workflows may reduce some explicit bias.
Catalog record
| CREATOR | Sihao XingSchool of Enterprise Computing and Digital TransformationTechnological University Dublin [email protected] Zaur GoulievSchool of Information and Communication StudiesUniversity College Dublin [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28505 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18038
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows Sihao XingSchool of Enterprise Computing and Digital TransformationTechnological University Dublin [email protected] Zaur GoulievSchool of Information and Communication StudiesUniversity College Dublin [email protected] Abstract Large language models (LLMs) are increasingly used in clinical settings, raising concerns about racial bias in both generated medical text and clinical reasoning. Existing studies have identified bias in medical LLMs, but many focus on single models and give less attention to mitigation. This study uses the EU AI Act as a governance lens to evaluate five widely used LLMs across two tasks, namely synthetic patient-case generation and differential diagnosis ranking. Using race-stratified epidemiological distributions in the United States and expert differential diagnosis lists as benchmarks, we apply structured prompt templates and a two-part evaluation design to examine implicit and explicit racial bias. All models deviated from observed racial distributions in the synthetic case generation task, with GPT-4.1 showing the smallest overall deviation. In the differential diagnosis task, DeepSeek V3 produced the strongest overall results across the reported metrics. When embedded in an agentic workflow, DeepSeek V3 showed an improvement of 0.0348 in mean p-value, 0.1166 in median p-value, and 0.0949 in mean difference relative to the standalone model, although improvement was not uniform across every metric. These findings support multi-metric bias evaluation for AI systems used in medical settings and suggest that retrieval-based agentic workflows may reduce some forms of explicit bias in benchmarked diagnostic tasks. Detailed prompt templates, experimental datasets, and code pipelines are available on our GitHub. * Corresponding author 1 Introduction With the rapid development of artificial intelligence, especially LLMs, AI is becoming increasingly influential in the healthcare industry, including medical text generation and clinical decision-making [ 20 ] . However, racial bias in healthcare remains a serious issue. According to a study by Montalmant and Ettinger [ 21 ] , in 2024, African-American women in the United States were approximately three times more likely to die from pregnancy-related causes than white women. This significant gap existed even when the health conditions were similar. This real-world example reflects the continued presence of structural racial inequality in the healthcare system, where race continues to influence who receives high-quality medical care and outcomes. Against this backdrop, there is increasing concern among both the public and regulators that medical AI systems may inherit or even amplify racial bias. In response, the European Union has introduced a risk-based regulatory framework for AI. In medical contexts, certain AI systems may fall within the AI Act’s high-risk regime, depending on how they are classified under Article 6 and the relevant sectoral route. For systems within scope, the Act places emphasis on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and accuracy and robustness [ 9 ] . In this study, the AI Act is used as a governance lens for structuring evaluation and documentation rather than as a formal legal compliance audit. More generally, racial bias in LLMs can arise from training data, annotation practices, and model design [ 31 ] . Racial bias should always be a major concern when evaluating medical AI systems. This study aims to investigate how LLMs perform in terms of racial bias in medical settings and to assess whether advanced AI technologies can effectively reduce such bias further. Specifically, the research addresses the following questions: • How do different large language models perform in terms of racial bias in healthcare-related tasks? • To what extent can AI agentic workflows mitigate or address racial bias exhibited by large language models in healthcare contexts? 2 Literature Review 2.1 LLMs in the Medical Field LLMs such as ChatGPT and Med-PaLM are transforming the medical field with diverse applications. For instance, the ChatGPT series including GPT-4 can generate clinical letters, medical records, and electronic health records (EHRs) based on conversational input, enabling healthcare professionals to process large-scale patient data more efficiently [ 24 , 38 ] . LLMs are now applied across diverse medical specialties. In oncology, they can summarise cases and provide treatment recommendations for multidisciplinary tumor boards, sometimes aligning closely with expert consensus. In dermatology, innovations such as SkinGPT-4 enable automated analysis of skin images and the generation of diagnostic reports. In mental health, models such as Med-PaLM 2 can assess the risks of depression and Post-Traumatic Stress Disorder (PTSD). However, their growing use has also raised concerns about bias [ 22 ] . 2.2 Implicit and Explicit Racial Bias Implicit racial bias in LLMs emerges when models unconsciously associate certain racial groups with specific diseases or medical characteristics [ 16 ] . Studies have shown that GPT models can associate race with medical outcomes in subtle ways. For instance, they predict higher costs and longer hospital stays for white patients, while using different word frequency patterns when generating HIV-related guidance. Such differences suggest that racial cues may persist in outputs even when sentiment analysis shows no variation [ 40 , 15 ] . Moreover, even models explicitly trained to mitigate bias may still exhibit implicit stereotypes, as revealed through adaptations of the Implicit Association Test [ 2 ] . Explicit racial bias is more evident in clinical decision-making tasks, such as diagnosis and treatment recommendations [ 27 ] . For instance, GPT-4 has been shown to prioritise different diseases or offer more optimistic treatment predictions for white patients, raising concerns about unequal outcomes and the underestimation of disease severity in minority groups [ 41 , 40 ] . Both implicit and explicit racial biases continue to pose challenges in LLM-powered medical applications. Recent studies have therefore focused on developing systematic methods to detect and quantify such biases, laying the groundwork for more rigorous evaluation frameworks. 2.3 Measuring Racial Bias Zack et al. [ 41 ] introduced a systematic framework for evaluating racial bias in healthcare-related LLM applications, focusing primarily on GPT-4. The framework consists of two complementary experiments. The first examines implicit bias through patient-case generation, where race-labeled synthetic cases are produced and their disease frequency distributions compared with epidemiological data. The second investigates explicit bias through differential diagnosis, where only the patient’s racial identity is altered in real cases and the resulting ranked diagnoses are compared with expert baselines. While this framework highlighted the presence of both implicit and explicit racial bias, its scope was limited to a single model. This limitation underscores the need for broader evaluations across different LLMs and clinical contexts. 2.4 Agentic Workflows in the Medical World Recent studies have demonstrated the versatility of AI agentic workflows in the medical domain. For example, agentic frameworks have been used to automate structured information extraction and decision-making in EHRs [ 37 ] , enhance knowledge retrieval and reasoning in complex scientific and clinical tasks [ 18 ] , and streamline biomedical data discovery with interactive, customizable workflows [ 32 ] . These systems commonly integrate LLMs with Retrieval Augmented Generation (RAG), decision optimization, and human-in-the-loop, improving both efficiency and accuracy across diverse medical applications. Yet, empirical evidence on whether such workflows can effectively mitigate racial bias in LLMs remains lacking. 2.5 EU AI Act and LLM Bias The EU AI Act provides an important regulatory context for the governance of AI systems used in healthcare. Under Article 6,