AMQA: An Adversarial Dataset for Benchmarking Bias of LLMs in Medicine and Healthcare
Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang, [0.5em] King’s College London · 2025
Read AMQA: An Adversarial Dataset for Benchmarking Bias of LLMs in Medicine and Healthcare by Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang, [0.5em] King’s College London (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19562.txt.
Introduction
A team of co-authors wrote a paper in two thousand twenty-five introducing a benchmark for bias in medical AI, called FairMedQA. Language models now answer medical questions at near-expert level. Yet bias tied to traits such as sex and race could put patients at risk, and its size in diagnosis is still an open question.
The benchmark holds over four thousand paired questions built from eight hundred one clinical cases. Each pair changes only the patient's demographic traits, so any change in the answer exposes bias. Twelve models were tested. Accuracy gaps between groups ran from three to nineteen points, larger than a prior benchmark could detect.
Catalog record
| CREATOR | Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang, [0.5em] King’s College London |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AMQA: An Adversarial Dataset for Benchmarking Bias of LLMs in Medicine and Healthcare Ying Xiao, Jie Huang, Ruijuan He, Jing Xiao, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang, [0.5em] King’s College London Abstract Large language models (LLMs) are reaching expert-level accuracy on medical diagnosis questions, yet their mistakes and the biases behind them pose life-critical risks. Bias linked to race, sex, and socioeconomic status is already well known, but a consistent and automatic testbed for measuring it is missing. To fill the gap, this paper presents AMQA—an Adversarial Medical Question-Answering dataset—built for automated, large-scale bias evaluation of LLMs in medical QA. AMQA has 4,806 medical QA pairs sourced from the United States Medical Licensing Examination (USMLE) dataset, generated by a multi-agent framework to generate different adversarial descriptions and question pairs. Using AMQA, we benchmark five representative LLMs and find surprisingly substantial disparities: even GPT-4.1, the least biased LLM tested, answers privileged-group questions over 10 percentage points more accurately than unprivileged ones. Compared with the existing benchmark CPV, AMQA reveals 15% larger accuracy gaps on average between privileged and unprivileged groups. Our dataset and code are publicly available ( https://github.com/XY-Showing/AMQA ) to spur reproducible research and advance trustworthy, bias-aware medical AI. 1 Introduction Large language models (LLMs) have reached a pivotal milestone in medicine and healthcare: on numerous clinical benchmarks, their performance now rivals or even exceeds that of board-certified physicians [ 2 , 30 , 34 ] . By synthesizing vast datasets spanning biomedical literature, electronic health records, and global medical knowledge, LLMs hold promise for expanding access to expert-level guidance, optimizing healthcare workflows, and enhancing patient outcomes while reducing costs. Nevertheless, LLMs also produce confidently stated errors, especially when involving historically unprivileged groups, such as those defined by race or socioeconomic status. In healthcare, where disparities already endanger unprivileged groups, these errors and biases pose not just theoretical and ethical risks but also life-threatening consequences. A growing body of evidence shows that LLM outputs vary with sensitive attributes such as race, sex, or socioeconomic status in medicine and healthchare [ 28 , 38 , 19 ] . For example, recent studies report Claude perpetuated incorrect race-based formulas to calculate kidney function and lung capacity calculations for the Black [ 24 , 25 ] . Despite these warnings, the community lacks a standard, automated way to test LLMs for such biases and to challenge LLMs to improve the accuracy for unprivileged groups. Existing fairness benchmarks in NLP rarely cover clinical content, while medical QA datasets such as MedQA or PubMedQA focus on factual diagnostic accuracy, lacking the diversity, demographic annotations, and adversarial scenarios needed to expose fairness-related failures across patient subgroups. There are several existing benchmarks targeting fairness in LLMs for medicine and healthcare, such as EquityMedQA [ 13 ] and the Counterfactual Patient Variations (CPV) dataset [ 1 ] . However, EquityMedQA focuses on open-ended questions and relies on human experts for bias assessment, making it resource-intensive and difficult to scale; CPV struggles to effectively trigger bias in LLMs, as it simply modifies a single sensitive attribute (e.g., changing “Male” to “Female”) without adversarially generating perturbations. Such minimal changes are often insufficient to significantly expose deeper bias patterns. There remains a critical need for effective, efficient, standardized, and scalable benchmarks with objective ground-truth answers to support automated and reproducible bias evaluation of LLMs in medical scenarios. To fill these gaps, we introduce AMQA , an A dversarial M edical Q uestion A nswering dataset . AMQA contains 4,806 carefully constructed question pairs derived from the United States Medical Licensing Examination (USMLE) corpus [ 11 ] , a gold-standard dataset for clinical knowledge assessment. Each pair includes a clinical vignette and its variant, created by varying demographic attributes such as race, sex, or socioeconomic status, while keeping clinical details constant. This design enables controlled testing of how LLM performance differs across privileged and unprivileged groups. AMQA adopts a novel multi-agent framework that generates adversarial patient descriptions. Crucially, AMQA includes only vignettes where the varied content does not influence the medical outcome. For example, a case involving a gynecological condition would not be used to create a male version. To ensure fairness and diagnostic neutrality, all items are manually reviewed, and those where sensitive attributes could affect the correct answer are excluded. Using AMQA, we benchmark five state-of-the-art LLMs (including GPT-4.1, Claude-3.7-sonnet, Gemini-2.0-flash, Qwen-max, and Deepseek-v3) and uncover striking disparities. For example, even GPT-4.1, which is the least biased LLM we test, answers questions about privileged groups with 10% higher accuracy than those about unprivileged groups. The most biased LLM exhibits an accuracy gap of more than 28%. Compared to the existing medical QA benchmark CPV, AMQA reveals 15% larger accuracy gaps on average. These findings underscore the critical need for automatic and systematic bias evaluation in medical LLMs, especially considering that even subtle disparities in high-stakes clinical settings can reinforce or worsen existing inequities in healthcare access and outcomes. Overall, we make the following key contributions: 1. We introduce AMQA , a novel adversarial dataset for bias evaluation in medical QA, enabling automated benchmarking without manual evaluation by medical experts; 2. We conduct a comprehensive empirical evaluation, uncovering systematic biases across sensitive attributes such as race, gender, and socioeconomic status in LLMs for healthcare; 3. We publicly release the AMQA dataset, accompanying evaluation tools, and the evidence of bias in current LLMs, to facilitate future research on medical bias and trustworthy AI. 2 Related Work 2.1 Medical Bias in Large Language Models Medical bias refers to instances where an LLM produces discriminatory, inaccurate, or misleading outputs in response to clinical scenarios due to sensitive attributes such as race or sex, rather than on clinical grounds [ 14 , 35 , 2 , 25 , 40 , 36 , 1 , 8 , 12 , 29 ] . Bias is typically rooted in social power structure, encoded in real-world data, and inherited by LLMs through large-scale training [ 2 , 9 , 14 ] . In the healthcare context, bias can arise in diverse ways, reflecting the complexity of medical tasks and real-world patient variation [ 5 , 21 ] . More specifically, in medical QA, such bias often manifests as disparities in clinical recommendations [ 33 ] or diagnostic accuracy [ 24 ] across demographic groups. Several recent studies have highlighted concrete examples of these issues. Omiye et al. [ 24 ] show that the Claude model from Anthropic erroneously applies a race-based formula when estimating lung function, leading to inaccurate results for Black patients. Similarly, Pfohl et al. [ 28 ] find that altering a patient’s race in a clinical vignette can lead an LLM to produce different medical decisions, even when the clinical details remain unchanged. According to expert evaluations on their EquityMedQA dataset [ 13 ] , 12.6% of LLM responses are found to be biased. 2.2 Datasets and Benchmarks for Medical Bias Evaluation Data plays a vital role in the lifecycle of artificial intelligence systems, spanning model pre-training, fine-tuning, and evaluation [ 9 , 2 ] . Among these stages, benchmark datasets for evaluating LLMs in healthcare are particularly crucial, as they provide standardized and reproducible grounds for assessing model behavior