Ethical AI on the Waitlist: Group Fairness Evaluation of LLM-Aided Organ Allocation

Hannah Murray*, Brian Hyeongseok Kim*, Isabelle Lee* · 2025

Read Ethical AI on the Waitlist: Group Fairness Evaluation of LLM-Aided Organ Allocation by Hannah Murray*, Brian Hyeongseok Kim*, Isabelle Lee* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03716.txt.

Introduction

Hannah Murray and co-authors test fairness in language model organ allocation in this 2025 paper. Hospitals weigh using such models for hard choices like kidney allocation. Fair scoring is vital, yet standard tests are too simple and no single right answer exists.

The team first checked whether models know the clinical facts allocation needs. They then set two tasks: pick one candidate for a kidney, or rank a whole waiting list. Fairness was scored by share-based and exposure-based measures. One measure looked fair while another showed groups clustered near the top. Preferences also flipped between the two tasks, even at rank one.

Catalog record

CREATORHannah Murray*, Brian Hyeongseok Kim*, Isabelle Lee*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03716

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ethical AI on the Waitlist: Group Fairness Evaluation of LLM-Aided Organ Allocation Hannah Murray*, Brian Hyeongseok Kim*, Isabelle Lee* Abstract Large Language Models (LLMs) are becoming ubiquitous, promising automation even in high-stakes scenarios. However, existing evaluation methods often fall short—benchmarks saturate, accuracy-based metrics are overly simplistic, and many inherently ambiguous problems lack a clear ground truth. Given these limitations, evaluating fairness becomes complex. To address this, we reframe fairness evaluation using Borda scores, a method from voting theory, as a nuanced yet interpretable metric for measuring fairness. Using organ allocation as a case study, we introduce two tasks: (1) Choose-One and (2) Rank-All . In Choose-One , LLMs select a single candidate for a kidney, and we assess fairness across demographics using proportional parity. In Rank-All , LLMs rank all candidates for a kidney, reflecting real-world allocation processes. Since traditional fairness metrics do not account for ranking, we propose a novel application of Borda scoring to capture biases. Our findings highlight the potential of voting-based metrics to provide a richer, more multifaceted evaluation of LLM fairness. 1 Introduction The growing capabilities of LLMs have sparked considerable interest across various domains including healthcare. Medical institutions are actively exploring the potential of LLMs in clinical settings. Recently, the Duke Institute for Health Innovation announced its efforts to leverage LLMs and generative AI for organ allocation ( Duke Institute for Health Innovation 2024 ) . Although these technologies offer promise, their use in healthcare raises significant concerns ( Ayoub et al. 2024 ; Qu & Wang 2024 ; Gallegos et al. 2024 ) . Proponents of LLMs argue that LLMs could improve organ allocation by analyzing vast amounts of data and uncovering intricate patterns that clinicians might overlook ( Meng et al. 2024 ) . This high reward, however, may also come with a high risk, with concerns about the transparency and fairness of its decision-making process. Current LLM evaluations remain brittle, often relying on simplistic metrics and tests, such as multiple-choice questions ( Balepur et al. 2025 ) and saturating benchmarks ( Saxon et al. 2024 ) . Organ allocation may appear straightforward: given a donated kidney and a waitlist of candidates, select the most suitable candidate. However, the reality is much more intricate and involves numerous conflicting and morally ambiguous considerations. Decisions about optimality must balance factors such as age, medical urgency, blood types, and antigens. Other factors, such as the amount of time a candidate has been on the waitlist and the distance to the donated kidney, are less clear-cut, while demographic characteristics, such as income or education level, should ideally not influence outcomes. This complexity creates moral ambiguity, as there is often no universally ”correct” answer on how to determine the best candidate for a given organ. These challenges are compounded by technical limitations of LLMs, including context length constraints (the number of tokens an LLM can process and effectively utilize in processing a prompt), inconsistency in results, biases toward certain groups, and understanding of morality ( Jiang et al. 2022 ; Kantharuban et al. 2024 ) . This raises questions about whether LLMs can be effectively and safely used in high-stakes, morally sensitive applications like organ allocation. Therefore, a methodological evaluation of LLMs is needed to ensure both their effectiveness and fairness in organ allocation. As such, our goal was to robustly evaluate the use of LLMs for high-stakes decision-making, using organ allocation as a case study. We assessed their performance both quantitatively and qualitatively, focusing on their ability to answer these questions: 1. Medical Viability: Can LLMs effectively utilize medically relevant features to assess the viability of organ-candidate matches? 2. Group Fairness: Do LLMs exhibit biases toward certain demographic groups when choosing or ranking candidates on the waitlist for organ transplants? We experimented with a dataset from the Organ Procurement and Transplantation Network (OPTN), 1 1 1 https://optn.transplant.hrsa.gov/ which contains records of kidney donors and candidates on the waitlist for kidney transplantation dating back to the 1980s. The dataset includes medically relevant features, such as blood type, height, weight, quality-of-life indicators, and demographic attributes, such as gender, ethnicity, education level, and employment status. Features used in our experiments are detailed in Table 2 in § A.1 . Our Contributions We conducted the first systematic study that evaluates three state-of-the-art LLMs in the kidney allocation task using real data from OPTN. We evaluated LLMs’ proficiency in using medical knowledge to identify viable kidney-candidate matches under various experimental conditions, including the presence or absence of instructions and examples. Furthermore, we proposed two tasks, Choose-One and Rank-All , to evaluate the fairness of LLMs in organ allocation. These tasks simulate real-world settings and measure fairness across demographic groups. Lastly, we applied existing group fairness metrics and propose a novel method using Borda scoring, a method rooted in voting theory, to evaluate LLM performance and fairness in domains where a single correct answer may not exist. Together, these contributions provide a comprehensive evaluation of LLMs’ potential and limitations in organ allocation, addressing both their medical proficiency and fairness in decision-making, and offering insights into their ethical use in high-stakes settings. 2 Methods 2.1 Medical Viability: Do LLMs Correctly Assess Organ-Candidate Compatibility? We first randomly selected five available kidneys and five potential candidates currently on the waitlist. We tasked three state-of-the-art LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0 Pro) to determine if each kidney-candidate pair was medically viable. We define “ground truth” medical viability criteria from prior literature: 1) donor and candidate height less than 12.5 cm apart, 2) donor and candidate weight less than 15kg apart, and 3) blood type compatibility ( Tandukar et al. 2022 ) . Other medical criteria, such as antigen matches, are not required in all cases, so we excluded these features from our first experiment. We tested three prompting strategies for each LLM: out-of-box (no criteria or examples provided), zero-shot (only criteria provided), and few-shot (criteria and four examples provided). “Criteria” refers to explicitly instructing the LLMs on which features to consider, while “examples” involves providing benchmark kidney-candidate pairs along with explanations of their medical viability. For the out-of-box and zero-shot setting, we ran each kidney-candidate viability problem 5 times, resulting in 125 trials per LLM. For the few-shot setting, we ran each kidney-candidate viability problem 15 times, each with a different set of examples, resulting in 375 trials per LLM. Every set consisted of one example of a viable match and three examples of non-viable matches (each for height, weight, and blood type). Examples of prompts for each strategy are in § A.7 . Performance was measured using accuracy, precision, recall, and F1 score. Accuracy provides a general assessment of model performance. Precision measures the proportion of correctly identified viable kidney-candidate pairs among all predicted matches. High precision ensures that most predicted matches are indeed medically viable, minimizing false positives (incorrect matches). Recall assesses how well the model detects all viable matches. It represents the proportion of medically viable kidney-candidate pairs that are correctly identified. A high recall means the model captures most viable matches, reducing false negatives (missed matches). The F1-score, the harmonic mean of precision and recall, balances these two metrics. These metrics are defined in § A.2 . 2.2 Group Fairness: Do

Read the complete text