When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models

Yubo Li, MS 1, Ramayya Krishnan, PhD 1, Rema Padman, PhD 1 · 2026

Read When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models by Yubo Li, MS 1, Ramayya Krishnan, PhD 1, Rema Padman, PhD 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21460.txt.

Introduction

Li and co-authors measure how much transplant patient education varies across United States centers. They ground the same patient questions in 102 handbooks from 23 centers, using retrieval-augmented language models. A five-label taxonomy then scores how much the grounded answers agree. They find that about one fifth of comparisons diverge in clinically meaningful ways, mostly on monitoring and lifestyle topics. Coverage gaps are larger still: most question-handbook pairs lack the relevant content entirely.

Catalog record

CREATORYubo Li, MS 1, Ramayya Krishnan, PhD 1, Rema Padman, PhD 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31657
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models Yubo Li, MS 1, Ramayya Krishnan, PhD 1, Rema Padman, PhD 1 Abstract \institutes 1 Carnegie Mellon University, Pittsburgh, PA, USA Abstract Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers’ handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement. 1 Introduction Large language models (LLMs) are increasingly used to answer medical questions and support patient education, often through retrieval-augmented generation (RAG), in which responses are grounded in external documents such as clinical guidelines or institutional patient education materials. While prior work has focused on improving retrieval quality and reducing hallucinations, less attention has been paid to an upstream issue: can the underlying source documents themselves contain differing guidance? If so, what are the similarities and differences between handbooks on specific topics and what is their provenance? This question is especially relevant in solid-organ transplantation, where substantial center-level variation is well documented. Analyses of national registry data show that the probability of receiving a deceased-donor kidney transplant within three years of waitlisting varies dramatically across centers within the same donation service area 3 . These institutional differences extend to patient-facing information: transplant center websites provide incomplete and inconsistent recipient selection criteria 8 , patient education materials vary widely in readability and quality 10 ; 7 , and a comparative analysis of transplant handbooks using NLP and generative methods found significant variation in the availability and interpretation of clinical guidance across centers 6 . These findings raise a critical question: when patients ask the same or a similar question to different centers’ patient handbook documents, does the resulting advice differ in clinically meaningful ways? If so, content selection for patient handbooks has implications for both clinical and management decision making. In this work, we address this problem by grounding the same patient questions in handbooks from different U.S. transplant centers and systematically comparing the resulting responses. Our goal is not to identify a single correct answer but to measure the extent and structure of institutional variation across organs, topics, and centers, contributing to broader efforts to understand the information gaps in transplant patient education within the U.S. transplantation system. 2 Methods 2.1 Data Sources and Processing Our study draws on two primary data sources: a corpus of transplant patient education handbooks that were generously shared by U.S. transplant centers and assembled by the non-profit Transplants.org for analysis, and a curated benchmark of patient questions extracted from multiple patient forums, spanning five solid-organ types. This section describes the collection, scope, and characteristics of each. Transplant Patient Handbooks. We obtained a corpus of 102 patient education handbooks from 23 major U.S. solid-organ transplant centers, representing 16 of the nation’s 20 largest programs by volume. The corpus spans five organ types — heart (26), lung (26), kidney (22), liver (17), and pancreas (11) — and the contributing centers are geographically distributed across the United States (Figure 1 ), covering both large academic medical centers and community-based transplant programs. All documents were obtained as PDFs from the institutions. Table 1 summarizes the corpus by organ type. Figure 1 : Center distribution map for the handbook corpus. The 23 contributing U.S. transplant centers are geographically dispersed across the country. Centers vary in how they organize patient education materials: some provide separate documents for the pre-transplant phase (evaluation, listing, and waiting) and the post-transplant phase (recovery, medications, and long-term follow-up), while others issue a single combined handbook. We treat each phase-specific document as a distinct unit, yielding 37 pre-transplant, 39 post-transplant, and 26 combined handbooks. Each is assigned a unique identifier encoding organ type, institution, and care phase. Transplant Patient Question Set. We curated a benchmark of 1,115 patient questions to serve as the evaluation set for cross-center comparison. Questions were collected from diverse sources reflecting the real information needs of transplant patients and caregivers. Source types include healthcare institution Q&A pages (31.2%), community forums such as Reddit and Mayo Clinic Connect (25.1%), medical organizations including the National Kidney Foundation and the American Liver Foundation (24.9%), and other sources including government health agencies and patient advocacy sites (18.8%). Table 1 : Summary of the transplant handbook corpus by organ type. Centers indicates the number of distinct institutions contributing handbooks for each organ. Heart Kidney Liver Lung Pancreas Handbooks 26 22 17 26 11 Centers 17 14 11 15 8 Pre-transplant 10 8 5 10 4 Post-transplant 11 10 4 11 3 Combined 5 4 8 5 4 Each question is annotated with: (i) an organ type label (heart, kidney, liver, lung, pancreas, or general), (ii) one or more clinical topic categories drawn from a 13-topic taxonomy, and (iii) fine-grained sub-topic tags (43 unique sub-topics). As shown in Figure 2(b) , the 13 topic categories are: Medical Complications (28.6% of topic annotations), Reproductive Health (26.1%), Lifestyle & Daily Living (20.2%), Pre-Transplant (15.8%), Medications (9.9%), Monitoring & Follow-up (9.7%), Mental & Emotional Health (9.6%), Surgery & Recovery (7.0%), Special Populations & Education (6.5%), and four smaller categories covering transplant logistics, financial issues, and community support. Questions are multi-labeled: a single question may be annotated with more than one topic and sub-topic to reflect cross-cutting concerns. General-type questions account for the largest share of the benchmark (27.9%; Figure 2(a) ) and address topics that span organ types, such as immunosuppressant side effects, reproductive health after transplantation, and mental health. At generation time, these questions are answered by every handbook in the corpus, while organ-specific questions are answered only by handbooks of the matching organ type. This design ensures comprehensive cross-center coverage for broadly relevant topics while maintaining clinical specificity for organ-level questions. Questions were lightly paraphrased for anonymization and to ensure they are self-contained (i.e., interpretable without conversational context). The original sources were predominantly U.S.-based (69.9% geolocated to the United States), consistent with the U.S. institutional focus of the handbook corpus. The benchmark was drawn from a larger pool of over 3,000 candidate questions collected across organ-specific sources, which were filtered for relevance, deduplication, and quality to produce the final set of 1,115. (a) By organ type. (b) By topic category. Figure 2 : Distribution of benchmark questions by organ type and topic category. 2.2 Document Extraction Figure 3 illustrates the end-to-end pipeline. Raw PDF handbooks are converted to structured JSON representations using LlamaParse 5 , a document parsing service that preserves section headings, paragraph boundaries, and page metadata. The output for each handbook is

Read the complete text