Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization
Suhas, Han-Chin, Lei, Mitch, Jon, Jessica, Jordan R, Susan, Sundararajan, Chaitanya, Jack, Joseph Paul · 2025
Read Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization by Suhas, Han-Chin, Lei, Mitch, Jon, Jessica, Jordan R, Susan, Sundararajan, Chaitanya, Jack, Joseph Paul (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00448.txt.
Introduction
Suhas B N and co-authors study made-up facts in AI summaries of doctor-patient talks, in a 2025 paper. Such errors can harm care and mislead decisions. Whether general error detectors work in medicine was unclear. Errors are also rare and random, which makes them hard to study.
The team built two data sets to force the issue. One removed facts from source talks to trigger false summary content. The other gathered errors that arose naturally in real summaries. General detectors struggled on clinical errors, they found. Skill on forced errors did not predict skill on natural ones. Their own fact-counting detectors, built on the forced set, carried over well to natural errors. The work adds expert-checked data and measures for safer clinical summaries.
Catalog record
| CREATOR | Suhas, Han-Chin, Lei, Mitch, Jon, Jessica, Jordan R, Susan, Sundararajan, Chaitanya, Jack, Joseph Paul |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization Suhas, Han-Chin, Lei, Mitch, Jon, Jessica, Jordan R, Susan, Sundararajan, Chaitanya, Jack, Joseph Paul Abstract Hallucinations in large language models (LLMs) during summarization of patient-clinician dialogues pose significant risks to patient care and clinical decision-making. However, the phenomenon remains understudied in the clinical domain, with uncertainty surrounding the applicability of general-domain hallucination detectors. The rarity and randomness of hallucinations further complicate their investigation. In this paper, we conduct an evaluation of hallucination detection methods in the medical domain, and construct two datasets for the purpose: A fact-controlled Leave-N-out dataset – generated by systematically removing facts from source dialogues to induce hallucinated content in summaries; and a natural hallucination dataset – arising organically during LLM-based medical summarization. We show that general-domain detectors struggle to detect clinical hallucinations, and that performance on fact-controlled hallucinations does not reliably predict effectiveness on natural hallucinations. We then develop fact-based approaches that count hallucinations, offering explainability not available with existing methods. Notably, our LLM-based detectors, which we developed using fact-controlled hallucinations, generalize well to detecting real-world clinical hallucinations. This research contributes a suite of specialized metrics supported by expert-annotated datasets to advance faithful clinical summarization systems. BN Shing Xu Strong Burnsky Ofor Mason Chen Srinivasan Shivade Moriarty Cohen AmazonUSA keywords Hallucination detection, clinical dialogue summarization 1 Introduction The ability to generate precise summaries of patient-clinician dialogues and medical documentation is fundamental to healthcare delivery, enabling rapid comprehension of patient histories and informed clinical decision-making while reducing clinician’s burnout [ 1 , 2 , 3 ] . Language models have enabled the development of automated clinical text summarization systems. However, a major challenge with these systems is the problem of “hallucination” - when the summary includes information that is not present in or misinterpreted from the original text. Hallucinations in clinical summaries pose significant risks to patient care, potentially leading to misdiagnosis, inappropriate treatment decisions, erosion of trust between patients and providers, as well as between providers and their software vendors. As healthcare increasingly adopts automated summarization tools, the ability to detect and prevent such hallucinations becomes crucial for ensuring patient safety and maintaining clinical accuracy. The development of effective hallucination detection methods in clinical summarization is hindered by the lack of suitable datasets. Existing datasets often lack the specificity required for clinical contexts, do not provide controllable hallucinations, and may not capture the granular errors that can occur in medical summaries. To address these limitations, we propose the creation of a fact-controlled synthetic dataset that allows for precise control over the type and extent of hallucinations. We complement this synthetic dataset with another dataset consisting of natural hallucinations to ensure that we can measure performance in both a closed controllable setting and in real-world scenarios. In this work, we systematically explore how existing methods detect hallucinations in clinical text summarization. We utilize two novel datasets created with clinical experts: Leave-N-Out (LNO) and Natural Hallucination (NH), which have omitted facts in different ways to exercise the capabilities of hallucination detection methods. This paper aims to address the following research questions: RQ1: How can we evaluate hallucination detection for medical tasks? RQ2: Does the performance of hallucination detection metrics on synthetic data transfer to real-world clinical summarization data? RQ3: How best can fact based hallucination detection be implemented? To investigate these questions, we evaluate many baseline metrics, including classical, entailment-based, and LLM-based approaches on datasets described in § 3 . Additionally, we develop many fact based methods for clinical summarization in § 4 . Our findings demonstrate that LLM-based methods, particularly those using fact alignment, significantly outperform traditional metrics in hallucination detection. These methods show strong generalization, maintaining high performance across both synthetic and real-world datasets. Notably, our proposed methods also excel in detecting high-severity hallucinations, crucial for ensuring patient safety in clinical applications. 2 Related Work Traditional lexical and semantic overlap metrics such as ROUGE [ 4 ] , BLEU [ 5 ] , and BERT-Score [ 6 ] offer straightforward implementation but struggle with semantic understanding. More sophisticated approaches include entailment-based metrics (e.g., FactCC [ 7 ] , SummaC [ 8 ] , AlignScore [ 9 ] ), question-answering based metrics (e.g., FEQA [ 10 ] , QuestEval [ 11 ] , Q 2 [ 12 ] ), and information theory-based metrics (e.g., InfoLM [ 13 ] ). Recent developments include ensemble approaches like FENICE [ 14 ] and LLM-based metrics such as DocLens [ 15 ] . While DocLens represents an innovative approach to medical text evaluation, its methodology, which relies on claim generation and citation matching, doesn’t directly address the need for hallucination detection in clinical summaries. Furthermore, its partial dependence on reference summaries conflicts with our goal of developing reference-free evaluation methods. 3 Hallucination Datasets We focus our research on the ACI-Bench dataset [ 16 ] , which comprises clinician-patient conversation transcripts and their corresponding clinical summaries (SOAP Notes [ 17 ] ). A SOAP note is a structured way for healthcare providers to document patient information, including Subjective (patient reports), Objective (observable data), Assessment (provider’s evaluation), and Plan (treatment strategy). The transcripts are generally longer and use layman’s terms, while the summaries are written using medical terminology. 3.1 Leave-N-Out Dataset Figure 1: Dataset creation of the Leave-N-Out dataset, where a conversation transcript and its summary are shown side-by-side. The original transcript contains a dialogue about knee pain (in green). In the edited transcript, all references to knee pain are seamlessly replaced (in gray), while the summary remains unchanged and includes the specific detail ‘knee pain’ (highlighted). This illustrates how the dataset systematically removes facts from transcripts while retaining them in summaries to simulate and study hallucinations in text summarization tasks, maintaining the meaning and natural flow of the conversation. In the Leave-N-Out dataset, we use an LLM (Sonnet 3.5 [ 18 ] ) to identify atomic facts from the summary and select N of them (such that they are orthogonal). We then use an LLM to rewrite the transcript to remove these occurrences while maintaining the meaning and natural flow of the conversation. For example, when N=0, it represents the original, unaltered ACI-Bench dataset, while N=1 indicates that all occurrences of a single fact have been removed from the transcript while leaving the summary unchanged. Refer to Figure 1 . For example, complex medical statements are decomposed into atomic facts. The statement “Patient is a 35-year-old male with hypertension” was broken down into three distinct facts: “Patient is 35 years old”, “Patient is male”, and “Patient has hypertension”. Removal of these facts would be considered N=3. Each fact is also annotated with a category of Age & Sex, Exam Findings, Treatment Plan, Symptoms, Labs & Imaging, Medical History, and Diagnosis and this is used to categorize high severity hallucinations as any category but Age & Sex. These edited transcripts were then verified and manually edited by medical scribes to ensure each fact was correctly removed. In total 570 modified transcripts were generated, of these LLM modified transcripts only 428 lines needed to be corrected, meaning 0.75 lines per transcript required human correction. 3.2 Natural Hallucination Dataset The Natural Hallucination (NH) dataset is a collection of model outputs that have hallucinations annotated by clinical scribes.