Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Models

Anindya Bijoy Das, Shibbir Ahmed, Shahnewaz Karim Sakib · 2025

Read Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Models by Anindya Bijoy Das, Shibbir Ahmed, Shahnewaz Karim Sakib (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19061.txt.

Introduction

Anindya Bijoy Das, Shibbir Ahmed, and Shahnewaz Karim Sakib test open-source language models on hospital discharge summaries, checking both what they extract and what they invent. Summaries must carry the reason for admission, the main events of the stay, and the follow-up steps, in exact form. Errors or made-up facts in such text can harm care. Their 2025 study runs broad simulations across several open models.

Models such as Qwen2.5 and DeepSeek-v2 captured admission reasons and in-stay events quite well. They were less steady on follow-up recommendations, often dropping or garbling them. The team also sorted the hallucinations the models produced into types and counted each kind. Their verdict is that open models are close to useful for this task but not yet trustworthy on the details that guide what a patient does next.

Catalog record

CREATORAnindya Bijoy Das, Shibbir Ahmed, Shahnewaz Karim Sakib
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Models Anindya Bijoy Das, Shibbir Ahmed, Shahnewaz Karim Sakib Abstract Clinical summarization is crucial in healthcare as it distills complex medical data into digestible information, enhancing patient understanding and care management. Large language models (LLMs) have shown significant potential in automating and improving the accuracy of such summarizations due to their advanced natural language understanding capabilities. These models are particularly applicable in the context of summarizing medical/clinical texts, where precise and concise information transfer is essential. In this paper, we investigate the effectiveness of open-source LLMs in extracting key events from discharge reports, such as reasons for hospital admission, significant in-hospital events, and critical follow-up actions. In addition, we also assess the prevalence of various types of hallucinations in the summaries produced by these models. Detecting hallucinations is vital as it directly influences the reliability of the information, potentially affecting patient care and treatment outcomes. We conduct comprehensive numerical simulations to rigorously evaluate the performance of these models, further probing the accuracy and fidelity of the extracted content in clinical summarization. I Introduction Clinical text summarization [ 1 ] is a crucial task in modern healthcare, as it enables efficient extraction of key medical information from lengthy and complex documents such as electronic health records (EHRs), discharge summaries, and radiology reports. The vast amount of unstructured textual data generated in clinical settings poses a significant challenge for healthcare professionals, who must rapidly interpret patient histories, diagnoses, and treatment plans to make informed decisions [ 2 ] . Effective summarization of medical texts can enhance clinical workflow efficiency and improve patient outcomes by ensuring that essential information is readily accessible [ 3 ] . Additionally, it plays a vital role in medical research, enabling quicker literature reviews and knowledge synthesis [ 4 ] . However, traditional summarization techniques often struggle with the specialized language, domain-specific jargon, and contextual nuances of medical texts, highlighting the need for more advanced, AI-driven approaches [ 5 ] . Fig. 1 : A sample hospital discharge report (top) where the patient initially could not understand the content. However, by using an LLM, the patient is able to understand the summary (bottom) of the discharge report. Note that the LLM-generated summary may contain hallucinations, and key information may be missing. For instance, the yellow and red highlighted portions are unsupported and incorrect/contradicted fact hallucinations, respectively. Large Language Models (LLMs) have emerged as powerful tools in artificial intelligence and machine learning, demonstrating remarkable capabilities in natural language understanding, generation, and contextual reasoning [ 6 ] . These models, trained on vast and diverse datasets, can comprehend and generate human-like text, making them particularly well-suited for tasks such as summarization, translation, and question-answering [ 7 ] . In the medical domain, LLMs offer significant potential for summarizing clinical texts [ 8 ] by capturing key information while preserving critical details necessary for decision-making. Unlike traditional rule-based or statistical summarization methods, LLMs can adapt to complex linguistic structures, recognize medical terminologies, and generate concise yet informative summaries tailored to specific clinical needs. By leveraging contextual embeddings and domain-specific fine-tuning, LLMs can improve the accuracy and relevance of medical text summarization, ultimately supporting healthcare professionals in reducing information overload and enhancing patient care [ 6 ] . Despite their potential, LLMs face notable challenges in summarizing medical texts, which can impact the reliability and accuracy of generated summaries. These challenges include: Key Information Extraction: Extracting clinically relevant events from medical texts is a critical requirement for summarization. However, LLMs may struggle to prioritize essential details, such as symptoms, treatments, and medication changes, especially when faced with lengthy or complex narratives. Without proper mechanisms for identifying and preserving crucial information, summaries may omit significant medical events, leading to incomplete or misleading interpretations. Hallucinations: LLMs are known to generate plausible yet incorrect or non-existent information, a phenomenon referred to as hallucination [ 9 , 10 ] . In the medical domain, hallucinated details in summary, such as incorrect dosages, fabricated conditions, or misrepresented patient histories, thus can have serious consequences for clinical decision-making. This challenge underscores the need for robust validation mechanisms, domain-specific fine-tuning, and external fact-checking to ensure the reliability of medical text summaries [ 11 ] . In this paper, we aim to highlight and analyze the challenges of key event extraction and hallucinations in LLM-driven medical text summarization. To achieve this, we conduct a comprehensive evaluation using discharge summaries from the MIMIC-IV dataset [ 12 ] , a widely used repository of de-identified clinical records. We explore various open-source large language models (LLMs) to assess their effectiveness in extracting essential medical events while minimizing hallucinations. The open-source LLMs used in this study include LLaMA [ 13 ] , Mistral [ 14 ] , Gemma [ 15 ] , Phi [ 16 ] , Falcon [ 17 ] , LLaVA [ 18 ] , DeepSeek [ 19 ] , and Qwen [ 20 ] . Our analysis focuses on understanding the strengths and limitations of these models in handling domain-specific language and preserving critical clinical information. Addressing these issues is essential to making LLM-driven summarization a practical and trustworthy tool for healthcare applications. II Prior Works and Our Contributions Medical text summarization has been an active area of research, with numerous studies exploring techniques to extract essential clinical information while maintaining accuracy and reliability. Traditional NLP approaches have relied on rule-based and machine-learning methods, while recent advancements in deep learning and large language models have significantly improved the capabilities of automatic summarization [ 21 ] . However, despite these advancements, two key challenges remain particularly difficult: (1) accurately extracting key medical events and (2) addressing hallucinations in LLM-generated summaries. Addressing these issues is crucial for ensuring that AI-driven summarization tools can be safely and effectively deployed in healthcare settings. II-A Previous Works Understanding the challenges of medical text summarization requires examining prior research on key event extraction and hallucination detection in LLM-generated summaries. In the following sections, we discuss existing works addressing these issues and their relevance to our study. II-A 1 Key Event/Information Extraction Extracting essential clinical information from lengthy medical documents has been a long-standing challenge in natural language processing. Traditional approaches relied on rule-based methods and statistical models, such as conditional random fields and hidden Markov models, to identify key events. More recently, deep learning-based techniques, including recurrent neural networks, convolutional neural networks, and transformer-based architectures [ 22 ] , have been employed to improve accuracy. Pre-trained biomedical models such as BioBERT [ 23 ] and ClinicalBERT [ 24 ] have demonstrated effectiveness in extracting medical entities from structured and unstructured clinical narratives. However, these methods often require extensive domain-specific training data and struggle with generalization across different medical contexts. With the advent of LLMs, researchers have explored their potential for key event extraction in clinical text summarization. Models such as GPT-4 [ 25 ] and Med-PaLM [ 26 ] have shown promise in capturing contextual dependencies, but they may still fail to prioritize clinically relevant insights accurately. Furthermore, without explicit fine-tuning on labeled clinical datasets, LLMs risk omitting critical patient information, making their direct application to medical summarization challenging. II-A 2 Detection

Read the complete text