Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities
Anindya Bijoy Das, Shahnewaz Karim Sakib, Shibbir Ahmed · 2025
Read Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities by Anindya Bijoy Das, Shahnewaz Karim Sakib, Shibbir Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07031.txt.
Introduction
Anindya Bijoy Das and co-authors study how large language models hallucinate in medical imaging, making up findings with full confidence. Models now read scans and also draft scans from text orders. A fluent error in either direction can steer a clinic wrong.
The study probes both roads: image to text, where reports spring from X-ray, CT, or MRI input, and text to image, where pictures spring from clinical prompts. Judges score outputs by expert-set criteria for factual slips and anatomy errors. Shared failure patterns emerge across scan types and tasks. The team traces causes to model build and training mix. Their map of error shapes gives safety teams concrete checks to run before such tools sit near patients.
Catalog record
| CREATOR | Anindya Bijoy Das, Shahnewaz Karim Sakib, Shibbir Ahmed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities Anindya Bijoy Das, Shahnewaz Karim Sakib, Shibbir Ahmed Abstract Large Language Models (LLMs) are increasingly applied to medical imaging tasks, including image interpretation and synthetic image generation. However, these models often produce hallucinations, which are confident but incorrect outputs that can mislead clinical decisions. This study examines hallucinations in two directions: image to text, where LLMs generate reports from X-ray, CT, or MRI scans, and text to image, where models create medical images from clinical prompts. We analyze errors such as factual inconsistencies and anatomical inaccuracies, evaluating outputs using expert informed criteria across imaging modalities. Our findings reveal common patterns of hallucination in both interpretive and generative tasks, with implications for clinical reliability. We also discuss factors contributing to these failures, including model architecture and training data. By systematically studying both image understanding and generation, this work provides insights into improving the safety and trustworthiness of LLM driven medical imaging systems. 1 Introduction Large Language Models (LLMs) [ 9 ] have shown remarkable capabilities across a range of natural language processing tasks like summarization, question answering, translation, and content generation, enabling advances in education, law, and research. In the medical domain, LLMs hold significant promise [ 3 ] for improving clinical workflows by automating documentation, assisting in decision support, and enabling patient-specific summarization of complex medical information [ 23 ] . When integrated with imaging data, LLMs can aid in generating radiology reports, explaining imaging findings, and synthesizing realistic images for education and training. These applications could enhance diagnostic efficiency, reduce clinician burden [ 5 ] , and improve communication with patients. However, realizing these benefits requires careful evaluation of the accuracy, reliability, and safety of LLM outputs, especially in high-stakes healthcare settings. Consider the case of brain MRI: essential for evaluating conditions like stroke, tumors, edema, and demyelination. A key finding here is midline shift [ 19 ] , indicating displacement of brain structures due to mass effect from hemorrhage, tumor, or swelling. The degree of midline shift is clinically significant, as even a few millimeters can indicate elevated intracranial pressure or impending herniation. If an LLM used for image interpretation fails to detect or misestimates midline shift, it may produce a report that downplays a life-threatening condition, potentially leading to harmful clinical decisions. On the generative side, if an LLM is prompted to produce an MRI scan showing a 6 mm midline shift but generates an inaccurate or subtle image, it may hinder medical education or misguide diagnostic model training. Figure 1 : Example of midline shift measurements on portable MRI (pMRI) for a 81-year-old male having right intracerebral hemorrhage with a midline shift 9.0 mm (left) and a 71-year-old male having right M1 occlusion with a midline shift 5.1 mm (right). Despite their potential, LLMs are prone to generating hallucinations, i.e., outputs that are fluent and confident but factually incorrect or unsupported by input data. In medicine, such errors can misrepresent patient data, introduce inaccuracies, or fabricate plausible-sounding diagnoses [ 8 ] . Hallucinations can arise due to limitations in training data, overgeneralization, or the model’s tendency to complete patterns based on learned linguistic structures rather than grounded evidence [ 11 ] . In image to text tasks, this may lead to erroneous interpretations of medical images, while in text to image generation, the model might synthesize anatomically implausible or clinically misleading content. These errors, if unrecognized, risk spreading misinformation and jeopardizing patient safety [ 29 ] . In this work, we systematically investigate hallucinations in large language model (LLM) applications to medical imaging, focusing on both image-to-text and text-to-image tasks. Our key contributions are summarized below: • We analyze LLM-generated clinical descriptions from medical images across modalities such as chest X-rays, CT scans, and brain MRIs, identifying common hallucination patterns and evaluating report accuracy against ground truth and expert assessments. • We evaluate text-to-image generation to determine whether the synthesized medical images faithfully represent the intended clinical findings, anatomical details, and align with plausible real-world clinical outcomes. • We conduct numerical experiments across image interpretation, image classification, and image generation tasks to systematically identify and analyze hallucinations in LLM-based medical imaging workflows. Together, these contributions can offer a comprehensive foundation for understanding and reducing hallucinations in LLM-driven medical imaging systems. 2 Motivating Scenarios To motivate our study, this section presents two examples exposing critical limitations of current multimodal LLMs in medical imaging. The first one highlights issues in image interpretation, while the second reveals failures in image generation. These cases underscore the need for deeper analysis and robust safeguards in trustworthy medical AI. Example 1 . Consider the case of chest X-rays, a common imaging modality used to assess lung and pleural conditions. One important finding radiologists look for is pleural effusion [ 20 ] , i.e., the abnormal accumulation of fluid in the pleural space surrounding the lungs. Pleural effusion can occur on the left, right, or both sides (bilateral), and its severity is typically categorized as absent, mild, moderate, or large. Accurate detection of this condition is essential for diagnosing underlying causes such as heart failure, infection, or malignancy. If an LLM tasked with interpreting chest X-rays fails to identify the presence or severity of pleural effusion, it may generate a misleading report that could result in delayed or incorrect treatment. Similarly, if an LLM is prompted to create a chest X-ray image showing moderate right-sided pleural effusion but produces an anatomically inconsistent or incorrect image, it can misinform both clinical training and algorithmic development. Figure 2 : Chest X-ray (a) PA view and (b) lateral view with bilateral moderate to large pleural effusions. Table 1 : LLMs’ performance to detect Pleural Effusion LLM Decision Grok 3 right-sided, large. GPT 4o bilateral, moderate to large Gemini 2.5 Flash bilateral, moderate to large Claude Sonnet 4 No Copilot left-sided, large Perplexity left-sided, large Example 2 . In this example, we prompted various multimodal LLMs to generate a chest X-ray that clearly shows toe fractures. This request is intentionally inconsistent, as toe fractures cannot appear in chest X-rays. GPT-4o [ 1 ] nonetheless proceeds to generate an image—erroneously displaying fractured finger bones, likely from a hand, while still labeling it as a chest X-ray. This response not only reflects a fundamental misunderstanding of medical imaging anatomy but also reveals a concerning hallucination that may mislead users. In contrast, Gemini-2.5 Flash [ 25 ] handles the request more responsibly: it produces two distinct images and provides clarification that a chest X-ray cannot capture toe fractures. This discrepancy highlights the variability in model behavior and emphasizes the need for rigorous evaluation of hallucinations in medical image generation. Figure 3 : Model responses to the prompt: “Generate a chest X-ray that clearly shows toe fractures.” (a) GPT-4o hallucinates, overlaying fingers (not even toes) on a chest X-ray. (b) Gemini 2.5 Flash returns separate, anatomically correct chest and foot X-rays. These risks underscore the importance of systematically identifying and mitigating hallucinations in both interpretive and generative applications of LLMs in trustworthy medical imaging [ 2 ] . Such hallucinations can lead to clinically misleading outputs, undermine user trust, and pose potential safety concerns in high-stakes environments. To