Beyond Diagnosis: Evaluating Multimodal LLMs for Pathology Localization in Chest Radiographs These authors contributed equally Work done while intern at Dana-Farber Cancer Institute

Advait Gosai, Arun Kavishwar, Stephanie L. McNamara, Soujanya Samineni, Renato Umeton, Alexander Chowdhury, William Lotter · 2025

Read Beyond Diagnosis: Evaluating Multimodal LLMs for Pathology Localization in Chest Radiographs These authors contributed equally Work done while intern at Dana-Farber Cancer Institute by Advait Gosai, Arun Kavishwar, Stephanie L. McNamara, Soujanya Samineni, Renato Umeton, Alexander Chowdhury, William Lotter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18015.txt.

Introduction

Advait Gosai and 6 co-authors test if chat models can point to where a chest X-ray goes wrong. Their 2025 study goes past naming a disease, to placing it on the image. That skill matters for care and for teaching, and it shows how well a model grasps body space.

Two broad chat models and one medical model were asked for map points on scans, with a grid laid over each scan. The best model placed just under half of the cases right, below a task-built net and far below a radiologist. Its misses were most often in a plausible body zone, just not the exact spot.

Catalog record

CREATORAdvait Gosai, Arun Kavishwar, Stephanie L. McNamara, Soujanya Samineni, Renato Umeton, Alexander Chowdhury, William Lotter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18015

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Diagnosis: Evaluating Multimodal LLMs for Pathology Localization in Chest Radiographs These authors contributed equally Work done while intern at Dana-Farber Cancer Institute Advait Gosai, Arun Kavishwar, Stephanie L. McNamara, Soujanya Samineni, Renato Umeton, Alexander Chowdhury, William Lotter Abstract Recent work has shown promising performance of frontier large language models (LLMs) and their multimodal counterparts in medical quizzes and diagnostic tasks, highlighting their potential for broad clinical utility given their accessible, general-purpose nature. However, beyond diagnosis, a fundamental aspect of medical image interpretation is the ability to localize pathological findings. Evaluating localization not only has clinical and educational relevance but also provides insight into a model’s spatial understanding of anatomy and disease. Here, we systematically assess two general-purpose MLLMs (GPT-4 and GPT-5) and a domain-specific model (MedGemma) in their ability to localize pathologies on chest radiographs, using a prompting pipeline that overlays a spatial grid and elicits coordinate-based predictions. Averaged across nine pathologies in the CheXlocalize dataset, GPT-5 exhibited a localization accuracy of 49.7%, followed by GPT-4 (39.1%) and MedGemma (17.7%), all lower than a task-specific CNN baseline (59.9%) and a radiologist benchmark (80.1%). Despite modest performance, error analysis revealed that GPT-5’s predictions were largely in anatomically plausible regions, just not always precisely localized. GPT-4 performed well on pathologies with fixed anatomical locations, but struggled with spatially variable findings and exhibited anatomically implausible predictions more frequently. MedGemma demonstrated the lowest performance on all pathologies, showing limited capacity to generalize to this novel task. Our findings highlight both the promise and limitations of current MLLMs in medical imaging and underscore the importance of integrating them with task-specific tools for reliable use. 1 Introduction As AI advances from task-specific to generalist models, there is growing interest in evaluating frontier models in medicine, including in medical imaging with the rise of multimodal large language models (MLLMs). Recent studies assessing GPT-4, a pioneering MLLM, on imaging-based quizzes and diagnostic tasks have yielded mixed results ( Hayden et al. 2024 ; Brin et al. 2024 ; Strotzer et al. 2024 ; Zhou et al. 2024 ; Suh et al. 2024 ; Eriksen et al. 2023 ) . For instance, Suh et al. 2024 reported that GPT-4 achieved clinical-level performance on NEJM Image Challenges , a multiple-choice, VQA dataset; however, Jin et al. 2024 found that the model’s rationale for its choices were often flawed. Given the clinical potential of accessible, general-purpose MLLMs, these findings highlight the need to evaluate their performance across diverse tasks and to move beyond diagnostic accuracy toward a deeper understanding of their reasoning. Furthermore, it remains unclear if domain-specific MLLMs are required to address the challenges of general purpose models, or whether these challenges can instead be overcome with newer model versions. A fundamental aspect of medical image interpretation is not only providing a diagnostic impression but also localizing findings associated with a diagnosis. In computer vision, diagnosis is typically framed as a classification task, whereas localization is approached through segmentation or bounding-box generation (i.e., detection) over relevant regions. In medical AI systems, these two tasks are treated as distinct regulatory categories: computer-aided diagnosis (CADx) for classification and computer-aided detection (CADe) for localization ( McNamara et al. 2024 ) . Critically, an AI model may perform well at classification, but struggle with precisely localizing the associated findings, or conversely struggle with distinguishing whether a finding is benign or pathological, but excel at identifying its location. While recent studies have evaluated the diagnostic performance of MLLMs, their ability to localize pathological findings (e.g., CADe) remains largely unexplored. Beyond potential clinical applications, evaluating MLLMs in localization tasks could provide insights into their underlying clinical intuition and medical knowledge. Figure 1: Study Overview. a) The nine pathologies in the CheXlocalize dataset were utilized. b) For each pathology class, a gridded image was produced for each radiograph where the pathology was present. c) Three multimodal LLMs were prompted to identify the grid cell where the stated pathology is most prominent. d) The predicted grid cells were compared to the ground truth cells, defined as cells with \geq 50% overlap with the ground truth mask. In this study, we systematically evaluate the ability of three MLLMs to localize pathologies on chest radiographs: two generalist models (GPT-4 ( OpenAI 2024 ) and GPT-5 ( OpenAI 2025 ) ) and a domain-specific model (MedGemma ( Sellergren et al. 2025 ) ). To do so, we develop a prompting strategy that overlays a grid on the image and asks the model to predict coordinates corresponding to the pathology’s location. We assess performance using the CheXlocalize ( Saporta et al. 2022 ) dataset, benchmarking against radiologists and a task-specific CNN model. To further explore the models’ medical understanding, we analyze the distribution of predicted locations for each pathology and categorize mislocalized cases based on their clinical plausibility. By moving beyond classification tasks, our analysis provides insights into the spatial reasoning abilities of MLLMs in medical imaging, highlighting both strengths and limitations. 2 Related Work Several recent studies have assessed general-purpose MLLMs such as GPT-4 in medical image interpretation ( Jin et al. 2024 ; Hayden et al. 2024 ; Brin et al. 2024 ; Strotzer et al. 2024 ; Zhou et al. 2024 ; Suh et al. 2024 ; Eriksen et al. 2023 ) . On 190 cases from the Diagnosis Please challenge, Suh et al. 2024 found that GPT-4 achieved diagnostic performance comparable to radiologists. GPT-4 also demonstrated strong accuracy on the NEJM Image Challenges, a multiple-choice diagnostic quiz for medical professionals ( Jin et al. 2024 ; Eriksen et al. 2023 ) . However, Jin et al. 2024 reported that while GPT-4 performed well on these tasks, its rationales were often flawed when explicitly prompted to explain its reasoning. On multiple-choice questions from the American College of Radiology Diagnostic Radiology In-Training Examination , GPT-4 achieved high accuracy on text-only questions (81.5%) but struggled with image-based ones (47.8%). Its diagnostic accuracy on real-world radiologic imaging cohorts has also shown considerable variability ( Strotzer et al. 2024 ; Brin et al. 2024 ; Zhou et al. 2024 ) . 3 Methods While prior research has focused primarily on the diagnostic capabilities of MLLMs, our study examines spatial localization accuracy for pathologies in medical images that the models are explicitly prompted to recognize. We utilize chest radiographs from CheXlocalize ( Saporta et al. 2022 ) , a subset of the CheXpert ( Irvin et al. 2019 ) dataset annotated with pixel-level masks for each of its classes. We focus on this task because it serves as both a common benchmark for medical AI and a clinically important application. Figure 1 contains an overview of the approach. 3.1 CheXlocalize Dataset The CheXlocalize dataset includes chest radiograph images collected from Stanford Hospital in both inpatient and outpatient settings ( Irvin et al. 2019 ) . The dataset contains expert radiologist annotations for the localization of 10 distinct findings ( Saporta et al. 2022 ) . These ground-truth annotations come in the form of pixel-wise segmentation maps, enabling precise benchmarking. The validation and test splits of this dataset consist of 234 radiographs from 200 patients and 668 radiographs from 500 patients, respectively. Here, we use the validation split for initial experimentation and report final

Read the complete text