RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering

Léo Butsanets 1, Charles Corbière 1, Julien Khlaut 1,2,3, Pierre Manceron 1, Corentin Dancette 1 · 2025

Read RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering by Léo Butsanets 1, Charles Corbière 1, Julien Khlaut 1,2,3, Pierre Manceron 1, Corentin Dancette 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17396.txt.

Introduction

Léo Butsanets and four co-authors release RadImageNet-VQA, a vast exam set for AI systems that answer questions about medical scans. Most older sets lean on plain X-rays or textbook art, and many let models guess well from the words alone. This set centers on CT and MRI exams, the cross-section scans doctors use for hard cases.

It pairs 750,000 images with 7.5 million expert-checked questions and answers. The questions cover spotting abnormalities, naming anatomy, and identifying diseases across eight body regions and 97 disease categories. Leading vision-language models still fail often at naming exact diseases, above all when answers must be written freely. Stripping the images away drops scores to near chance, proving the words alone give nothing away.

Catalog record

CREATORLéo Butsanets 1, Charles Corbière 1, Julien Khlaut 1,2,3, Pierre Manceron 1, Corentin Dancette 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering Léo Butsanets 1, Charles Corbière 1, Julien Khlaut 1,2,3, Pierre Manceron 1, Corentin Dancette 1 Abstract In this work, we introduce RadImageNet-VQA , a large-scale dataset designed to advance radiologic visual question answering (VQA) on CT and MRI exams. While existing medical VQA datasets are limited in scale, dominated by X-ray imaging or biomedical illustrations, and prone to text-based shortcuts, RadImageNet-VQA is built from expert-curated annotations and provides 750K images paired with 7.5M QA samples. It covers three key tasks—abnormality detection, anatomy recognition, and pathology identification—spanning 8 anatomical regions and 97 pathology categories, and supports open-ended, closed-ended, and multiple-choice questions. Extensive experiments show that state-of-the-art vision-language models still struggle with fine-grained pathology identification, especially in open-ended settings and even after fine-tuning. Text-only analysis further reveals that model accuracies collapse to near-random without image inputs, confirming that RadImageNet-VQA is free from linguistic shortcuts. The full dataset and benchmark are publicly available at https://huggingface.co/datasets/raidium/RadImageNet-VQA . \midlauthor 1 Raidium, Paris, France. 2 Université de Paris Cité, Hôpital Européen Georges Pompidou, AP-HP, Paris, France. 3 Department of Vascular and Oncological Interventional Radiology, INSERM, Paris, France. † † year: 2026 keywords Medical Visual Question Answering, Vision-Language Models, Radiology. 1 Introduction Interpreting radiology exams requires integrating complex visual patterns with clinical reasoning, often with medical context 31 ; 10 ; 14 . Vision-language models (VLMs) 18 ; 4 ; 9 ; 21 have recently achieved strong performance on general multimodal benchmarks. Medical variants, such as MedGemma 25 and Lingshu 29 , now incorporate curated medical data to better align with clinical knowledge. These models are particularly attractive for radiology, where generating reports and supporting clinical reasoning require combining visual and textual information. Yet, evaluating free-text radiology reports remains difficult: commonly used similarity metrics often fail to capture clinical correctness, factual consistency, or semantic alignment with ground-truth findings 13 ; 11 ; 28 . In contrast, radiologic visual question answering (VQA) offers a more structured alternative to probe model reasoning. Its question-answer format mirrors how radiologists interrogate imaging findings, providing an interpretable way to evaluate a model’s image-grounded reasoning. Existing medical VQA datasets 15 ; 20 ; 23 ; 34 present several limitations that hinder proper evaluation of these models. Early efforts rely on limited expert annotation and thus offer small collections with narrow anatomical and pathological coverage 15 ; 20 . Larger recent datasets draw heavily from X-ray images or biomedical figures scraped from publications 23 ; 34 , providing little representation of other common imaging modalities, in particular CT or MRI. Many also contain textual shortcuts that allow models to answer correctly without interpreting the image, as shown in Section 4.2 . Beyond evaluation, progress in medical VLMs also requires access to large, well-curated training data aligned with diagnostic tasks. This emphasizes the need for large-scale VQA resources that spans multiple radiologic tasks while minimizing possible shortcut cues to offer a more rigorous foundation for assessing and advancing medical VLMs. Figure 1: Overview of the RadImageNet-VQA dataset , which provides radiology-focused supervision across three VQA tasks (anatomy recognition, fine-grained pathology identification, and abnormality detection) using diverse open-ended, closed-ended, and multiple-choice formats. It also includes radiologic captioning pairs for image–text alignment. In this paper, we introduce RadImageNet-VQA , a large-scale dataset designed for training and benchmarking radiologic VQA on CT and MRI exams (see Figure 1 ). Built from the CT/MRI subset of RadImageNet 22 and its expert-curated anatomical and pathological annotations, RadImageNet-VQA provides 750K images with 7.5M generated samples, including 750K medical captions for visual-text alignment and 6.75M question-answer pairs that span three radiology tasks: fine-grained pathology identification, anatomy recognition, and abnormality detection. The dataset includes open-ended, closed-ended, and multiple-choice questions across 8 anatomical regions and 97 pathologies, generated with prompt-based templates and constructed to probe visual-grounded understanding while minimizing text-only shortcut answering. For evaluation, we construct a stratified benchmark of 1,000 images with 9,000 question-answer pairs covering all tasks and question types. Through extensive zero-shot evaluation and fine-tuning experiments with RadImageNet-VQA, we show that while anatomy recognition is nearly solved for current VLMs, pathology identification remains a major bottleneck, especially for open-ended responses. Text-only analyses further confirm that RadImageNet-VQA greatly reduces shortcut cues present in existing datasets. Fine-tuning yields substantial gains across models, yet we observe that initializing models with medically pretrained vision encoders such as MedSigLIP 25 provides no advantage over general-purpose encoders. Our main contributions are as follows: \bullet We release RadImageNet-VQA, a radiologic dataset that includes both a curated benchmark and a large training corpus. It spans 8 anatomical regions, 97 pathologies, and three task families (abnormality, anatomy, pathology) with multiple question formats, providing 750K images paired with 7.5M generated samples. \bullet We conduct an extensive evaluation of state-of-the-art VLMs showing that existing models perform well on anatomy and basic abnormality detection but still struggle with fine-grained pathology identification. Text-only ablations confirm that RadImageNet-VQA minimizes shortcut cues. \bullet Fine-tuning on RadImageNet-VQA produces substantial gains across all model families, while medical pretrained vision encoders do not improve downstream performance. 2 Related Work 2.1 Vision-language models in radiology Recent VLMs, such as LLaVA 21 , QwenVL 4 , InternVL 9 , typically combine a vision encoder, often initialized from CLIP 24 or SigLIP 33 backbones, with a large language model (LLM) via a lightweight adapter module. In the medical domain, progress has largely been driven by data-centric adaptation, where general-purpose VLMs are aligned to medical images through image–text pretraining and multimodal instruction tuning 17 ; 25 ; 29 . While several 3D radiology VLMs have recently been proposed 6 ; 26 ; 27 ; 2 , their development is constrained by the scarcity of large volumetric datasets, making broad and diverse 2D datasets still essential for training and evaluation. Beyond training, reliable evaluation also remains challenging. Radiology report generation 5 is appealing but difficult to assess automatically, as standard metrics correlate poorly with clinical correctness. On the other hand, visual-question answering (VQA) provides a more controlled and interpretable proxy framework for probing clinical reasoning grounded in images. This has motivated increasing interest in radiologic VQA benchmarks, particularly for CT and MRI, which remain underrepresented in existing resources. Dataset Modality Question Types # Images # QAs Train # QAs Test VQA-RAD 15 X-ray, CT open, closed 0.3K 2.8K 0.45K SLAKE 20 X-ray, CT, MRI open, closed 0.6K 4.9K 1.1K MMMU-Med 32 diverse multiple-choice 1.9K – 1.8K ReXVQA 23 X-ray multiple-choice 160K 573K 40.8K M3D-VQA 3 3D CT multiple-choice 96K 662K 2K 3D-RAD 12 3D CT open, closed 16K 136K 34K RadImageNet-VQA (Ours) CT, MRI open, closed, multiple-choice 750K 7.5M 9K Table 1: Comparison of radiologic VQA datasets compared by imaging modality, supported question types, and number of images, and the number of question-answer pairs for training and evaluation. RadImageNet-VQA provides the largest CT/MRI coverage and the most extensive QA corpus. 2.2 Medical VQA datasets Early radiologic VQA, such as VQA-RAD 15 and SLAKE 20 introduced structured QA annotations but are limited by scale and by narrow anatomical and pathological coverage. More recent efforts, such as the medical subset of MMMU 32 extend medical breadth, yet their radiology content is sparse and relies largely on images from textbooks or publications rather than CT

Read the complete text