Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy

Sushant Gautam, Michael A. Riegler, Pål Halvorsen · 2025

Read Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy by Sushant Gautam, Michael A. Riegler, Pål Halvorsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09958.txt.

Introduction

Sushant Gautam, Michael Riegler, and Pål Halvorsen build a harder test for AI systems that answer questions about medical images, in a 2025 paper. Their arena is gastrointestinal endoscopy, and the task is visual question answering over real scope images. Existing datasets are too simple and too clean to predict clinical usefulness, so the team expanded a prior dataset with nearly one hundred and sixty thousand new question-answer pairs.

Language models generated the questions under a systematic scheme, tiered by the depth of reasoning required. The team also dirtied the images with augmentations mimicking real imaging artifacts, so models must be robust, not just accurate. Evaluation runs on two tracks: standard performance, and performance under those perturbations. The dataset follows FAIR principles and is fully public, code included. The authors aim it at developers of clinical decision support who need benchmarks that resemble the mess of real practice.

Catalog record

CREATORSushant Gautam, Michael A. Riegler, Pål Halvorsen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy Sushant Gautam, Michael A. Riegler, Pål Halvorsen Abstract Medical Visual Question Answering (MedVQA) is a promising field for developing clinical decision support systems, yet progress is often limited by the available datasets, which can lack clinical complexity and visual diversity. To address these gaps, we introduce Kvasir-VQA-x1, a new, large-scale dataset for gastrointestinal (GI) endoscopy. Our work significantly expands upon the original Kvasir-VQA by incorporating 159,549 new question-answer pairs that are designed to test deeper clinical reasoning. We developed a systematic method using large language models to generate these questions, which are stratified by complexity to better assess a model’s inference capabilities. To ensure our dataset prepares models for real-world clinical scenarios, we have also introduced a variety of visual augmentations that mimic common imaging artifacts. The dataset is structured to support two main evaluation tracks: one for standard VQA performance and another to test model robustness against these visual perturbations. By providing a more challenging and clinically relevant benchmark, Kvasir-VQA-x1 aims to accelerate the development of more reliable and effective multimodal AI systems for use in clinical settings. The dataset is fully accessible and adheres to FAIR data principles, making it a valuable resource for the wider research community. Code and data: github.com/Simula/Kvasir-VQA-x1 and huggingface.co/datasets/SimulaMet/Kvasir-VQA-x1 Keywords: medical VQA, gastrointestinal endoscopy, multimodal AI, dataset benchmark, visual perturbations 1 Background & Summary Medical AI is witnessing a transformative shift—from pattern recognition to context-aware reasoning—driven by advances in multimodal learning. Within this landscape, Medical Visual Question Answering (MedVQA) has emerged as a compelling benchmark for evaluating real-world capabilities of vision-language models. 1.1 The Promise and Challenge of Medical Visual Question Answering Medical Visual Question Answering is an emergent interdisciplinary field at the intersection of artificial intelligence, computer vision, and medicine [ 30 , 24 , 31 , 16 ] . It aims to develop systems that interpret medical images and respond to clinically pertinent questions posed in natural language [ 16 ] . This capability holds transformative potential for clinical decision support, offering avenues to enhance diagnostic accuracy, reduce clinician workload, improve patient comprehension, and enable more equitable access to medical expertise through automated assistance in telemedicine and low-resource settings [ 5 , 45 , 30 , 16 ] . Unlike general-domain visual question answering (VQA), MedVQA poses unique challenges due to the complexity of medical images and the depth of domain-specific knowledge required to answer clinical questions [ 28 ] . Visual cues in medical imaging are often subtle and entangled with artifacts, requiring expert-level reasoning [ 43 ] . Moreover, clinical questions may demand multi-step inference or integration of prior medical knowledge, i.e., tasks that go beyond simple pattern recognition or factual recall [ 9 , 14 , 21 ] . As such, MedVQA is increasingly viewed as a frontier task for evaluating both vision-language reasoning and real-world applicability of multimodal AI systems in high-stakes environments [ 16 ] . 1.2 The Paradigm Shift Towards Generative Models Early MedVQA systems were predominantly discriminative, selecting answers from predefined candidate sets [ 51 , 50 ] . While effective for constrained tasks, such systems fall short when confronted with the open-ended, nuanced questions encountered in clinical practice [ 15 , 30 ] . This limitation has driven a paradigm shift toward generative models, enabled by recent breakthroughs in Large Language Models (LLMs) and Vision–Language Models (VLMs) [ 10 , 4 , 2 , 32 , 36 ] . State-of-the-art systems such as Med-Flamingo, LLaVA-Med, MedGemma, and Qwen2.5-VL exemplify this trend, combining advanced image encoders with powerful, instruction-tuned decoders capable of producing rich, context-sensitive responses [ 35 , 26 , 4 , 34 , 13 ] . These models signal a movement towards AI systems that can engage in more natural and human-like diagnostic reasoning [ 35 , 26 ] . Yet, the promise of generative MedVQA systems is hindered by a lack of suitably complex, domain-aligned datasets for training and evaluation [ 53 ] . Most existing benchmarks, including VQA-RAD [ 24 ] , SLAKE [ 24 ] , and PMC-VQA [ 53 ] , suffer from small scale, limited question diversity, or over-representation of specific modalities (e.g., radiology). Many question-answer (QA) pairs are simple, fact-based, or automatically generated, which risk introducing noise and fail to promote advanced clinical reasoning [ 53 ] . Moreover, traditional evaluation metrics (e.g., BLEU and ROUGE) often fall short in capturing the correctness or clinical utility of generative outputs, highlighting a need for new benchmarks and assessment methods rooted in real-world use cases and clinician feedback [ 1 , 37 , 42 , 52 ] . 1.3 GI Endoscopy: A Unique Frontier for VQA Table 1 : Comparison of Existing Medical VQA Datasets with Relevance to Gastrointestinal (GI) Applications Dataset Year Primary Focus / Modality Size (Images / QA Pairs) Key GI-Relevant Question Types Notable Limitations for Advanced GI VQA Kvasir-VQA [ 12 ] 2024 GI Endoscopy 6,500 / \sim 58,849 Yes/No, choice, color, location, count Potential for limited QA complexity for deep reasoning; scope for more conditions MedVQA-GI [ 18 ] 2023 GI Endoscopy (Colonoscopy) \sim 4,000 / Multiple per image Polyp count/location, instrument ID, color, artifacts Incomplete expert validation; some subjective answers; limited question diversity planned for expansion PMC-VQA [ 53 ] 2023 General Medical (80% Radiology) 149k / 227k Modality, organ, abnormality (general) Limited GI-specific focus; QA generation process risks SLAKE [ 31 ] 2021 Radiology (CT, MRI, X-ray) 642 / \sim 14k Organ, position, abnormality (general) Small scale; not GI-specific VQA-RAD [ 24 ] 2018 Radiology 315 / \sim 3,500 Abnormality, modality, organ (general) Very small scale; simple QA; not GI-specific MedFrameQA [ 49 ] 2024 Multi-image Medical Reasoning 9,237 frames / 2,851 Multi-image comparative reasoning Not specifically GI-focused; tests general MLLM reasoning Gastrointestinal (GI) endoscopy is a critical diagnostic and interventional tool in medicine, generating large volumes of high-resolution images that are rich in clinical content but visually complex [ 6 ] . These images frequently contain artifacts such as specular reflections, motion blur, and variable lighting, making them a challenging modality for automated interpretation [ 3 , 23 ] . Despite this, the GI domain has received relatively limited attention in VQA research compared to radiology or pathology [ 12 ] . Table 1 provides a brief comparison of existing GI-focused VQA datasets. Notable GI-specific resources such as HyperKvasir [ 6 ] , Kvasir-Instrument [ 23 ] , and Kvasir-VQA [ 12 ] have laid important groundwork, but they often feature QA pairs centered on simple tasks, such as identifying the presence of a polyp or recognizing a tool, and thus do not fully capture the reasoning depth required for advanced clinical understanding. Similarly, while MedVQA-GI (2023) introduces diverse QA types for colonoscopy, limitations in expert validation and linguistic diversity constrain its utility for training robust generative models [ 19 ] . 1.4 Towards Robust, Reasoning-Centric Evaluation To drive progress in this domain, we introduce Kvasir-VQA-x1 , a significantly expanded and meticulously curated dataset designed to benchmark reasoning-intensive visual question answering in GI endoscopy. Building upon the rich visual foundations of HyperKvasir [ 6 ] , Kvasir-Instrument [ 23 ] and Kvasir-VQA [ 12 ] , Kvasir-VQA-x1

Read the complete text