3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models
Sraavya Sambara 1∗, Sung Eun Kim 1,2∗, Xiaoman Zhang 1, Luyang Luo 1 Shreya Johri 1, Mohammed Baharoon 1, Du Hyun Ro 2, Pranav Rajpurkar 1 · 2025
Read 3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models by Sraavya Sambara 1∗, Sung Eun Kim 1,2∗, Xiaoman Zhang 1, Luyang Luo 1 Shreya Johri 1, Mohammed Baharoon 1, Du Hyun Ro 2, Pranav Rajpurkar 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20967.txt.
Introduction
Sraavya Sambara leads seven co-authors on 3DReasonKnee, the first data set for grounded reasoning over three dimensional medical images, built from knee MRI scans. Clinicians reason step by step about named anatomy, yet vision language models struggle to locate regions in volumes and explain findings in that workflow.
Nearly half a million high quality quintuples are derived from almost eight thousand knee volumes, each linking the scan, a question, region grounding, reasoning steps, and an answer. Training on such data aims at outputs that match diagnostic practice and can be checked by clinicians. Trustworthy collaboration between doctors and AI is the stated goal.
Catalog record
| CREATOR | Sraavya Sambara 1∗, Sung Eun Kim 1,2∗, Xiaoman Zhang 1, Luyang Luo 1 Shreya Johri 1, Mohammed Baharoon 1, Du Hyun Ro 2, Pranav Rajpurkar 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models Sraavya Sambara 1∗, Sung Eun Kim 1,2∗, Xiaoman Zhang 1, Luyang Luo 1 Shreya Johri 1, Mohammed Baharoon 1, Du Hyun Ro 2, Pranav Rajpurkar 1 Abstract Current Vision-Language Models (VLMs) struggle to ground anatomical regions in 3D medical images and reason about them in a step-by-step manner—a key requirement of real-world diagnostic assessment. This ability is essential for aligning model outputs with the diagnostic workflows clinicians use in practice, enabling trustworthy clinician-AI collaboration. Existing 3D datasets provide localization labels, but none support this “grounded reasoning” ability. To address this gap, we introduce 3DReasonKnee , the first 3D grounded reasoning dataset for medical images, which provides 494k high-quality quintuples derived from 7,970 3D knee MRI volumes. Each quintuple includes: (1) the 3D MRI volume, (2) a diagnostic question targeting a specific anatomical region (3) a 3D bounding box localizing the relevant anatomical structures, (4) clinician-generated diagnostic reasoning steps that explicitly detail the 3D reasoning process, and (5) structured severity assessments for the relevant anatomical region. The meticulous creation and validation of 3DReasonKnee, involving over 450 hours of expert clinician time for manually segmenting MRIs and generating reasoning chains, ensures its superior quality and clinical relevance. We establish ReasonKnee-Bench to evaluate localization and diagnostic accuracy, providing novel insight into VLM ability to perform grounding and severity assessment across diverse anatomical regions and diagnostic inquiries. We benchmark five state-of-the-art VLMs, providing baseline performance for ReasonKnee-Bench. By providing this unique resource of expert-annotated 3D reasoning pathways, 3DReasonKnee serves as a repository of orthopedic surgeons’ diagnostic expertise and offers a vital testbed for advancing multimodal medical AI systems towards 3D, clinically aligned, localized decision-making capabilities. The dataset can be found in HuggingFace: rajpurkarlab/3DReasonKnee . \copyrightinfo *These authors contributed equally to this work. © 2025 The Authors. Open Access chapter published by World Scientific Publishing Company and distributed under the terms of the Creative Commons Attribution Non-Commercial (CC BY-NC) 4.0 License. 1 Introduction Figure 1: Region-First Reasoning Workflow in Clinician Knee MRI Assessment. This schema illustrates the step-wise diagnostic process employed by clinicians, involving (Step 1) identification of lesion category and localization to specific anatomical subregions, (Step 2) assessment of severity and lesion type within the localized region, and (Step 3) characterization of specific features. Medical VLMs Lack Fine-Grained 3D Grounded Reasoning. Recent progress in Medical Vision-Language models (VLMs) has demonstrated remarkable capabilities in versatile tasks requiring integrated reasoning across modalities, with models achieving state-of-the-art performance on visual question answering, report generation, and other global understanding tasks [ 14 , 37 ] . However, these models still fall short of clinician-level performance when confronted with problems demanding fine-grained diagnostic reasoning and assessment. Clinicians typically employ a region-first workflow (Figure 1 ), pinpointing relevant anatomical subregions, assessing them for abnormalities such as lesions or structural changes, and finally assigning structured severity grades based on established clinical criteria. For instance, the MRI Osteoarthritis Knee Score (MOAKS) framework, as well as other semi-quantitative grading systems, necessitates this level of fine-grained analysis to evaluate specific knee structures and assign severity grades to features like bone marrow lesions and cartilage damage. [ 20 , 8 ] This ”grounded” understanding of pathology within localized regions is crucial for accurate diagnosis, prognosis, and treatment planning in clinical settings. Bridging this gap between clinician reasoning and model intelligence is critical for enabling trustworthy clinical decision support and seamless integration of AI into diagnostic workflows. 3D Grounded Reasoning is Not Well-Supported by Existing Datasets. Advances in Chain-of-thought (CoT) prompting have proven effective in enabling models to break down complex problems into interpretable, step-by-step solution processes [ 27 , 36 ] . In 3D medical image analysis, diagnostic reasoning represents a structured cognitive process that extends far beyond simple pattern recognition. It requires precise spatial understanding across multiple planes to properly evaluate volumetric structures. However, instilling such reasoning capabilities in Medical VLMs for 3D medical data is hampered by a lack of suitable training data. While a growing number of datasets are facilitating the development of medical VLMs, such as MultiMedEval [ 21 ] for diverse evaluation and MedTrinity-25M [ 30 ] for large-scale pretraining, these resources often focus on 2D images with reasoning traces or 3D corpora emphasizing localization without detailed diagnostic rationale (Table 1 ). A significant gap remains in datasets that facilitate grounded reasoning on 3D medical images, particularly those combining volumetric data, integrated voxel-level localization, and expert-annotated reasoning chains. Introducing 3DReasonKnee to Advance 3D Grounded Reasoning. Our work introduces 3DReasonKnee, the first 3D grounded reasoning dataset for medical images, comprising 494k high-quality quintuples spanning 7,970 unique MRIs. Each quintuple includes: (1) a 3D knee MRI volume, (2) a diagnostic question targeting a specific anatomical region, (3) a 3D bounding box localizing the relevant anatomical structures, (4) clinician-generated diagnostic reasoning steps detailing the 3D reasoning process, and (5) structured severity assessments for the relevant anatomical region. We frame 62 distinct clinical instruction tuning questions and provide volumetric bounding boxes for 22 distinct anatomical regions, drawing from the MOAKS framework. The creation of 3DReasonKnee involved over 450 clinician hours dedicated to manually labeling MRI data and generating detailed reasoning traces. In doing so, 3DReasonKnee provides a clinically aligned resource that supports interpretable model development and enables rigorous evaluation of model performance on narrowly defined diagnostic tasks. To leverage this rich data, we establish ReasonKnee-Bench to rigorously evaluate both localization and diagnostic accuracy in VLMs. Our evaluation metrics include diagnostic accuracy assessment and 3D IoU for the model-generated localization bounding boxes, providing new insights into VLM ability to perform grounding and severity assessment across diverse anatomical regions and diagnostic inquiries. We emphasize the significant potential of 3DReasonKnee, with its expert-annotated reasoning chains and validated localization labels serving as a valuable clinical gold standard, to drive advancements in VLM grounded reasoning capabilities. This work lays the foundation for VLMs that better reflect how clinicians localize and reason about findings in practice, an important step toward their reliable and interpretable use in diagnostic care. \tbl Representative datasets and benchmarks for medical VLMs. Dim. denotes dimensionality (2D vs. 3D). Grounding indicates the presence of spatial annotations (bounding boxes, masks, or voxel labels); refers to expert-authored diagnostic reasoning steps or structured interpretation pathways that mirror how clinicians synthesize visual findings into diagnostic conclusions Dataset Dim. Modality Size Grounding Reasoning Path‐VQA 2D pathology images 32 k samples \times \times Chest ImaGenome 2D chest X-ray 242 k samples ✓ \times PadChest‐GR 2D chest X-ray 4,555 studies ✓ \times PubMedVision 2D biomedical figures 1.3 M samples \times \times RadRBench-CXR 2D chest X-ray 59K samples \times ✓ GMAI-Reasoning10K 2D mixed modalities 10K samples \times ✓ MedTrinity-25M 2D mixed modalities 25M samples ✓ ✓ M3D‐RefSeg 3D CT 210 vols ✓ \times MedMD RP3D 3D CT/MR 51 k vols \times \times RadGenome-Chest CT 3D CT 1.3 M samples ✓ \times 3D ReasonKnee 3D knee MRI 494 k samples ✓ ✓ 2 Related Work Existing Datasets and Benchmarks for Medical VLMs. A growing number of datasets and benchmarks are facilitating the development of medical vision-language models. MultiMedEval [ 21 ] offers a consolidated evaluation framework across various tasks and domains, with its VQA component utilizing 2D datasets such as PMC-VQA [ 35 ] , Path-VQA [