Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

Quoc-Huy Trinh, Xi Ding, Yang Liu, Zhenyue Qin, Xingjian Li, Gorkem Durak, Halil Ertugrul Aktas, Elif Keles, Ulas Bagci, Min Xu · 2026

Read Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space by Quoc-Huy Trinh, Xi Ding, Yang Liu, Zhenyue Qin, Xingjian Li, Gorkem Durak, Halil Ertugrul Aktas, Elif Keles, Ulas Bagci, Min Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13800.txt.

Introduction

Trinh and co-authors ask how well medical vision-language models reason about space in 3D scans. Such skill matters for reading images, yet few datasets test it with rich spatial labels. Their pipeline uses software tools and several cooperating agents to build spatial questions, checked by expert radiologists. The result, SpatialMed, holds 31,253 question-answer pairs across organs and tumor types. Tests of 24 top models show that current systems still reason poorly about 3D space.

Catalog record

CREATORQuoc-Huy Trinh, Xi Ding, Yang Liu, Zhenyue Qin, Xingjian Li, Gorkem Durak, Halil Ertugrul Aktas, Elif Keles, Ulas Bagci, Min Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43346
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space Quoc-Huy Trinh, Xi Ding, Yang Liu, Zhenyue Qin, Xingjian Li, Gorkem Durak, Halil Ertugrul Aktas, Elif Keles, Ulas Bagci, Min Xu Abstract Visual spatial intelligence is critical for medical image interpretation, yet remains largely unexplored in Multimodal Large Language Models (MLLMs) for 3D imaging. This gap persists due to a systemic lack of datasets featuring structured 3D spatial annotations beyond basic labels. In this study, we introduce an agentic pipeline that autonomously synthesizes spatial visual question-answering (VQA) data by orchestrating computational tools such as volume and distance calculators with multi-agent collaboration and expert radiologist validation. We present SpatialMed, the first comprehensive benchmark for evaluating 3D spatial intelligence in medical MLLMs, comprising nearly 10K question-answer pairs across multiple organs and tumor types. Our evaluations on 14 state-of-the-art MLLMs and extensive analyses reveal that current models lack robust spatial reasoning capabilities for medical imaging. 1 Introduction Visual-spatial intelligence—the ability to perceive, reason about, and manipulate spatial relationships—is a cornerstone of clinical radiology. When interpreting computed tomography (CT) scans, for example, radiologists routinely assess tumor dimensions to stage malignancies, estimate organ volumes to evaluate function, and quantify inter-structure distances to plan surgical approaches. These quantitative spatial judgments directly inform treatment decisions: a 3cm margin between a tumor and a major vessel may determine resectability, while a 20% increase in organ volume could indicate disease progression. Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in visual understanding, achieving strong performance on tasks ranging from image captioning to visual question answering in natural scenes ( 34 ; 46 ; 25 ; 14 ) . Recent work has extended MLLMs to 2D spatial reasoning, enabling models to describe object positions, compare sizes, and reason about spatial relationships in photographs. However, current research remains restricted to diagnostic classification ( 36 ; 12 ; 11 ; 30 ; 37 ; 43 ) , anatomical segmentation ( 52 ; 15 ; 3 ; 2 ) , and report generation ( 19 ; 18 ; 21 ; 20 ; 7 ) , leaving quantitative visual-spatial reasoning under-explored. To our knowledge, no existing work systematically evaluates an MLLM’s ability to compute inter-structure distances or estimate volumetric magnitude, both of which are foundational for surgical robotics and AI-assisted diagnostics. Dataset CT Scans QA Pairs Spatial QA Quantitative 3D Relation Grounded QA Benchmark Task RadQA ( 44 ) 0 6.1K × × × × × Radiology report QA INSPECT ( 26 ) 23,248 – × × × × × Diagnosis M3D VQA ( 3 ) 96,170 509K × × × × × Diagnosis CT-CHAT ( 19 ) 25,692 2.7M × × × × × Diagnosis CT-RATE ( 22 ) 25,692 – × × × × ✓ Abnormality Detection, Report Generation BIMCV-R ( 13 ) 8,069 – × × × × ✓ Text–Image Retrieval 3D-RAD ( 16 ) 16,188 136K ✓ × × × × Diagnosis ReXGroundingCT ( 2 ) 3,142 – × × ✓ ✓ × Referring Segmentation AbdomenAtlas 3.0 ( 7 ) 9,262 – × ✓ ✓ × ✓ Report Generation, Tumor Segmentation RadGenome-Chest CT ( 54 ) 25,692 1.2M ✓ ✓ × ✓ ✓ Report Generation, Segmentation QA RadImageNet-VQA (CT subset) ( 8 ) 750K 6.75M × × × × ✓ Diagnosis SpatialMed (Ours) 2375 9.8K ✓ ✓ ✓ ✓ ✓ Spatial QA, Volume Estimation Table 1: Comparison of existing medical imaging datasets with respect to their support for spatial question answering and related spatial reasoning capabilities. SpatialMed is explicitly designed as a benchmark for grounded, quantitative 3D spatial reasoning. The Data Bottleneck. The primary obstacle to progress is the scarcity of appropriate training and evaluation data. Existing volumetric radiology datasets provide classification labels, narrative reports, or segmentation masks, but they systematically lack the structured spatial annotations necessary for quantitative reasoning: (1) Precise 3D spatial relationships – Relative positions, orientations, and topological adjacencies between anatomical structures and lesions; (2) Inter-structure distances – Euclidean or geodesic measurements between structure centroids, surfaces, or boundaries; (3) Quantitative lesion measurements – Volumes, axis dimensions, and cross-sectional areas. While such information is essential for clinical interpretation, acquiring high-quality spatial annotations is prohibitively expensive. It requires expert radiologists to perform case-by-case 3D localization and measurement with consistent anatomical definitions across heterogeneous patient populations. Alternative approaches face their own limitations. Mining spatial information from radiology reports proves unreliable: reports are predominantly narrative, weakly structured, and lack the quantitative relationships and coordinate references needed for spatial understanding. They describe findings qualitatively (”mass abutting the portal vein”) rather than providing the geometric precision (”center-to-center distance: 2.3cm”) required for systematic evaluation. Our approach. To address this gap, we introduce an agentic pipeline that automatically generates clinically meaningful spatial visual question-answering (VQA) data for CT scans. We focus on CT because it is the most widely utilized 3D diagnostic modality in clinical practice and provides a reliable foundation for evaluating objective spatial reasoning. The pipeline leverages spatial computational tools, including volume calculators, distance calculators, and bounding box extractors, to derive precise spatial information about anatomical structures and tumors from CT scans. Multiple agents with retrieval-augmented generation (RAG) are employed to produce diverse spatial VQA pairs. To ensure clinical validity, three board-certified radiologists review the generated data and provide feedback for quality refinement. Building upon this pipeline, we present SpatialMed , a comprehensive benchmark comprising around 9.8k question-answer pairs across 2,375 CT scans. The benchmark spans 117 anatomical structures and 8 tumor types across multiple organs, including the kidney, pancreas, liver, lung, prostate, and brain. Questions are categorized into two types: multiple-choice questions and volume estimation. Our extensive evaluation of state-of-the-art MLLMs on SpatialMed reveals substantial limitations in their spatial understanding of CT data. These findings underscore the critical need for targeted advances in endowing MLLMs with robust spatial reasoning capabilities for medical imaging. In summary, our work makes three primary contributions: (1) We design an agentic pipeline, which leverages three spatial computational tools, multi-agent collaboration and radiologist feedback to automatically generate high-quality spatial VQA data for CT scans; (2) We introduce SpatialMed, the first CT-based benchmark for evaluating MLLMs’ spatial intelligence in medical imaging, comprising 9782 question-answer pairs across 2375 CT scans covering 117 anatomical structures and 8 tumor types; (3) We evaluate 14 MLLMs on SpatialMed, with 12 processing multi-view 2D CT images and 2 processing 3D CT scans, revealing substantial performance gaps that demonstrate existing models lack robust spatial reasoning capabilities for medical imaging. Figure 1 : Task demonstrations in the SpatialMed, covering six spatial reasoning tasks, with corresponding 3D CT visualizations. 2 Related Works Visual-Spatial Intelligence. Spatial reasoning is a fundamental capability of vision–language models for understanding and reasoning about relationships within physical space. To study this problem, several visual question answering (VQA) benchmarks ( 53 ; 27 ; 39 ; 50 ) have been proposed, which demonstrate strong performance in evaluating spatial reasoning over natural images. However, these benchmarks are limited to the 2D image domain and do not support volumetric data or the medical domain. Recently, PRS-Med ( 45 ) was introduced for the medical context to address positional reasoning challenges in medical images, enabling multimodal medical language models to reason about anatomical positions in space. While this represents an important step, spatial reasoning

Read the complete text