Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI

Lama Moukheiber, Caleb M. Yeung, Haotian Xue, Alec Helbling, Zelin Zhao, Yongxin Chen, [0.8em] Georgia Institute of Technology · 2026

Read Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI by Lama Moukheiber, Caleb M. Yeung, Haotian Xue, Alec Helbling, Zelin Zhao, Yongxin Chen, [0.8em] Georgia Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15808.txt.

Introduction

Moukheiber, Yeung, Xue, Helbling, Zhao, and Chen offer SGMRI-VQA, a 41,307-pair test of multi-frame, pinned thought on volume MRI. It is built from eye-doc marks in fastMRI+ brain and knee. Each pair holds a doc-lined trail with frame-tagged box spots. From spotting to placing to counting, sorting, and captioning, asking what, where, and across which frames. Tuning Qwen3-VL-8B with box watch beats no-shot baselines on grounding, showing space watch aids grounded clinic thought.

Catalog record

CREATORLama Moukheiber, Caleb M. Yeung, Haotian Xue, Alec Helbling, Zelin Zhao, Yongxin Chen, [0.8em] Georgia Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36883
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI Lama Moukheiber, Caleb M. Yeung, Haotian Xue, Alec Helbling, Zelin Zhao, Yongxin Chen, [0.8em] Georgia Institute of Technology Abstract Spatial reasoning and visual grounding are core capabilities for vision-language models (VLMs), yet most medical VLMs produce predictions without transparent reasoning or spatial evidence. Existing benchmarks further evaluate VLMs on isolated 2D images, overlooking the volumetric nature of clinical imaging where findings can span multiple frames or appear on only a few slices. We introduce Spatially Grounded MRI Visual Question Answering (SGMRI-VQA) , a 41,307-pair benchmark for multi-frame spatially grounded reasoning on volumetric MRI. Built from expert radiologist annotations in the fastMRI+ dataset across brain and knee studies, each QA pair pairs a clinician-aligned chain-of-thought trace with frame-indexed bounding-box coordinates, and tasks are organized hierarchically across detection, localization, counting/classification, and captioning—requiring models to jointly reason about what is present, where it is, and across which frames it extends. We benchmark 10 VLMs and show that supervised fine-tuning of Qwen3-VL-8B with bounding-box supervision consistently improves grounding performance over strong zero-shot baselines, indicating that targeted spatial supervision is an effective path toward grounded clinical reasoning. Our code, benchmark, and model outputs are available at https://lamawmouk.github.io/SGMRI-VQA . Project Page Hugging Face GitHub 1 Introduction Spatial reasoning and visual grounding are core capabilities for vision-language models (VLMs), requiring not only recognition of visual content but also precise localization of findings within complex scenes. Recent VLMs have demonstrated progress on these tasks for natural images—identifying objects, describing their spatial relationships, and producing bounding box coordinates ( 3 ; 7 ; 27 ; 1 ; 20 ) . In the medical domain, VLMs have achieved notable performance on visual question answering benchmarks, combining visual perception with clinical language understanding to identify abnormalities, classify findings, and generate diagnostic descriptions ( 34 ; 11 ; 31 ; 26 ; 4 ) . Yet most VLMs remain opaque, offering predictions without the transparent, step-by-step reasoning clinicians rely on—and critically, without spatially grounding their findings. This raises the question: can VLMs spatially ground their medical knowledge with chain-of-thought reasoning, localizing findings with precision across complex volumetric scenes? Currently, existing benchmarks evaluate VLMs on only single 2D images ( 16 ; 5 ; 21 ) . However, real-world medical imaging—particularly MRI—is inherently volumetric: a single scan produces dozens to hundreds of sequential slices that together form a continuous 3D representation. Consequently, findings may span multiple frames or appear only on specific slices. MedFrameQA ( 32 ) introduced multi-image medical VQA in YouTube videos on modalities including MRI, X-Ray, CT, and radiograph, but with only 2 to 5 temporally related video frames rather than spatially contiguous volumetric slices commonly found in medical scans such as knee and brain MRI. Therefore, limited benchmarks have addressed cross-slice spatial grounding—localizing what is present, where it is located, and across which frames it extends. To comprehensively evaluate how well VLMs handle multi-frame spatial reasoning across volumetric MRI slices, we introduce Spatially Grounded MRI Visual Question Answering ( SGMRI-VQA ), a large-scale VQA benchmark for MRI that combines slice-level and volume-level evaluation with volumetric visual grounding and chain-of-thought reasoning across two anatomical domains (brain and knee MRI). Built on expert radiologist annotations from the fastMRI+ dataset ( 36 ) with QA pairs and reasoning iteratively reviewed by a highly skilled clinical expert, our benchmark spans both image-level tasks (detection, localization, classification, diagnosis, captioning) and volume-level tasks following a hierarchical structure (detection \rightarrow localization \rightarrow classification \rightarrow counting \rightarrow captioning) that mirrors how a radiologist reconstructs a 3D understanding across slices. For localization, we specify both which frame and where within the frame each finding occurs, providing anatomical descriptions alongside bounding box coordinates. Through our evaluation of 10 VLMs with metrics that separately measure textual accuracy, reasoning quality, and spatial grounding precision, we show that baseline VLMs demonstrate strong textual spatial understanding but struggle with pixel-level spatial grounding and accurate localization of findings. This reveals a gap between textual spatial understanding—recognizing anatomical regions described in language (e.g., the left or right hemisphere or the frontal lobe in brain MRI, and the medial compartment in knee MRI)—and pixel-level spatial reasoning, which requires precisely localizing findings in the image using spatial coordinates such as bounding boxes. This gap in spatial reasoning has not been captured by prior benchmarks, which either lack grounding evaluation or are limited to single-image settings. We address this limitation by fully fine-tuning Qwen3-VL-8B on our training set, which results in improved spatial grounding performance. This demonstrates that targeted supervision on spatially annotated medical data can help bridge this gap. To support future work, we open-source our benchmark, evaluation pipeline, and all model outputs at https://lamawmouk.github.io/SGMRI-VQA to encourage research on volumetric spatial grounding in knee and brain MRI data. In summary, our contributions are as follows: • We introduce SGMRI-VQA , a large-scale multi-frame VQA benchmark for knee and brain MRI images, with chain-of-thought reasoning and QA pairs iteratively reviewed by a highly skilled clinical expert. Image-level evaluation spans five tasks (detection, localization, classification, diagnosis, captioning), while volume-level evaluation follows a hierarchical structure (detection, localization, classification, counting, captioning) that mirrors how a radiologist reconstructs a 3D understanding across slices. • We introduce cross-slice visual grounding with spatially grounded localization —providing both anatomical descriptions and bounding box coordinates with slice indices, jointly evaluating where anatomically and where spatially each finding occurs. • We fine-tune Qwen3-VL-8B using full fine-tuning on our training set with bounding box supervision. Image-level QA pairs serve as fine-grained augmentation for volume-level training, providing slice-level supervision that helps the model learn spatial grounding within individual frames, and improves overall spatial precision. • We compare the performance of our finetuned Qwen3-VL-8B against 10 VLMs (3 proprietary models, 5 open-source models, and 2 domain-specific medical models), across two anatomical domains, multiple tasks, and at both image-level and volume-level granularity, quantifying the gap between textual competence and spatial grounding ability. Figure 1 : Examples of the diverse question tasks in SGMRI-VQA. Each task is illustrated with real MRI slices and chain-of-thought reasoning, generated at both volume-level and image-level granularities across brain and knee domains, with tasks providing spatially grounded bounding box coordinates. 2 Related Works Medical Visual Question Answering. Medical VQA has progressed rapidly, from early benchmarks like VQA-RAD ( 12 ) , SLAKE ( 15 ) , and PathVQA ( 10 ) to large-scale evaluations spanning hundreds of thousands of questions across diverse modalities ( 34 ; 11 ; 31 ; 26 ) . Yet all of these benchmarks operate on single 2D images—none require models to reason across multiple frames or localize findings within a volumetric context. MedFrameQA ( 32 ) took a step toward multi-image evaluation with 2–5 YouTube video frames, but these are temporally related procedural images, not spatially contiguous slices from a single scan. Our SGMRI-VQA is the first benchmark that requires models to reason over a 12–60 sequential knee and brain MRI slices forming a continuous 3D representation, producing both chain-of-thought textual answers and spatially grounded bounding box coordinates with cross-slice indices. Vision-Language Models. Proprietary VLMs such as GPT-4o ( 1 ) , Gemini ( 27 ) achieve strong performance on general multimodal benchmarks, while open-source models—LLaVA ( 17 ; 20 )

Read the complete text