MedVL-SAM2: A unified 3D medical vision–language model for multimodal reasoning and prompt-driven segmentation

Yang Xing, Jiong Wu, Savas Ozdemir, Ying Zhang, Yang Yang, Wei Shao, Kuang Gong, [0.5em] Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · 2026

Read MedVL-SAM2: A unified 3D medical vision–language model for multimodal reasoning and prompt-driven segmentation by Yang Xing, Jiong Wu, Savas Ozdemir, Ying Zhang, Yang Yang, Wei Shao, Kuang Gong, [0.5em] Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09879.txt.

Introduction

Yang Xing and six co-authors unify three-dimensional medical imaging with language in one model. Their 2026 arXiv paper presents MedVL-SAM2, a 3D medical vision-language model. It handles multimodal reasoning and prompt-driven segmentation together.

Medical vision-language models already write reports and answer questions about scans at the image level. Fine-grained grounding and volumetric reasoning in 3D remain hard. Doing both inside one general framework is harder still. A single model that reasons about a scan and outlines anatomy from a text prompt could streamline radiology work in busy hospitals. It could also give smaller clinics tools that now sit only in large research centers with big teams.

Catalog record

CREATORYang Xing, Jiong Wu, Savas Ozdemir, Ying Zhang, Yang Yang, Wei Shao, Kuang Gong, [0.5em] Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-09879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MedVL-SAM2: A unified 3D medical vision–language model for multimodal reasoning and prompt-driven segmentation Yang Xing, Jiong Wu, Savas Ozdemir, Ying Zhang, Yang Yang, Wei Shao, Kuang Gong, [0.5em] Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA Abstract Recent progress in medical vision–language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report generation and visual question answering (VQA). However, achieving fine-grained visual grounding and volumetric spatial reasoning in 3D medical VLMs remains challenging, particularly when aiming to unify these capabilities within a single, generalizable framework. To address this challenge, we proposed MedVL-SAM2, a unified 3D medical multimodal model that concurrently supports report generation, VQA, and multi-paradigm segmentation, including semantic, referring, and interactive segmentation. MedVL-SAM2 integrates image-level reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging, and incorporates a SAM2-based volumetric segmentation module to enable precise multi-granular spatial reasoning. The model is trained in a multi-stage pipeline: it is first pre-trained on a large-scale corpus of 3D CT image–text pairs to align volumetric visual features with radiology-language embeddings. It is then jointly optimized with both language-understanding and segmentation objectives using a comprehensive 3D CT segmentation dataset. This joint training enables flexible interaction via language, point, or box prompts, thereby unifying high-level visual reasoning with spatially precise localization. Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D segmentation tasks. Extensive analyses further show that the model provides reliable 3D visual grounding, controllable interactive segmentation, and robust cross-modal reasoning, demonstrating that high-level semantic reasoning and precise 3D localization can be jointly achieved within a unified 3D medical VLM. 1 Introduction Leveraging large-scale radiology image-text pairs, medical vision–language models (VLMs) exhibit strong reasoning ability over both global image semantics and clinical narratives. Figure 1 : Comparison of the proposed model to traditional text-response-only medical vision-language models (VLMs) and mask-response-only medical vision foundation models (VFMs). Furthermore, several studies have incorporated visual grounding into VLMs, enabling them to spatially localize disease-related regions or anatomical structures referenced in language prompts. However, the reliance on 2D vision-language architectures inherently constrains their effectiveness in 3D medical image analysis. Specifically, 2D VLMs lack volumetric perception and spatial consistency across slices, both of which are essential for understanding complex 3D structures such as tumors, vessels, and organs. Consequently, while 2D grounding may localize features within an individual slice, it fails to capture the spatial continuity, anatomical context, and morphological variations present across the full 3D volume. Recent efforts have begun to explore 3D medical VLMs, such as Med3DVLM 46 and CT-Chat 9 , which process volumetric data and generate textual outputs. Nevertheless, these models remain limited to language-only output, lacking pixel-level grounding or segmentation capabilities, which restricts their applicability in clinically meaningful interpretation and interaction. A significant gap thus remains in developing 3D medical VLMs capable of comprehensive visual grounding and aligning textual concepts with 3D spatial regions. To address this, several grounded 3D VLMs have been proposed, including M3D 2 and VILA-M3 30 . M3D incorporated 3D information but is constrained by the low input resolution of its vision encoder and the design of its projection layer, both of which introduce substantial volumetric information loss. Meanwhile, VILA-M3 relies on domain-specific expert modules for downstream tasks, and its performance may become unstable when these components are not effectively activated. These limitations underscore the need for a unified 3D medical VLM capable of understanding volumetric image structures while preserving strong language–vision alignment. To address these challenges, we propose MedVL-SAM2, a unified 3D medical VLM that extends a LLaVA-style architecture to incorporate volumetric perception and multi-modal reasoning. Fig. 1 illustrates the differences of our model compared to traditional medical vision-language models (VLMs) and vision foundation models (VFMs). MedVL-SAM2 incorporates image-level understanding and pixel-level segmentation within a single framework, enabling both high-level language-driven analysis and fine-grained spatial localization. Specifically, our model integrates the prompt-driven paradigm of SAM2 into a 3D vision-language backbone, supporting multi-modal interaction through language, point, and bounding-box prompts. Our key contributions are summarized as follows: • The proposed model provides a unified 3D medical VLM capable of jointly performing image-level reasoning and pixel-level perception, supporting report generation, VQA, and multiple types of 3D segmentation. • The framework incorporates multi-modal prompting through language, point, and bounding-box inputs, enabling flexible and interactive 3D image segmentation. • The method achieves superior performance on two large-scale benchmarks, consistently surpassing current approaches in both reasoning and segmentation tasks. 2 Related Works Table 1 : Capability comparison between our model and current state-of-the-art methods across report generation (RG), visual question answering (VQA), semantic segmentation (SS), referring segmentation (RS), and interactive segmentation (IS). Method RG VQA SS RS IS BiomedParse 55 \times \times \checkmark \checkmark \times medSAM2 27 \times \times \checkmark \times \checkmark CT-CHAT 9 \checkmark \checkmark \times \times \times Med3DVLM 46 \checkmark \checkmark \times \times \times Med-2E3 38 \checkmark \checkmark \times \times \times MS-VLM 15 \checkmark \checkmark \times \times \times M3D 2 \checkmark \checkmark \checkmark \checkmark \times Ours \checkmark \checkmark \checkmark \checkmark \checkmark 2.1 Vision-Language Models (VLMs) Vision-language models (VLMs) integrate visual encoders with LLMs for vision-language understanding. Early approaches rely on complex fusion mechanisms, such as cross-attention (Flamingo 1 ) and query transformers (BLIP-2 20 ), while more recent approaches have converged toward simpler, unified designs. LLaVA 24 ; 17 shows that a vision encoder with a linear projection is sufficient for effective vision-language alignment, leading to numerous extensions (LLaVA-Next 19 , VILA 22 , InternVL 6 ), and recent unified foundation models (GPT-5 31 , Gemini2.5 7 , Qwen3 48 ). While these models excel in text generation and high-level reasoning, they lack inherent spatial grounding capabilities. Recent works have integrated pixel-level localization through segmentation adapters (LISA 13 , VISA 47 ), region encoders (GLaMM 34 , Ferret 50 ; 52 ), and multi-prompt frameworks (SA2VA 51 ). However, these grounding-enabled VLMs operate solely on 2D images or videos and cannot process 3D volumetric medical data. 2.2 Medical VLMs The integration of multimodal reasoning into medical imaging has led to rapid progress in medical VLMs for report generation and VQA. Early efforts such as LLaVA-Med 18 and Med-Flamingo 29 adapt general-purpose multimodal frameworks to radiology datasets, enabling report generation and VQA. Subsequently, HuatuoGPT-Vision 53 , HealthGPT 23 , MedTrinity 45 , and RadFM 43 extend this line of work by coupling visual-language reasoning with radiology-specific representations to improve clinical understanding. More recent models, including MIMO 5 and MedPLIB 11 , improve fine-grained visual grounding through language-guided segmentation and pixel-level reasoning. However, all these models are constrained to 2D slice-based analysis and cannot capture volumetric spatial continuity, which is essential for accurate 3D medical reasoning. To address this limitation, several models have advanced toward volumetric understanding. CT-Chat 9 and Med3DVLM 46 align 3D CT embeddings with radiology reports for volumetric text generation but lack segmentation or grounding. VividMed 25 , MS-VLM 15 and Med-2E3 38 extended CT-CHAT model by integrating both 3D and 2D features, but still restricted to text-only tasks. M3D 2 introduces segmentation heads for spatial reasoning but suffers from information loss caused by the compression of input image dimension and projection layer, while ViLA-M3 30 employs an agent-triggered segmentation mechanism that highly depended on explicit token activation. 2.3 Vision Foundation

Read the complete text