Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis
Hao Wu 1# , Hui Li 2# , Yiyun Su 3 · 2025
Read Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis by Hao Wu 1# , Hui Li 2# , Yiyun Su 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24013.txt.
Introduction
Hao Wu, Hui Li, and Yiyun Su re-engineer a segmentation foundation model to help vision-language models diagnose disease in three-dimensional medical scans. Such scans challenge diagnostic models twice over: information from several modalities must be fused, and small lesions, the very features that matter, are easy to overlook. Their two-stage framework, Hilbert-VLM, segments first and diagnoses second.
The segmentation stage rebuilds the Segment Anything Model 2 around a Hilbert-Mamba block: Hilbert space-filling curves guide the scanning of the Mamba state-space model so three-dimensional neighborhoods stay neighbors, preserving spatial locality. A cross-attention mechanism and scale-aware decoder recover fine detail. Segmentation masks and their text attributes are fused into a rich prompt for the diagnostic model. On the BraTS2021 brain tumor benchmark the system reported a Dice score near eighty-two percent and classification accuracy near seventy-nine percent.
Catalog record
| CREATOR | Hao Wu 1# , Hui Li 2# , Yiyun Su 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24013
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis Hao Wu 1# , Hui Li 2# , Yiyun Su 3 Abstract Abstract—Recent studies suggest that Visual Language Models (VLMs) hold great potential for tasks such as automated medical diagnosis. However, processing complex three-dimensional (3D) multimodal medical images poses significant challenges—specifically, the effective integration of complementary information and the occasional oversight of subtle yet critical pathological features. To address these issues, we present a novel two-stage fusion framework termed Hilbert-VLM . This framework leverages the HilbertMed-SAM module for precise lesion segmentation, with the generated multimodal enhanced prompts then guiding the VLM toward accurate disease classification. Our key innovation lies in the systematic redesign of the Segment Anything Model 2 (SAM2) architecture: we incorporate Hilbert space-filling curves into the scanning mechanism of the Mamba State Space Model (SSM) to maximize the preservation of spatial locality in 3D data, a property critical for medical image analysis. We also introduce a novel Hilbert-Mamba Cross-Attention (HMCA) mechanism and a scale-aware decoder to capture fine-grained details. Meanwhile, the prompt enhancement module unifies segmentation masks and their corresponding textual attributes into an information-dense prompt to support VLM inference. Extensive experiments were conducted to validate the effectiveness of the Hilbert-VLM model. On the BraTS2021 segmentation benchmark, it achieves a Dice score of 82.35% , with a diagnostic classification accuracy ( ACC ) of 78.85% . These results demonstrate that the proposed model offers substantial potential to improve the accuracy and reliability of medical VLM-based analysis. I Introduction Accurate diagnosis of brain pathologies is critical for clinical evaluation and treatment [ 7 ] . Radiologists rely on multimodal MRI sequences (e.g., T1 and FLAIR) to synthesise complementary information [ 8 ] . However, this manual process is time-consuming and prone to inter-observer variability [ 5 , 6 ] . Large visual-language models (VLMs), such as Qwen-VL, are poised to bring about a paradigm shift in automated diagnosis, as demonstrated by the work of Achiam et al. [ 1 ] and Bai [ 3 ] , thanks to their advanced cross-modal understanding capabilities. However, applying general-purpose VLMs, which are designed for 2D natural images, to 3D multimodal medical imaging remains challenging. For instance, these models cannot easily fuse the complex and heterogeneous spatial data from 3D scans. Secondly, pathological lesions tend to be small and the differences between them and healthy tissue are often very subtle. This may cause VLMs to overlook critical details [ 4 ] . Such perceptual failures, stemming from inadequate visual information fusion, limit the reliability of VLMs in high-stakes clinical applications [ 2 ] . Current approaches, such as fine-tuning or prompt engineering, do not address this fusion problem in a fundamental way. General-purpose segmentation models like SAM2 [ 10 ] also show limited performance without deep domain adaptation [ 9 , 11 ] . We posit that accurate diagnosis requires a framework that can first ’perceive accurately’ and then ’reason correctly.’ We propose Hilbert-VLM, a ”segmentation-first, fusion-prompt” two-stage framework. First, we develop HilbertMed-SAM, a segmentation model designed for 3D multimodal imaging to achieve precise lesion delineation. Second, an innovative prompt module integrates the visual segmentation mask and textualized lesion attributes from HilbertMed-SAM into a unified, enhanced prompt to guide a VLM (we use Qwen-VL) for disease classification. Our main contributions are: (i) A novel two-stage framework that integrates a Vision Foundation Model (VFM) and a VLM by decoupling ‘perception‘ and ‘cognition‘ tasks; (ii) The Hilbert-Mamba-Empowered SAM2 architecture, which re-engineers the encoder, memory, and decoder of SAM2 using Hilbert curves and Mamba SSMs for superior multimodal fusion and minute lesion detection; (iii) A Visual-Text Fusion Prompt mechanism that encodes and fuses the segmentation mask and lesion metadata to significantly improve VLM diagnostic accuracy; and (iv) SOTA performance on both lesion segmentation and disease classification tasks, offering a robust solution for AI-driven clinical decision support systems. II Related Work II-A Foundation Models for Medical Image Segmentation Medical image segmentation is fundamental to clinical tasks. Architectures have progressed from CNNs (like 3D U-Net) to Transformer-based frameworks (like UNETR). More recently, the Segment Anything Model (SAM) introduced promptable segmentation, which led to medical-specific adaptations such as MedSAM. SAM2 incorporates additional memory-enhancing mechanisms, thereby unlocking new potential for 3D volume processing. However, we believe that the unique spatio-temporal nature of three-dimensional, multimodal medical data means that SAM2 requires more than just minor adjustments, it needs a fundamental architectural overhaul. II-B Vision-Language Models in Clinical Applications Early visual language models (VLMs), such as CLIP, rely on large-scale paired image-text datasets. However, these datasets are difficult to obtain in medical contexts due to strict privacy restrictions and high costs. This has driven adaptations of general-purpose VLMs (e.g., Qwen-VL, Med-PaLM) using limited in-domain data. However, these adapted models often fail to recognise subtle details, which can result in misdiagnoses. To address this, we require two advances: an upstream model for precise visual perception, and a mechanism to convert visual discoveries, such as our segmentation masks, into a format suitable for VLM-based reasoning. Our multimodal fusion prompt module is designed to accomplish this task. II-C State-Space Models and Locality Preservation in Vision State Space Models (SSMs), such as Mamba, are considered to be an effective alternative to transformers. Mamba is inherently one-dimensional. When combined with standard image flattening, it disrupts the spatial locality of 3D data, which hinders its application in medical imaging. Hilbert space-filling curves solve this by mapping 3D data to 1D while preserving spatial proximity. This allows Mamba to retain spatiotemporal information. We integrated this Hilbert-guided scanning with Mamba to redesign SAM2’s encoder, memory, and decoder. The resulting architecture balances computational efficiency with high-fidelity spatial awareness, overcoming the earlier perception failures. Fig. 1: The architecture of HilbertMed-SAM encompasses multimodal input and HMCA fusion, the Hilbert Mamba optimisation module and a dual-path (Stage 1/Stage 2) decoder for the final prediction. III Methodology III-A Overall Framework of Hilbert-VLM Hilbert-VLM is a ”segment-then-prompt” two-stage framework (Figure 1 ). Stage 1 involves precise lesion segmentation. Stage 2 then uses these results to guide VLM for disease classification. III-B HilbertMed-SAM: Re-engineering SAM2 for Medical Imaging The cornerstone of this framework is HilbertMed-SAM, which has been redesigned for multimodal 3D medical imaging (see Figure 1 ). It comprises a foundational Hilbert-Mamba block, a multimodal encoder, memory-infused enhancement modules, and a dual-path decoder. III-B 1 Foundational Principle: The Hilbert-Mamba Block The Hilbert-Mamba Block (Figure 2 ) prevents locality issues in standard serialisation. A Hilbert Flatten uses a Hilbert curve to map 3D features to 1D, preserving relationships. This locality-aware sequence is processed by a Mamba block, whose State-Space Model (SSM) core efficiently models long-range dependencies while leveraging the preserved local context. Fig. 2: The Hilbert Mamba block is based on a particular scanning principle. Unlike standard grating scans, which destroy spatial locality, our approach uses a 3D Hilbert curve scanning strategy. III-B 2 Multimodal Encoder with HMCA The encoding process (Figure 1 ) begins by passing each modality through a separate SAM Encoder. To merge complementary information, we introduce the HMCA module. First, the Key and Value ( K,V ) representations from the second modality are contextualized by a Hilbert-Mamba block and refined by an MLP. Second, the Query ( Q ) from the