Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representations

Ho Hin Lee, Alberto Santamaria-Pang, Jameson Merkov, Matthew Lungren, Ivan Tarapov · 2025

Read Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representations by Ho Hin Lee, Alberto Santamaria-Pang, Jameson Merkov, Matthew Lungren, Ivan Tarapov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10057.txt.

Introduction

Ho Hin Lee and co-authors (2025) present a model that predicts cancer survival by reading several kinds of medical images together. Tumors show themselves differently in scans and in tissue slides, and single-image methods miss part of that picture. Their model, called M4Survive, takes feature sets from large pretrained medical models covering radiology and pathology images, then joins them in one shared space tuned for survival risk.

Small adapter networks built on the Mamba architecture do the joining, which keeps the computing cost low while letting the image types inform each other. On standard benchmark data, the fused model predicted survival more accurately than single-modality models and than older fixed fusion methods. The authors present the work as evidence that shared representations from foundation models can sharpen risk estimates in oncology, helping doctors judge which patients face the hardest course.

Catalog record

CREATORHo Hin Lee, Alberto Santamaria-Pang, Jameson Merkov, Matthew Lungren, Ivan Tarapov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10057

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representations Ho Hin Lee, Alberto Santamaria-Pang, Jameson Merkov, Matthew Lungren, Ivan Tarapov Abstract Accurate survival prediction in oncology requires integrating diverse imaging modalities to capture the complex interplay of tumor biology. Traditional single-modality approaches often fail to leverage the complementary insights provided by radiological and pathological assessments. In this work, we introduce M4Survive (Multi-Modal Mamba Modeling for Survival Prediction), a novel framework that learns joint foundation model representations using efficient adapter networks. Our approach dynamically fuses heterogeneous embeddings from a foundation model repository (e.g., MedImageInsight, BiomedCLIP, Prov-GigaPath, UNI2-h), creating a correlated latent space optimized for survival risk estimation. By leveraging Mamba-based adapters, M4Survive enables efficient multi-modal learning while preserving computational efficiency. Experimental evaluations on benchmark datasets demonstrate that our approach outperforms both unimodal and traditional static multi-modal baselines in survival prediction accuracy. This work underscores the potential of foundation model-driven multi-modal fusion in advancing precision oncology and predictive analytics. Both codes and interactive demo are available at: https://github.com/microsoft/healthcareai-examples 1 Introduction Survival prediction in oncology requires the integration of multi-scale imaging data to capture the complex interplay of biological and clinical variables. Radiology offers macro-level structural insights, while pathology provides detailed micro-level cellular information. However, traditional unimodal approaches fail to leverage the complementary prognostic value embedded within these disparate data sources, and classical radiomics methods often struggle to correlate features across these modalities, thereby limiting their capacity to model tumor heterogeneity and predict treatment response. Foundational reviews by Lambin et al. and Aerts et al. have demonstrated how quantitative image analysis can inform risk prediction, while also highlighting the inherent limitations of single-modality approaches [ 12 , 1 ] . Similarly, surveys on deep multi-modal learning and segmentation methods reveal that, although current techniques show promise, they frequently lack efficient cross-scale integration and computational scalability [ 16 , 9 ] . Recent advances in multi-modal imaging analysis have shown that aggregating information from different modalities can significantly enhance model performance. For instance, transformer-based models have been developed that integrate radiological, pathological, and molecular data for glioblastoma survival prediction, demonstrating substantial improvements over unimodal techniques despite challenges such as increased computational complexity and interpretability issues [ 10 ] . Similar studies in colon cancer have fused pathology and genomic data, further highlighting both the potential and the inherent difficulties of integrating heterogeneous sources, especially when accounting for temporal variations and data heterogeneity [ 15 ] . Despite these promising developments, existing methods predominantly operate under a “complete data” paradigm, wherein every patient record is assumed to have all modalities available [ 5 , 2 , 13 , 19 ] . In routine clinical practice, however, missing modalities are common—leading to data scarcity and an elevated risk of overfitting when incomplete cases are discarded. Moreover, static multi-modal fusion strategies have proven insufficient for fully capturing the dynamic and cross-scale interactions between radiology and pathology data [ 3 ] . Other approaches, such as modality dropout and uni-modal feature reconstruction, have attempted to mitigate these issues but often fall short of effectively leveraging the available data and thoroughly comparing model performance across scenarios with and without missing modalities [ 8 , 14 ] . To address these challenges, we introduce M4Survive (Multi-Modal Mamba Modeling for Survival Prediction), a novel framework that learns joint representations from foundation models trained on radiology and pathology data. By leveraging Mamba-based adapter networks, M4Survive fuses heterogeneous embeddings from multiple pre-trained models (e.g., MedImageInsight, BiomedClip, Prov-GigaPath, UNI2-h) into a correlated latent space optimized for survival risk estimation. The key contributions of our work are: • Adaptive Multi-Modal Fusion: Our framework employs efficient adapter networks to dynamically integrate multi-scale imaging data. This approach accommodates variations in data availability and effectively captures the complex interactions between macro-level and micro-level features—ensuring robustness in real-world scenarios. • Enhanced Predictive Accuracy and Interpretability: By jointly analyzing radiology and pathology data, M4Survive outperforms traditional unimodal and static multi-modal baselines. The resulting correlated latent space not only improves predictive accuracy but also enhances interpretability, offering deeper insights into how different imaging modalities contribute to survival prediction. • Scalability and Clinical Applicability: Although our current focus is on radiology and pathology, the adapter-based architecture of M4Survive is inherently scalable and only needs 15 seconds for training with small amount of CPU/GPU memory. Figure 1 : Overview of M4Survive. MRI diagnostic images with variable contrast and cancer site histopathology images are independently processed through dedicated healthcare foundation models to generate modality-specific embeddings. Such embeddings are subsequently mapped into a joint-modality semantic space following with modality-specific encoders and leverage a Mamba adapter to perform multi-modal fusion for survival predictions. 2 Methods In contrast to conventional fusion strategies that simply combine features via element-wise averaging or concatenation, our approach learns a joint foundation semantic space using modality-specific embeddings and an efficient adapter network. Our method follows a two-step hierarchical pipeline for multi-modal fusion. First, we pre-compute image embeddings using variable foundation models and then train modality-specific encoder networks to project these embeddings into a unified semantic space. Second, the resulting joint embeddings are fed as multiple tokens into a lightweight Mamba adapter network, which facilitates fine-grained cross-modal interactions for enhanced survival prediction. 2.1 Learning Joint Foundation Semantic Space Our approach begins by leveraging both general and modality-specific foundation models to extract high-dimensional embeddings from multi-modality images. In our setting, we use 2D diagnostic MRI slides with four different contrasts ( I_{T1},I_{T1PC},I_{T2},I_{FLAIR} ) and 2D cancer site pathology images (i.e., H&E, denoted as I_{Path} ) images. It is important to note that M4Survive can scale to accommodate longer sequences of images or 3D imaging, provided that the foundation models can generate the necessary embeddings. For instances, we leverage MedImageInsight and Prov-GigaPath as our radiology-pathology foundation models respectively and compute embeddings as follow: \displaystyle\langle F_{T1},F_{T1PC},F_{T2},F_{FLAIR}\rangle \displaystyle=MedImageInsight(\langle I_{T1},I_{T1PC},I_{T2},I_{FLAIR}\rangle) (1) \displaystyle F_{Path} \displaystyle=ProvGigaPath(I_{Path}) Since these embeddings originate from distinct feature spaces, we employ small, efficient encoder networks (i.e. 2 layers of Multi-Layer Peceptron (MLP)) to project each modality’s features into a unified semantic space as tokens: \displaystyle T_{Rad} \displaystyle=MLP_{Rad}(\langle F_{T1},F_{T1PC},F_{T2},F_{FLAIR}\rangle) (2) \displaystyle T_{Path} \displaystyle=MLP_{Path}(I_{Path}) Where T_{rad}\in\mathcal{R}^{4\times Channel_{rad}} and T_{Path}\in\mathcal{R}^{1\times Channel_{Path}} are the jointly normalized tokens of all MRI contrast modalities and the pairwise pathology image respectively. This projection step is crucial as it aligns the disparate representations, thereby bridging the semantic gap between the two modalities. By normalizing and compressing the modality-specific embeddings into the same hidden dimension and assume it is the learnable joint semantic space, our method facilitates a coherent and integrative representation that enables the downstream model to effectively reason over complementary information from both imaging domains. 2.2 Multi-Modal Fusion with Mamba Adapter The modality-specific embedding encoders encode the foundation model outputs and concatenate it as a sequence of input tokens for a lightweight Mamba adapter network [ 11 ] . Rather than collapsing these features into a single vector via concatenation or averaging, our token-based fusion strategy leverages a selective state-space model to preserve fine-grained, modality-specific details. For each token t_{n}\in\mathcal{R}^{d_{u}} in a long sequence concatenating with T_{rad} and T_{Path} , where n=1,...,5 is the total number of image modalities, the mamba network updates a hidden state h_{n} using the recurrence and generate an output representation of y_{n} as follows:

Read the complete text