Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction
Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira, Sanne Abeln, Wilson Silva · 2025
Read Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction by Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira, Sanne Abeln, Wilson Silva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16069.txt.
Introduction
Aniek Eijpe and co-authors build a model that predicts how long cancer patients survive, using two kinds of data at once. One kind is the slide image of tumor tissue, and the other is the tumor's gene activity. Work published in 2025 sets out their method, called DIMAF, which tries to keep what the two data kinds share separate from what each holds alone.
Many older models mix those two kinds of signals together, which hides why a prediction was made. DIMAF uses an attention step to split shared and single-source signals, plus a loss term that pushes them apart. A scoring step then shows how much each part drove the forecast. On four public data sets, the method beat leading rivals on accuracy and on keeping the signals apart.
Catalog record
| CREATOR | Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira, Sanne Abeln, Wilson Silva |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16069
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction Aniek Eijpe, Soufyan Lakbir, Melis Erdal Cesur, Sara P. Oliveira, Sanne Abeln, Wilson Silva Abstract To improve the prediction of cancer survival using whole-slide images and transcriptomics data, it is crucial to capture both modality-shared and modality-specific information. However, multimodal frameworks often entangle these representations, limiting interpretability and potentially suppressing discriminative features. To address this, we propose Disentangled and Interpretable Multimodal Attention Fusion (DIMAF), a multimodal framework that separates the intra- and inter-modal interactions within an attention-based fusion mechanism to learn distinct modality-specific and modality-shared representations. We introduce a loss based on Distance Correlation to promote disentanglement between these representations and integrate Shapley additive explanations to assess their relative contributions to survival prediction. We evaluate DIMAF on four public cancer survival datasets, achieving a relative average improvement of 1.85 % 1.85\% in performance and 23.7 % 23.7\% in disentanglement compared to current state-of-the-art multimodal models. Beyond improved performance, our interpretable framework enables a deeper exploration of the underlying interactions between and within modalities in cancer biology. 1 Introduction Integrating multimodal cancer data offers a promising approach for enhancing deep learning-based survival prediction and personalized cancer care. Each modality can provide complementary insights into tumor biology, and a single data modality might therefore not be sufficient to fully capture the heterogeneity and complexity of cancer [ 26 ] . For example, genomics can reveal genetic mutations driving tumor development, while medical images can capture tissue alterations. By combining modalities, we can leverage the strengths of each data source to improve predictive accuracy [ 2 , 26 ] , aligning with clinical practice. Among multimodal strategies, integrating omics data with Whole Slide Images (WSIs) has gained increasing attention [ 21 , 5 , 6 , 13 ] . Early multimodal approaches typically opted for simple mechanisms, like concatenation [ 21 ] , or a gated Kronecker product [ 5 ] . However, these mechanisms generally fail to capture complex interactions between omics data and WSIs [ 6 ] . Recent advances address this through multilevel fusion, as seen in HBFSurv [ 15 ] , or attention-based methods [ 6 , 30 ] . Among the latest attention-based frameworks, SurvPath [ 13 ] processes bulk transcriptomics data with biological pathways [ 16 ] to generate semantically meaningful representations and fuses them with WSI patch features using a memory-efficient transformer. Building on this, MMP [ 25 ] summarizes the WSIs with Gaussian Mixture Models (GMMs) [ 24 ] , paving the way for full transformer-based fusion and post-hoc interpretability analysis. Despite significant advances, multimodal methods focus primarily on shared information across modalities [ 13 , 25 , 30 , 6 ] , leading to the suppression of modality-specific information [ 31 ] . While shared information can enhance robustness [ 26 ] , information specific to each modality can provide additional discriminative power. To address this, Disentangled Representation Learning (DRL) [ 1 ] can be used to decompose the input into modality-shared and modality-specific representations. Recent works have explored this, promoting disentanglement with minimizing an orthogonal [ 29 ] or a contrastive loss [ 23 ] , and using adversarial training [ 29 , 23 ] . Similarly, PIBD [ 31 ] promotes disentanglement by minimizing a contrastive log-ratio upper bound of mutual information [ 7 ] . However, none of these methods have explicitly measured disentanglement or assessed the contribution of the modality-specific and modality-shared representations to survival prediction. In this work, we introduce Disentangled and Interpretable Multimodal Attention Fusion (DIMAF) , an inherently interpretable framework that disentangles the fusion between WSIs and bulk transcriptomics for cancer survival prediction. Specifically, we use two self-attention layers to capture the intra-modal interactions and two cross-attention layers to model the inter-modal interactions, resulting in separate modality-specific and modality-shared representations. Unlike MMP [ 25 ] , we disentangle the multimodal interactions to provide a structured, disentangled view of multimodal information, enhancing interpretability and eliminating the need for post-attention FNNs. Measuring and enforcing disentanglement between latent representations without the ground truth is challenging and remains underexplored [ 3 ] . To address this, we employ Distance Correlation (DC) [ 27 , 18 ] , a statistical metric measuring the dependence between two representations. DC does not require pre-defined kernels, as the Hilbert-Schmidt independence criterion [ 11 ] does, and captures both linear and non-linear dependencies, unlike orthogonality constraints. We assess the interpretability of our framework by having a pathologist evaluate the learned WSI representations and use SHapley Additive exPlanations (SHAP) [ 20 ] , a feature attribution method, to quantify the relative contributions of the modality-specific and modality-shared representations. We evaluate DIMAF on four public cancer survival datasets, demonstrating improved performance and disentanglement over state-of-the-art multimodal models. Beyond improved performance, our framework provides a strong foundation for deeper exploration of multimodal cancer biology. In summary, our work makes the following key contributions: • We introduce DIMAF , an interpretable multimodal cancer survival model that explicitly disentangles the intra- and inter-modal interactions between WSIs and transcriptomics data within an attention-based fusion mechanism. • To promote disentanglement and learn distinct modality-specific and modality-shared representations, we incorporate a disentanglement loss based on DC. • We employ SHAP to gain insights into the importance of modality-shared and modality-specific representations. • We evaluate DIMAF on four public cancer survival datasets, demonstrating enhanced performance, disentanglement, and interpretability. 2 Methodology We propose a disentangled approach for survival analysis that leverages multimodal data. Specifically, for each patient i , the objective is to obtain a risk prediction r_{i} , given their WSI \mathbf{H_{i}} and transcriptomics data \mathbf{g_{i}} . Figure 1 provides an overview of our framework, consisting of three main components: unimodal feature extraction, disentangled attention fusion, and survival prediction. Figure 1: Overview of the proposed framework. First, the transcriptomics data \mathbf{g_{i}} is processed through pathway tokenizing and f_{g,\{1,..,N_{g}\}} to obtain the final pathway embedding \mathbf{Z_{g,i}} . From the WSI \mathbf{H_{i}} , we obtain patch features \{\mathbf{z_{i,j}}\}_{j=1}^{N_{hi}} , which are aggregated and processed through f_{h} to obtain the slide embedding \mathbf{Z_{h,i}} . Then, \mathbf{Z_{g,i}} and \mathbf{Z_{h,i}} are processed through the Disentangled Attention Fusion module, obtaining \mathbf{z_{hh,i}},\mathbf{z_{hg,i}},\mathbf{z_{gh,i}},\text{ and }\mathbf{z_{gg,i}} . These representations are further disentangled in the Disentanglement Module and used to compute the survival risk r_{i} in the Survival Module. 2.1 Unimodal feature extraction We process the transcriptomics data and WSIs separately to obtain interpretable unimodal representations. Following prior methods [ 31 , 25 ] , we tokenize the transcriptomics data \mathbf{g_{i}}\in\mathbb{R}^{D_{g}} , representing expression levels of D_{g} different genes, by using N_{g} biological pathways as interpretable features. Each pathway represents a group of genes active in specific biological processes in cancer [ 16 ] . The tokenized gene expressions are processed through separate Self-Normalizing Networks (SNNs) [ 14 ] ( f_{g,\{1,..,N_{g}\}} ) and N_{g} random-initialized and learnable pathway encodings [ 25 ] are concatenated to each of the obtained pathway embeddings. Lastly, these pathway embeddings with the concatenated encodings are stacked (S) to obtain the final pathway embedding \mathbf{Z_{g,i}}\in\mathbb{R}^{N_{g}\times D_{gh}} . The WSI \mathbf{H_{i}} is split into N_{hi} non-overlapping patches, from which features are extracted using a frozen encoder f_{hp} . To obtain a slide representation from these patch features, we use PANTHER [ 24