MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality

Kyungwon Kim, Dosik Hwang · 2026

Read MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality by Kyungwon Kim, Dosik Hwang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26071.txt.

Introduction

Kim and Hwang present MUST, a survival prediction model that copes with missing medical data types. Cancer care often lacks one modality, such as pathology or genomics. MUST splits each modality's representation into a private part and a shared part. Algebra in a low-rank shared space enforces the split. This shows exactly what is lost when a modality is absent. A diffusion model then rebuilds the missing private part. On five TCGA datasets, the authors report top scores with full data and robust scores with gaps.

Catalog record

CREATORKyungwon Kim, Dosik Hwang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52037
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26071

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality Kyungwon Kim, Dosik Hwang Abstract Accurate survival prediction from multimodal medical data is essential for precision oncology, yet clinical deployment faces a persistent challenge: modalities are frequently incomplete due to cost constraints, technical limitations, or retrospective data availability. While recent methods attempt to address missing modalities through feature alignment or joint distribution learning, they fundamentally lack explicit modeling of the unique contributions of each modality as opposed to the information derivable from other modalities. We propose MUST (Modality-Specific representation-aware Transformer), a novel framework that explicitly decomposes each modality’s representation into modality-specific and cross-modal contextualized components through algebraic constraints in a learned low-rank shared subspace. This decomposition enables precise identification of what information is lost when a modality is absent. For the truly modality-specific information that cannot be inferred from available modalities, we employ conditional latent diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors. Extensive experiments on five TCGA cancer datasets demonstrate that MUST achieves state-of-the-art performance with complete data while maintaining robust predictions in both missing pathology and missing genomics conditions, with clinically acceptable inference latency. Our project page is available at https://kylekwkim.github.io/MUST/ . 1 Introduction Leveraging multiple data modalities has been shown to substantially improve performance across various domains Baltrušaitis et al. 2018 ; Zhang et al. 2023 ; Girdhar et al. 2023 . In survival prediction, integrating diverse information sources—such as whole slide images (WSI), genomic profiles, and other clinical data—enables more comprehensive prognostic models than single-modality approaches Chen et al. 2022 ; Vale-Silva and Rohr 2021 ; Cheerla and Gevaert 2019 . Among these, WSI capturing tissue morphology and genomic data revealing molecular alterations have emerged as particularly critical Chen et al. 2020 , as improved survival prediction directly translates to better risk stratification for treatment planning Acosta et al. 2022 . However, a critical gap exists between research settings with complete multimodal data and clinical reality where modalities are frequently incomplete. In practice, genomic profiling remains expensive and time-consuming, limiting its routine clinical use. Historical patient cohorts often contain extensive histopathology archives without corresponding molecular data. Conversely, emerging precision medicine initiatives may have genomic data without complete tissue samples. This incompleteness poses a fundamental challenge: existing multimodal models typically assume complete data availability and exhibit severe performance degradation with missing modalities Acosta et al. 2022 ; Li et al. 2022 . Current approaches to handling missing modalities can be categorized into three paradigms. Feature alignment methods Wang et al. 2023b ; Radford et al. 2021 ; Ma et al. 2021 train models to produce similar representations regardless of which modalities are available, but do not explicitly identify what information is missing—they merely enforce output similarity without modeling the semantic structure of cross-modal relationships. Imputation-based methods Dorent et al. 2019 ; Sharma and Hamarneh 2019 attempt to synthesize missing features through generative models, but often operate in high-dimensional feature spaces, producing noisy pseudo-features that can degrade downstream performance. Joint distribution learning methods like LD-CVAE Zhou et al. 2025 learn joint posteriors for conditional generation, capturing modality interactions but failing to disentangle what each modality uniquely contributes versus what can be derived from others. While information disentanglement has been studied in multimodal learning Daunhawer et al. 2023 ; Lee and Pavlovic 2021 , these approaches focus on representation interpretability rather than enabling algebraic recoverability of missing components. A fundamental limitation shared by existing methods is the absence of explicit modeling of what information is modality-specific versus cross-modally inferable. We propose MUST (Modality-Specific representation-aware Transformer), which explicitly decomposes each modality’s global representation into modality-specific and cross-modal contextualized components through bidirectional cross-attention and algebraic constraints in a learned low-rank shared subspace. Unlike unconstrained cross-attention methods (e.g., SurvPath Jaume et al. 2024 , CMTA Zhou and Chen 2023 ) that lack missing modality mechanisms, and unlike disentanglement approaches that separate components for interpretability alone, our framework enforces algebraic invertibility: shared components are deterministically derivable from any available modality. For truly modality-specific information, we train conditional latent diffusion models Ho et al. 2020 ; Rombach et al. 2022 to generate plausible representations conditioned on recovered shared information and structural priors Özbey et al. 2023 ; Chung et al. 2022 . The main contributions are: \bullet An explicit decomposition architecture with bidirectional cross-attention and algebraic constraints in a low-rank shared subspace that identifies precisely what information is missing when a modality is absent, enabling deterministic recovery of shared components. \bullet A two-stage reconstruction strategy combining deterministic algebraic projection for shared information with LDM-based probabilistic generation for modality-specific information, isolating stochastic generation to truly modality-specific residuals only. Extensive experiments on five TCGA cancer datasets demonstrate that MUST achieves state-of-the-art performance with complete data (C-index: 0.742) while maintaining robust predictions under missing genomics (0.716) and missing pathology (0.739), substantially outperforming existing methods in all scenarios. Figure 1 : Overall architecture of MUST. The framework extracts global representations g_{P} and g_{G} via self-attention, computes shared information through bidirectional cross-attention, and decomposes each modality into specific and shared components for survival prediction. 2 Related Work 2.1 Multimodal Survival Prediction Deep learning has revolutionized survival prediction in computational pathology, from CNNs for automatic feature extraction from gigapixel WSIs Campanella et al. 2019 to attention-based methods like CLAM Lu et al. 2021 and TransMIL Shao et al. 2021 for patch-level aggregation. For multimodal integration, Chen et al. Chen et al. 2020 systematically compared early, late, and intermediate fusion across cancer types. Chen et al. Chen et al. 2021 employ multimodal co-attention transformers to capture fine-grained interactions between pathology and genomic features. Jaume et al. Jaume et al. 2024 leverage foundation models for enhanced feature extraction, while Zhang et al. Zhou and Chen 2023 use cross-modal transformers for attention-based fusion. Lipkova et al. Xu and Chen 2023 proposed multi-omic transformers for integrating diverse molecular data. However, these methods assume complete modality availability and cannot handle missing data scenarios, limiting their clinical applicability. 2.2 Handling Missing Modalities The missing modality problem has motivated several research directions. Alignment-based approaches train models to produce consistent representations across modality combinations. Ma et al. Ma et al. 2021 use meta-learning, Chen et al. Zhang et al. 2022 employ mutual information maximization, and Wang et al. Wang et al. 2023a learn shared and specific features through distribution alignment. While effective, these methods implicitly enforce similarity without explicitly identifying what information is missing. Imputation-based methods synthesize missing features through generative models, often producing noisy pseudo-features in high-dimensional spaces. Joint distribution learning methods like Zhou et al. Zhou et al. 2025 learn joint posteriors through conditional VAEs, enabling conditional generation but without disentangling modality-specific from shared information. While disentanglement has been studied in multimodal learning Daunhawer et al. 2023 ; Lee and Pavlovic 2021 ; Lee et al. , these approaches focus on representation interpretability rather than deterministic recovery. Similarly, disentanglement of domain-invariant from domain-specific features has proven effective in medical image segmentation for domain generalization Kim et al. 2023 , validating the broader principle that separating shared from specific components yields robust representations. In parallel, regularization-based methods like VICReg Bardes et al. 2021 learn decorrelated features through variance-covariance constraints, and disentangled VAEs Higgins et al. 2017 separate independent latent

Read the complete text