M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model

Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen · 2026

Read M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model by Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08936.txt.

Introduction

Liu and five colleagues propose M-IDoL, a medical foundation model trained without labels that splits what it learns by imaging type instead of blending every modality into one embedding. Blending, they argue, blurs modality identity and variety, weakening later clinical tasks. Pretrained on 1.15 million medical images, the model tops 20 rival foundation models across 21 downstream tasks spanning five modalities, from X-ray and fundus to pathology. Its learned features separate imaging types cleanly while picking out fine detail within each.

Catalog record

CREATORYihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63859
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08936

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen Abstract Medical foundation models (MFMs) aim to learn universal representations from multimodal medical images that can generalize effectively to diverse downstream clinical tasks. However, most existing MFMs suffer from information ambiguity that blend multimodal representations in a single embedding space, leading to the degradation of modality specificity and diversity. In this paper, we propose M-IDoL, a self-supervised M FM that introduces I nformation D ecomposition for multim o dal representation L earning via two objectives: i) maximize inter-modality entropy by dispersing multimodal representation into separable Mixture-of-Experts (MoE) subspaces to achieve representation specificity across modalities; and ii) minimize intra-modality uncertainty by performing fine-grained semantic discrimination within each MoE subspace to enrich representation diversity per modality. By pre-training on 1.15 million medical images, M-IDoL i) delivers superior generalization across 21 downstream clinical tasks, outperforming 20 foundation models on five imaging modalities (e.g., X-ray, fundus, OCT, dermoscopy and pathology), and ii) learns modality-specific and diverse representations, showing clearer separation of feature cluster across modalities and finer-grained feature discrimination within each modality. 1 Introduction Figure 1 : (a) Medical images exhibit inter-modality specificity and intra-modality diversity. (b) Modality-specific models excels in intra-modality diversity by focusing on stable imaging statistics. (c) MFMs suffer from information ambiguity due to uniform maximization of redundancy (Rdncy.) information across modalities. (d) M-IDoL mitigates ambiguity via information decomposition, enhancing modality-specific and diverse representation. X and Y denote representations of two augmented views, Z denotes representations from modalities distinct from X and Y . \mathcal{I}(\cdot;\cdot) is mutual information, H(\cdot;\cdot) is information entropy. Shapes (e.g., star, circle) indicate modalities; colors indicate intra-modality semantics (e.g., diseases). t-SNE clusters for (b)–(d) are in Fig. 5 . Medical foundation models (MFMs) have emerged as a promising paradigm for multi-domain medical imaging ( 70 ; 42 ; 37 ) . A prevailing strategy for current MFMs is unified Contrastive Learning (CL) ( 3 ) , where a single model is jointly optimized across medical imaging modalities (e.g., fundus, X-ray and dermoscopy) by maximizing Mutual Information (MI) between a pair of augmented images ( 69 ; 30 ) . Built on large-scale CL pre-training, MFMs are encouraged to learn universal representations that can be effectively adapted to a wide range of clinical tasks ( 42 ) . Despite their empirical scalability, the effectiveness of these unified MFMs is constrained due to the neglect of visual discrimination across medical imaging modalities ( 6 ; 70 ) . Specifically, as shown in Fig. 1 (a), medical images are inherently heterogeneous, with each modality providing distinct anatomical information ( 42 ) . For example, dermoscopy distinguishes early melanoma from benign nevi by subtle pigment irregularities, whereas lung-nodule radiographs rely on faint opacities and structural distortion. This fine-grained visual discrimination emphasizes the importance of inter-modality specificity and intra-modality diversity in medical multimodal representations ( 6 ) . However, prevailing MFMs are largely modality-agnostic, implicitly forcing a single embedding space to accommodate all modalities under a uniform CL objective ( 30 ; 42 ; 67 ) . This limitation can be termed as information ambiguity . To be specific, in modality-specific CL pre-training, as shown in Fig. 1 (b), MI maximization primarily reduces prediction uncertainty caused by augmentation noise. This encourages the model to focus on stable statistical patterns that support fine-grained semantic discrimination, such as small lesions, subtle texture changes, and boundary variations, thereby allowing the representation to achieve intra-modality diversity ( 72 ; 36 ; 57 ; 69 ; 60 ) . However, in unified pre-training, current MFMs typically apply the same CL objective to multi-modality data ( 42 ; 30 ) . As shown in Fig. 1 (c), this causes MI maximization to uniformly reduce predictive uncertainty from both augmentation noise and modality semantic shift. Such uniform reduction inevitably maximizes inter-modality mutual information, which is widely regarded as modality-shared redundancy in multimodal learning (‘Rdncy.’ in Fig. 1 (c)) ( 68 ; 64 ; 1 ) . As a result, MFMs bias their training objective toward learning semantically shared representations across modalities. This bias finally undermines fine-grained semantic discrimination, resulting in information ambiguity that degrades both inter-modality specificity and intra-modality diversity. A straightforward way to alleviate this information ambiguity is removing modality-shared redundancy from the CL objective, i.e., to maximize intra-modality MI without interfering with the representation learning of other modalities (Fig. 1 (d), left). However, in practice, this requires estimating a trivariate MI term ( \mathcal{I}(X;Y;Z) in Fig. 1 (d)) among the representations of a pair augmented images ( X and Y ), and representations from other modalities ( Z ). Such multivariate MI estimation is difficult and unstable at MFM scale in high-dimensional embeddings ( 59 ; 21 ) . To address this challenge, we propose M-IDoL, a self-supervised M edical foundation model that introduces I nformation D ecomposition to enhance modality-specific and diverse in multim o dal representation L earning. Specifically, as shown in Fig. 1 (d), M-IDoL decomposes the multivariate MI into two entropy-based objectives that explicitly encourages the model to i) maximize inter-modality information entropy (i.e., H(X|Z) in Fig. 1 (d)) by increasing the entropy of X given Z , ensuring that each representation retains information which is not predictable from other modalities, thereby enhancing modality specificity, and and ii) minimize intra-modality predictive uncertainty (i.e., H(X|Y,Z) in Fig. 1 (d)) by reducing uncertainty between paired views of the same image caused by augmentation noise rather than semantic shifts across modalities, thus improving fine-grained discrimination per modality to enhance representation diversity. In practice, inspired by the success of Mixture-of-Experts (MoE) in multimodal learning ( 6 ; 11 ; 68 ) , M-IDoL optimizes these two objectives by designing a MoE projector. Specifically, to optimize the first objective, M-IDoL introduces a routing-consistency loss to enforce a balanced expert assignment. This pushes multimodal representations into distinct MoE subspaces, promoting visual discrimination and thereby enhancing modality specificity (Fig. 1 (d)-i). To optimize the second objective, M-IDoL introduces an intra-modality contrastive loss that pulls same-modality representations within each MoE subspace. This reduces augmentation-caused predictive uncertainty beyond the variance contributed by other-modality representations, promoting modality diversity (Fig. 1 (d)-ii). Together, these two losses enable M-IDoL to reduce modality-shared redundancy, thereby mitigating the information ambiguity of unified CL pre-training. In summary, we make the following contributions: • We propose M-IDoL, a self-supervised MFM that introduces information decomposition to mitigate information ambiguity by explicitly removing modality-shared redundancy in unified multimodal CL pre-training. • Via information decomposition, M-IDoL introduces an MoE projector that i) encourages modality-specific representations by maximizing inter-modality information entropy, and ii) increases per-modality diversity by minimizing intra-modality predictive uncertainty. • Pre-trained on 1.15M unlabeled medical images across five modalities (X-ray, fundus, OCT, dermoscopy, pathology), M-IDoL effectively learns modality-specific and diverse representations, consistently outperforming existing MFMs on 21 downstream datasets. Figure 2 : Left: Overview of M-IDoL. Via information decomposition, M-IDoL optimizes two objectives: (a) the routing-consistency loss \mathcal{L}_{\text{route}} , which learns modality-separable MoE subspaces to maximize H(X|Z) for inter-modality specificity, and (b) the intra-modality contrastive loss \mathcal{L}_{\text{cst}} , which promotes fine-grained discrimination within each modality to minimize H(X|Y,Z) for intra-modality diversity. Right: (a) Inter-modality Entropy

Read the complete text