Seeking Necessary and Sufficient Information from Multimodal Medical Data
Boyu Chen *, Weiye Bao *, Junjie Liu, Michael Shen, Bo Peng, Paul Taylor, Zhu Li 🖂, Mengyue Yang 🖂 · 2026
Read Seeking Necessary and Sufficient Information from Multimodal Medical Data by Boyu Chen *, Weiye Bao *, Junjie Liu, Michael Shen, Bo Peng, Paul Taylor, Zhu Li 🖂, Mengyue Yang 🖂 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00289.txt.
Introduction
Chen and colleagues ask which signals in combined medical scans and data are truly needed, and which alone settle an outcome. They argue features meeting both tests should predict better and stay useful when a data type goes missing. Their method splits each combined representation into shared and modality-specific parts, then builds learnable necessity-and-sufficiency goals for each part. Tests on synthetic and real medical data show the approach works.
Catalog record
| CREATOR | Boyu Chen *, Weiye Bao *, Junjie Liu, Michael Shen, Bo Peng, Paul Taylor, Zhu Li 🖂, Mengyue Yang 🖂 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25930 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seeking Necessary and Sufficient Information from Multimodal Medical Data Boyu Chen *, Weiye Bao *, Junjie Liu, Michael Shen, Bo Peng, Paul Taylor, Zhu Li 🖂, Mengyue Yang 🖂 Abstract Learning multimodal representations from medical images and other data sources can provide richer information for decision-making. While various multimodal models have been developed for this, they overlook learning features that are both necessary (must be present for the outcome to occur) and sufficient (enough to determine the outcome). We argue learning such features is crucial as they can improve model performance by capturing essential predictive information, and enhance model robustness to missing modalities as each modality can provide adequate predictive signals. Such features can be learned by leveraging the Probability of Necessity and Sufficiency (PNS) as a learning objective, an approach that has proven effective in unimodal settings. However, extending PNS to multimodal scenarios remains underexplored and is non-trivial as key conditions of PNS estimation are violated. We address this by decomposing multimodal representations into modality-invariant and modality-specific components, then deriving tractable PNS objectives for each. Experiments on synthetic and real-world medical datasets demonstrate our method’s effectiveness. Code will be available on GitHub. † † footnotetext: * These authors contributed equally. † † footnotetext: 🖂 Corresponding author. 1 Introduction Multimodal representation learning has become essential in medical imaging, as integrating multiple data sources provides richer information for prediction. Various multimodal models have been developed for this task, such as multimodal fusion [ 16 , 27 , 5 , 12 , 18 ] , contrastive learning [ 14 , 26 , 22 , 25 , 9 ] , and disentanglement [ 3 , 20 , 11 , 7 ] . However, they overlook the importance of learning the features that are both necessary and sufficient for the outcome. Necessity means a feature must be present for the outcome to occur, but its presence alone does not confirm the outcome (e.g., pulmonary infiltrates on chest CT are typical in pneumonia, but infiltrates alone are not diagnostic). Sufficiency means a feature can confirm the outcome when present, but it may be absent even if the outcome occurs (e.g., a visible pneumothorax line on chest X-ray confirms pneumothorax, but early pneumothorax may lack this line). Ideally, predictive features are both necessary and sufficient (e.g., a fracture line on X-ray indicates a fracture, and fractures present with such lines). Learning such features provides advantages for multimodal medical data. First, it improves predictive performance: necessity ensures critical information is captured, while sufficiency guarantees the captured information is predictive [ 23 , 1 , 2 ] . Second, it can handle missing modalities, which is a pervasive challenge in clinical practice [ 16 , 27 , 5 , 6 , 11 ] . When each modality learns to capture necessary and sufficient features, any available subset of modalities retains adequate predictive signal, enabling robust inference with incomplete data. Recent work has applied the Probability of Necessity and Sufficiency (PNS) to learn such features in unimodal data [ 23 , 21 , 4 , 1 , 2 ] . However, extending PNS to multimodal settings remains underexplored and is non-trivial as key conditions of PNS estimation are violated (details in section 2 ). We address this by decomposing multimodal representations into modality-invariant and modality-specific components, then deriving tractable PNS objectives for each. Experiments on synthetic and real-world medical datasets show the effectiveness of our method. 2 Preliminaries This section covers PNS computation, challenges in multimodal extension, and representation decoupling—foundations for our method in section 3 . PNS is the probability of a feature set being both necessary and sufficient for an outcome. Let Z\subset\mathbb{R}^{d_{z}} be the features of outcome Y\subset\mathbb{R}^{d_{y}} , where z and \bar{z} are two distinct values of Z . The PNS of Z with respect to Y for z and \bar{z} is defined as: \text{PNS}(z,\bar{z}):=P(Y_{\mathrm{do}(Z=z)}=y|Z=\bar{z},Y\neq y)\cdot P(Z=\bar{z},Y\neq y)+P(Y_{\mathrm{do}(Z=\bar{z})}\neq y|Z=z,Y=y)\cdot P(Z=z,Y=y) [ 19 , 23 ] . The do-operator \mathrm{do}(Z=z) denotes an intervention setting Z to z, which cannot be directly computed from observational data. However, under two conditions, PNS can be estimated using observational data: Definition 1 (Exogeneity [ 19 ] ) Z is exogenous to Y if Z has a direct effect on Y without confounding factors, i.e., P(Y_{do(Z=z)}=y)=P(Y=y\mid Z=z) . Definition 2 (Monotonicity [ 19 ] ) Y is monotonic w.r.t. Z when (Y_{do(Z=z)}\neq y)\wedge(Y_{do(Z=\bar{z})}=y) or (Y_{do(Z=z)}=y)\wedge(Y_{do(Z=\bar{z})}\neq y) is false, which can be presented as: P(Y_{do(Z=z)}=y)P(Y_{do(Z=\bar{z})}\neq y)=0 or P(Y_{do(Z=z)}\neq y)P(Y_{do(Z=\bar{z})}=y)=0 . Lemma 1 ( [ 19 ] ) Under monotonicity and exogeneity: \displaystyle\text{PNS}(z,\bar{z})= \displaystyle P(Y=y\mid Z=z)-P(Y=y\mid Z=\bar{z}) Recent studies have applied PNS to guide representation learning in unimodal data [ 23 , 21 , 4 , 1 , 2 ] , but extending it to multimodal scenarios faces challenges in satisfying these two conditions. Exogeneity is violated since interactions across modalities can introduce hidden confounding effects without strong assumptions [ 23 , 21 , 17 , 15 ] . Without exogeneity, the interventional probabilities in definition 2 cannot be converted to conditional probabilities, making it intractable to establish monotonicity from observational data. To address these challenges, our work builds upon the assumption that multimodal data can be decoupled into two types of latent variables: one shared across modalities and the other conditioned on the modality type [ 3 , 20 , 11 , 13 ] . As illustrated in fig. 1 (a), the data generation process involves two latent factors. Let (X^{1},...,X^{N},Y) be a multimodal sample with N modalities, abbreviated as (X,Y) , where X^{M}\subset\mathbb{R}^{d_{M}} denotes the input from modality M and Y\subset\mathbb{R}^{d_{y}} denotes the outcome. A modality sample ( X^{M} , Y ) is generated from: a modality-invariant latent variable Z_{I}\subset\mathbb{R}^{d_{Z_{I}}} capturing shared information across modalities, and modality-specific latent variables Z_{S}\subset\mathbb{R}^{d_{Z_{S}}} conditioned on modality type M , encoding characteristics unique to each modality [ 3 , 20 , 11 , 13 ] . A decoupling (or disentanglement) \mathcal{D}(\cdot) model aims to capture information from these latent factors by extracting two types of representations from X^{M} : modality-invariant representations \mathcal{R}_{I}^{M}\subset\mathbb{R}^{d_{Z_{I}}} approximating Z_{I} , and modality-specific representations \mathcal{R}_{S}^{M}\subset\mathbb{R}^{d_{Z_{S}}} approximating Z_{S} conditioned on modality type M . A typical architecture of \mathcal{D}(\cdot) (shown in fig. 1 (b)) which consists of a feature extractor \Phi(\cdot) that decouples the representations [\mathcal{R}_{I}^{M},\mathcal{R}_{S}^{M}]:=\phi(X^{M}) , and three types of predictors: (1) modality-specific predictors F_{M}(\cdot) predicting Y from \mathcal{R}_{S}^{M} ; (2) an invariant predictor F_{I}(\cdot) predicting Y from \mathcal{R}_{I}^{M} ; (3) a joint predictor F_{P}(\cdot) predicting Y from all representations. We denote the overall model prediction as \hat{Y}=\mathcal{D}(X) . The training objective is: \mathcal{L}^{D}:=\mathcal{L}_{pred}+\mathcal{L}_{dec}+\textstyle\sum_{M=1}^{N}\left(\mathcal{L}_{I}^{M}+\mathcal{L}_{S}^{M}\right) (1) where \mathcal{L}_{I}^{M}:=\mathcal{L}_{I}^{M}(F_{I}(\mathcal{R}_{I}^{M}),Y) and \mathcal{L}_{S}^{M}:=\mathcal{L}_{S}^{M}(F_{M}(\mathcal{R}_{S}^{M}),Y) measure the prediction accuracy of \mathcal{R}_{I}^{M} and \mathcal{R}_{S}^{M} , respectively, while \mathcal{L}_{pred}:=\mathcal{L}_{pred}(\mathcal{D}(X),Y) is the prediction loss, and \mathcal{L}_{dec} includes decoupling and other constraints. The specific architectures of feature extractors, predictors, and loss functions vary across prior multimodal decoupling work. Figure 1: Overview of the proposed MPNS framework. The figure shows the multimodal data generation process, a baseline decoupling model, and our proposed complement branch with PNS-based optimization. 3 Method Following section 2 , we decompose representations into invariant and specific components, enabling separate PNS analysis for each. 3.1 Learning Complement Representations Computing PNS via lemma 1 requires feature pairs (z,\bar{z}) where z predicts correctly while \bar{z} does not. We achieve this by