Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining

Yucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang, Mengling Feng, [6pt] National University of Singapore, Singapore · 2026

Read Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining by Yucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang, Mengling Feng, [6pt] National University of Singapore, Singapore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16392.txt.

Introduction

Xing and eight others present MIST, a plug-in replacement for the projection layer in multiple instance learning for whole-slide pathology images. Standard projection layers rely on morphology alone, limiting endpoints like biomarker status and survival that depend on molecular state not fully visible in H&E stains. MIST uses paired spatial transcriptomics during training only to build virtual molecular stains, clustering gene expression into cross-modal prototypes anchored in frozen feature space. It needs no transcriptomics at inference and improves 240 of 256 configurations, with an average gain of 3.5 percent.

Catalog record

CREATORYucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang, Mengling Feng, [6pt] National University of Singapore, Singapore
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65773
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining Yucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang, Mengling Feng, [6pt] National University of Singapore, Singapore Abstract Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone. 1 Introduction Computational pathology (CPath) increasingly relies on whole-slide image (WSI) analysis for tissue subtyping, biomarker prediction, and prognosis 8 ; 20 ; 7 . Because WSIs are gigapixel-scale and heterogeneous, multiple instance learning (MIL) has become the dominant paradigm 3 . A slide is split into patches, each embedded by a frozen pathology foundation model (FM), and the resulting patch set is aggregated into a slide-level prediction. This pipeline has been effective, but it exposes a fundamental mismatch. Many clinical endpoints, including biomarker status, treatment response, and survival, are governed by molecular state, whereas frozen pathology FMs are pretrained mainly from H&E morphology and pathology reports 13 . They organize tissue along morphological axes. Histologically similar regions can correspond to distinct molecular programs (Fig. 1 A), suggesting that morphology-only MIL faces not only a capacity bottleneck, but also a modality bottleneck . We address this bottleneck at the projection layer. A standard MIL pipeline contains a frozen patch encoder, a per-patch projection layer, and a slide-level aggregator 28 . Prior work has extensively improved encoders through self-supervised and vision-language pretraining 6 ; 23 , and aggregators through attention, clustering, and low-rank structure 29 ; 24 ; 41 . The projection layer is a strategically important intervention point because it can reshape patch features while preserving both the frozen encoder and the downstream aggregator. Existing projection-layer redesigns still operate inside the FM’s morphological feature space 33 . They can better use morphology, but they cannot introduce molecular axes that the encoder never observed during pretraining. Pathologists routinely overcome this limitation by adding molecularly targeted stains. When H&E morphology alone cannot resolve a diagnostic or prognostic question, immunohistochemistry (IHC) allows the same tissue to be interpreted under an additional molecular label space 25 . Inspired by this role of staining, we introduce Molecularly Informed Staining Transform (MIST) for MIL. MIST converts paired spatial transcriptomics (ST), available only during training, into virtual molecular stains that reorganize H&E patch features along gene-defined axes. It leaves the pathology FM frozen, requires no transcriptomics at inference, and can be inserted before standard MIL aggregators. Paired ST is a natural source of molecular supervision because it co-registers H&E morphology with spot-level gene expression 18 . Existing training-only ST approaches usually inject this signal at the encoder stage by fine-tuning or re-pretraining a pathology FM with vision-omics objectives 12 ; 39 . This validates the value of molecular supervision, but ties the resulting representation to a specific FM checkpoint and requires costly retraining when new pathology FMs are adopted. MIST instead moves molecular supervision one stage later. By operating at the projection layer, it provides a reusable molecularly informed adaptation module without modifying the frozen encoder. MIST is a form of cross-modal adaptation for frozen foundation-model pipelines. Instead of requiring the auxiliary modality at inference, MIST uses it during training to define a molecularly grounded coordinate system. The downstream MIL objective then learns how to use this coordinate system for each endpoint. Figure 1: (A) MIST separates patches with similar morphology but distinct molecular profiles. (B) Mean performance gains across downstream tasks for eight MIL aggregators. MIST realizes feature-level virtual molecular staining in three steps. First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space, allowing each prototype to encode the morphological signature of a distinct molecular concept. Second, each patch is softly associated with multiple prototypes through affinities, reflecting the fact that local tissue regions often express overlapping molecular programs. Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept, which are combined according to the affinity weights and added back to the original patch embedding, preserving the underlying H&E morphology as a stable reference for downstream MIL. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators using a frozen encoder. MIST improves 240 of 256 configurations over the standard projection layer, with consistent gains across endpoint types, including survival prediction (+5.2%), tissue subtyping (+3.3%), and biomarker prediction (+2.6%), yielding an overall average improvement of +3.5%. Ablation studies further show that replacing gene-derived prototypes with random or morphology-derived alternatives removes much of the gain, suggesting that the improvement primarily arises from cross-modal molecular priors rather than additional morphology-only modeling capacity. Our contributions are as follows: • A modality-bottleneck view of MIL projection layers. We identify the projection layer as a modular but modality-limited stage in frozen-FM pathology MIL, and show that molecular priors provide an effective intervention beyond morphology-only projection capacity. • Virtual molecular staining through cross-modal prototypes. We introduce a projection-layer mechanism that injects spatial transcriptomic priors during training while requiring only H&E at inference, enabling molecularly informed adaptation without retraining the pathology FM. • Broad empirical evidence and molecular interpretability. Across 23 tasks and 8 MIL aggregators, MIST improves most evaluated configurations (Fig. 1 B), while spatial, biological, and pathological analyses indicate that cross-modal prototype affinities capture spatially coherent and pathologically meaningful molecular programs. 2 Related Work MIL for Computational Pathology. The standard MIL pipeline for gigapixel WSI analysis decomposes into three stages: patch feature extraction, per-patch linear projection, and aggregation into a slide representation. Existing work focuses on the first and third stages. At the encoder level, pathology foundation models (FMs) have evolved along two directions distinguished by pretraining supervision: vision-only FMs learn patch representations through self-supervision on unlabeled H&E 6 ; 35 ; 44 ; 5 , while vision-language FMs additionally align H&E with pathology reports through contrastive objectives 23 ; 40 ; 9 . At the aggregation level, MIL methods pool patch sets through inductive biases such as attention 17 , clustering-constrained attention with instance supervision 24 , dual-stream max-pooling 22 , correlated self-attention 29

Read the complete text