Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology
Arbel Hizmi, Artemii Bakulin, Shai Bagon, Nir Yosef · 2026
Read Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology by Arbel Hizmi, Artemii Bakulin, Shai Bagon, Nir Yosef (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09076.txt.
Introduction
Hizmi and three colleagues transfer knowledge from spatial transcriptomics to histology slides. That method reveals tissue niches, tight regions of distinct cell types and functions that matter for biology and clinics, but it is costly and scarce. Slides stained with H&E are common yet coarser. Training on paired data, their system beats shape-only rivals on the same image cues at matching niche structure. It also brings back local mix checked by cell-type analysis. It then applies to new regions using slides alone, with no such input.
Catalog record
| CREATOR | Arbel Hizmi, Artemii Bakulin, Shai Bagon, Nir Yosef |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33766 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Modal Knowledge Distillation from Spatial Transcriptomics to Histology Arbel Hizmi, Artemii Bakulin, Shai Bagon, Nir Yosef Abstract Spatial transcriptomics provides a molecularly rich description of tissue organization, enabling unsupervised discovery of tissue niches — spatially coherent regions of distinct cell-type composition and function that are relevant to both biological research and clinical interpretation. However, spatial transcriptomics remains costly and scarce, while H&E histology is abundant but carries a less granular signal. We propose to leverage paired spatial transcriptomics and H&E data to transfer transcriptomics-derived niche structure to a histology-only model via cross-modal distillation. Across multiple tissue types and disease contexts, the distilled model achieves substantially higher agreement with transcriptomics-derived niche structure than unsupervised morphology-based baselines trained on identical image features, and recovers biologically meaningful neighborhood composition as confirmed by cell-type analysis. The resulting framework leverages paired spatial transcriptomic and H&E data during training, and can then be applied to held-out tissue regions using histology alone, without any transcriptomic input at inference time. Figure 1 : Framework overview : Our goal is to transfer spatially resolved molecular niche structure from spatial transcriptomics to histology, so that niche organization can be predicted from H&E alone at inference time. The key idea is that transcriptomics provides a richer view of local tissue state, while histology is abundant and widely available. During training, we therefore use a frozen spatial-transcriptomics teacher to supervise a histology student on matched cell neighborhoods. For each cell, both teacher and student operate on spatial neighborhoods: the student takes UNIv2 features from H&E cell crops, while the teacher takes scVI features from the paired transcriptomic measurements. The frozen pretrained NOLAN teacher produces niche-assignment logits, and the student is trained with a distillation loss to match them. At inference, only the histology branch is used, and the student predicts niche labels from H&E neighborhoods alone. Flame symbol: “trainable”; Snowflake symbol: “frozen”. 1 Introduction Many tissue phenomena — such as immune infiltration, stromal remodeling, and tumor progression — are not properties of isolated cells, but emerge from the spatial organization and interactions of multiple cell types. A central goal in tissue analysis is therefore to identify spatially coherent regions that reflect characteristic local cell compositions and interactions, commonly referred to as cellular niches . The task of partitioning tissue into such regions is known as tissue niche segmentation . While traditional approaches to tissue structure analysis rely on manual annotation, recent work has shown that spatial transcriptomics provides a rich substrate for unsupervised niche discovery. Spatial transcriptomics jointly measures gene expression and spatial coordinates across thousands of cells, providing a high-dimensional molecular fingerprint of the tissue microenvironment. Methods such as BANKSY, SpaGCN, MENDER, and NOLAN Singhal et al. 2024 ; Hu et al. 2021 ; Yuan 2024 ; Bakulin et al. 2025 exploit this richness to define niches directly in molecular feature space, incorporating spatial neighborhood structure to produce coherent, biologically interpretable tissue partitions. However, spatial transcriptomics assays remain costly and are often available only for a subset of samples or tissue regions, limiting their applicability at scale. In contrast, hematoxylin and eosin (H&E) histology is universally available across both research and clinical workflows, with archives spanning millions of slides. Yet H&E does not directly measure gene expression, and morphological features alone may not faithfully recover molecularly defined tissue organization. Recent foundation models have dramatically improved H&E representation learning Chen et al. 2024 ; Xu et al. 2024 ; Vorontsov et al. 2024 ; Saillard et al. 2024 , but even rich morphological embeddings are not guaranteed to capture the molecular distinctions that define transcriptomic niches — particularly in regions where cell-type differences are subtle or where niche identity is determined by microenvironmental context rather than cell morphology alone. We address the gap between abundant H&E imaging and the molecular information available in spatial transcriptomics as a cross-modal distillation problem. When paired spatial transcriptomics and H&E data are available for the same cells, spatial transcriptomics provides a richer molecular view of cell state and neighborhood composition than histology alone. We use the spatial transcriptomics modality to define a reference niche structure with a transcriptomic teacher, and train a histology-based student to match the teacher’s niche predictions from cell-level image embeddings and their spatial neighborhoods. This encourages the student to learn morphology signals that are informative of transcriptomic niches. At inference time, the student uses histology features only, enabling niche prediction without molecular measurements. We evaluate (i) how well the histology student reproduces the spatial transcriptomics-derived reference structure and (ii) whether the predicted niches preserve biologically meaningful organization, including cell-type composition patterns and (iii) correspondence of predicted niches with pathology annotations of tissue areas. 2 Related work 2.1 Spatial transcriptomics and cellular niches Many important tissue functions, such as tumor growth and immune infiltration, are governed by the spatial arrangement of cells rather than the characteristics of individual cells. Several methods have been proposed to identify spatial domains or cellular niches by combining gene expression embeddings with spatial relationships Singhal et al. 2024 ; Hu et al. 2021 ; Yuan 2024 . A common implicit assumption in existing methods is that tissue can be partitioned into discrete regions. However, Bhate et al . Bhate et al. 2022 show that tissue organization is often more complex: tissue regions may lack sharp boundaries, and tissue properties can change gradually across space. As a result, precise niche labels are difficult to define and are often ambiguous. Bakulin et al . Bakulin et al. 2025 further argue that manual annotations can be low-resolution or inaccurate under continuous spatial gradients, motivating unsupervised methods. Given the ambiguity of manual niche annotations, we use NOLAN Bakulin et al. 2025 as our transcriptomics teacher. NOLAN is a self-supervised model that learns niche-aware representations from spatial neighborhoods, enabling the discovery of spatially coherent molecular niches without manual annotations. Because NOLAN computes representations from neighborhood-based inputs, the resulting embeddings depend on local cellular context rather than only on individual-cell features. This property is important for our setting, where niches are defined by local spatial context. Accordingly, our method follows NOLAN’s neighborhood-based formulation by using local spatial neighborhoods to learn niche-related representations. 2.2 Computational pathology Deep learning has become central to computational pathology, with recent advances increasingly driven by self-supervised learning and foundation models for H&E histology. In particular, self-supervised objectives trained on large collections of unlabeled histology images have enabled the development of domain-specific histopathology encoders Chen et al. 2024 ; Xu et al. 2024 ; Vorontsov et al. 2024 ; Saillard et al. 2024 . These models learn transferable visual representations that capture tissue morphology across organs and staining variability, and serve as strong backbones for a broad range of downstream tasks. Prior work often evaluates such representations on tasks such as classification and survival prediction. In this work, we use UNIv2 Chen et al. 2024 as the histology feature encoder. UNIv2 provides general-purpose histology embeddings with strong performance across diverse clinical tasks Chen et al. 2024 . However, while histology captures rich morphological structure, it does not directly measure molecular state; consequently, tissue niche segmentation from histology alone may not align with molecularly defined organization. To address this gap, we use a transcriptomics teacher and a histology student (initialized from UNIv2),