SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome

Dabin Jeong, Amirhossein Vahidi, Ciro Ramírez-Suástegui, Marie Moullet, Kevin Ly, Mohammad Vali Sanian, Sebastian Birk, Yinshui Chang, Adam Boxall, Daniyal Jafree, Lloyd Steele, Vijaya Baskar MS, Muzlifah Haniffa , Mohammad Lotfollahi Wellcome Sanger Institute, Wellcome Genome Campus, Cambridge, UK · 2025

Read SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome by Dabin Jeong, Amirhossein Vahidi, Ciro Ramírez-Suástegui, Marie Moullet, Kevin Ly, Mohammad Vali Sanian, Sebastian Birk, Yinshui Chang, Adam Boxall, Daniyal Jafree, Lloyd Steele, Vijaya Baskar MS, Muzlifah Haniffa , Mohammad Lotfollahi Wellcome Sanger Institute, Wellcome Genome Campus, Cambridge, UK (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15464.txt.

Introduction

Dabin Jeong, Amirhossein Vahidi, and twelve co-authors present SIGMMA, a method that learns from two views of the same tissue at once. One view is a stained tissue image, the standard microscope slide used in pathology. The other is a spatial gene map that shows which genes are active at each spot. Earlier methods matched the two views at only one scale.

SIGMMA aligns the views at several scales and models cells as a graph, so links between nearby cells shape what is learned. That design captures fine cell detail and broad tissue layout together. Across data sets it improved gene activity prediction and matching between the two views by clear margins over prior methods.

Catalog record

CREATORDabin Jeong, Amirhossein Vahidi, Ciro Ramírez-Suástegui, Marie Moullet, Kevin Ly, Mohammad Vali Sanian, Sebastian Birk, Yinshui Chang, Adam Boxall, Daniyal Jafree, Lloyd Steele, Vijaya Baskar MS, Muzlifah Haniffa , Mohammad Lotfollahi Wellcome Sanger Institute, Wellcome Genome Campus, Cambridge, UK
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15464

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome Dabin Jeong, Amirhossein Vahidi, Ciro Ramírez-Suástegui, Marie Moullet, Kevin Ly, Mohammad Vali Sanian, Sebastian Birk, Yinshui Chang, Adam Boxall, Daniyal Jafree, Lloyd Steele, Vijaya Baskar MS, Muzlifah Haniffa , Mohammad Lotfollahi Wellcome Sanger Institute, Wellcome Genome Campus, Cambridge, UK Cambridge Centre for AI in Medicine, University of Cambridge, Cambridge, UK Institute of AI for Health, Helmholtz Center Munich, Neuherberg, Germany Cambridge Stem Cell Institute, University of Cambridge, Cambridge, UK {mh32,ml19}@sanger.ac.uk Abstract Recent advances in computational pathology have leveraged vision–language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma , a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell–cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmma learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78% in the gene-expression prediction task and avg. 26.93% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses. 1 Introduction Figure 1 : Motivation. (a) Limitations of previous vision-language model-based HE-ST alignment. (b) How Sigmma addresses these limitations by (1) multi-scale alignment and (2) adopting a cell graph structure that preserves 2D coordinates and cell-cell relationships. (c) Sigmma captures multi-scale information, with ST representations of each cell becoming more heterogeneous at larger scales. CC,connected component. Tissue architecture is hierarchically organized across multiple spatial scales, from the micro environment (Fig. 1 blue) comprising small clusters of interacting cells within local regions, to the meso environment (Fig. 1 green) encompassing cellular neighborhoods of dozens of cells, and up to the macro environment (Fig. 1 red) characterized by macroscopic structures, e.g., tertiary lymphoid structures ( 36 ) . Understanding these hierarchical contexts requires both morphological and molecular views of tissue. Computational histopathology has been advanced by large-scale analysis of cellular morphology, tissue composition, and spatial organization, most commonly Hematoxylin Eosin (HE) images. In parallel, single-cell spatial transcriptomics (ST) enables the molecular profiling of individual cells with 2D spatial coordinates, providing direct links between morphology and gene expression profiles. Together, these technologies allow researchers to map how individual cell types, cell-cell interactions, and spatial organization of cells contribute to physiological and pathological processes ( 43 ) . Multi-modal learning of HE images with ST profiles provides a unified representation of tissue morphology and molecular state, enabling the identification of molecular heterogeneity that is not discernible from HE images alone (Fig. 1 c). Why do we need a graph structure for ST modeling? Recent contrastive learning approaches learn joint representations of HE images and ST in a shared space to support cross-modal tasks, including image-to-expression retrieval and image-to-expression prediction. Notably, vision–language (VL) models originally developed for HE–biomedical text (e.g., caption, scientific papers, clinical notes) have been adapted for ST ( 8 ; 13 ; 44 ) . However, current VL-based HE–ST contrastive alignment approaches represent ST as a 1D gene sequence aggregated across cells ( 8 ; 13 ; 44 ) , thereby inevitably discarding the original 2D spatial organization and cell–cell interactions within the tissue (Fig. 1 a, bottom). In contrast, graph representations inherently encode spatial topology and relational structure, enabling explicit modeling of cell–cell interactions and the surrounding tissue context (Fig. 1 b bottom). Why do we need hierarchical multi-scale HE-ST contrastive alignment? The hierarchical organization of tissue makes multi-scale HE–ST alignment inherently challenging. Multi-scale alignment implicitly requires correspondence across region-of-interest (ROI) granularities. Embeddings learned from contrastive loss (e.g., InfoNCE loss) maximize the lower bound of mutual information between two pairs ( 28 ) , which tends to emphasize salient features at the expense of finer details ( 42 ) . Specifically, graph-structured ST data complicates multi-scale alignment because message passing expands the receptive field based on graph connectivity rather than image ROI scales, leading to mismatched spatial scopes across modalities. To overcome this limitation, multi-scale contrast alignment methods incorporate multiple ROI sizes, thus capturing both coarse- and fine-grained tissue features. (Fig. 1 b). Our contribution. We propose Sigmma , a hierarchical, graph-based, multi-scale alignment framework for HE–ST. • Graph-structured representation of ST. We represent ST as a cell graph that preserves spatial topology and the structure of cell-cell relationships. A hierarchical graph module integrates intra- and inter-subgraph relationships to capture local neighborhoods and long-range dependencies that are lost in sequence-based ST representations. • Multi-scale cross-modal alignment. We introduce a multi-scale HE–ST alignment framework that enforces alignment consistency across multiple spatial resolutions. Our multi-scale contrastive objective aligns representations from micro, meso, and macro contexts, improving fine-grained and coarse-grained correspondence. • ST graph–HE image scale reconciliation. We progressively expand the graph receptive field through hierarchical graph learning, matching it to the image ROI size and enabling consistent correspondence between modalities across scales. • Performance improvements and interpretability. Sigmma yields improvements in downstream tasks, including gene-expression prediction and image–expression retrieval, across five datasets and produces embeddings that reveal biologically meaningful tissue organization. 2 Related Work ST at single-cell resolution. ST has emerged as a powerful approach to map gene expression within the spatial context of tissues. Specifically, it measures gene expression together with 2D spatial coordinates, indicating the location and level of expression of specific genes. There are two main techniques for measuring ST: Visium ( 39 ) and Xenium ( 21 ) . Visium is a sequencing-based platform which captures transcriptomic signals at the spot level, where each spot typically aggregates the expression profiles of multiple neighboring cells. In contrast, the Xenium platform utilizes high-resolution in situ hybridization and imaging to measure gene expression at the cellular/subcellular levels, offering deeper insights into cell–cell interactions. In this work, we use Xenium rather than Visium because Xenium provides cell-level spatial transcriptomics, enabling alignment with HE images while explicitly modeling each cell’s 2D spatial context. Tiling of HE WSI image. Whole-slide images (WSIs) are gigapixel-scale, making direct application of vision models computationally prohibitive and forcing heavy down-sampling that removes critical cellular-level signals ( 18 ) . Since discriminative patterns are small, sparse, and spatially scattered, tile-level modeling enables vision models to learn high-resolution local features by training on small image tiles, and leads to WSI-level tasks by aggregating tile-level embeddings ( 2 ; 34 ; 7 ; 24 ; 23 ) . As molecular phenotypes and cellular contexts vary across localized regions, tile-level alignment can provide a more fine-grained correspondence between image features and transcriptomic signals than slide-level alignment. Motivated by this, our work focuses on tile-level alignment between HE and ST features, enabling cross-modal learning at a spatially-resolved and fine-grained level. Foundation models for HE and ST. Foundation models have recently emerged in computational histopathology for both HE images and ST. For HE image, DINO ( 29 ) -based vision foundation models enable scalable learning of

Read the complete text