Comparing Computational Pathology Foundation Models using Representational Similarity Analysis
Vaibhav Mishra, William Lotter · 2025
Read Comparing Computational Pathology Foundation Models using Representational Similarity Analysis by Vaibhav Mishra, William Lotter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15482.txt.
Introduction
Vaibhav Mishra and William Lotter compare six large models built for pathology, the study of tissue under a microscope. Such models promise help on many medical tasks. Most past checks graded what the models can do, not how they store what they learn. The pair look inside, using tools first made for brain science.
They feed the models stained tissue patches from a cancer atlas and compare their inner patterns. Two models stand apart as the most distinct, while a third sits closest to the rest on average. Models trained the same way did not always match inside. All models leaned hard on which slide a patch came from, and stain fixes lowered that lean.
Catalog record
| CREATOR | Vaibhav Mishra, William Lotter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15482
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comparing Computational Pathology Foundation Models using Representational Similarity Analysis Vaibhav Mishra, William Lotter Abstract Foundation models are increasingly developed in computational pathology (CPath) given their promise in facilitating many downstream tasks. While recent studies have evaluated task performance across models, less is known about the structure and variability of their learned representations. Here, we systematically analyze the representational spaces of six CPath foundation models using techniques popularized in computational neuroscience. The models analyzed span vision-language contrastive learning (CONCH, PLIP, KEEP) and self-distillation (UNI (v2), Virchow (v2), Prov-GigaPath) approaches. Through representational similarity analysis using H&E image patches from TCGA, we find that UNI2 and Virchow2 have the most distinct representational structures, whereas Prov-Gigapath has the highest average similarity across models. Having the same training paradigm (vision-only vs. vision-language) did not guarantee higher representational similarity. The representations of all models showed a high slide-dependence, but relatively low disease-dependence. Stain normalization decreased slide-dependence for all models by a range of 5.5% (CONCH) to 20.5% (PLIP). In terms of intrinsic dimensionality, vision-language models demonstrated relatively compact representations, compared to the more distributed representations of vision-only models. These findings highlight opportunities to improve robustness to slide-specific features, inform model ensembling strategies, and provide insights into how training paradigms shape model representations. Our framework is extendable across medical imaging domains, where probing the internal representations of foundation models can help ensure effective development and deployment. 1 Introduction Numerous foundation models have recently been developed for computational pathology. Trained on large-scale datasets in a self-supervised fashion, these models are often designed to generate patch-level embeddings from H&E-stained whole slide images (WSIs), which have proven valuable for downstream tasks such as tumor subtyping, biomarker prediction, and prognostication ( Bilal et al. 2025 ; Li et al. 2025 ; Neidlinger et al. 2024 ) . However, recent analyses have also raised questions regarding the robustness of these models to data-specific confounders, such as hospital-specific signatures ( de Jong et al. 2025 ; Kömen et al. 2024 ) . Furthermore, while studies have compared downstream performance across models, the extent to which these models produce similar or divergent embeddings remains unclear. Given the emphasis of interpretability and robustness in medical AI, there is a pressing need to move beyond performance metrics and systematically examine the representations learned by foundation models in computational pathology (CPath). Here, we leverage techniques from computational neuroscience to explore the representational spaces of six CPath foundation models. Using representational similarity analysis (RSA) ( Kriegeskorte et al. 2008 ) , we examine how H&E-patch embeddings vary across models, diseases, and individual WSIs. The models analyzed represent the two predominant self-supervised learning strategies in the field: vision-language contrastive learning and vision-only self-distillation. As a comparative baseline, we include a DinoV2 model ( Oquab et al. 2024 ) trained on natural images. We aimed to quantify similarities and differences between these models, including characterizing the slide- and disease-dependence of their representations, as well as their intrinsic dimensionality. 2 Related Work 2.1 Foundation models in computational pathology Following strategies developed for natural images, the two most predominant paradigms for developing CPath foundation models are vision-language contrastive learning and vision-only self-distillation ( Bilal et al. 2025 ; Li et al. 2025 ) . Vision-language contrastive learning, pioneered by CLIP ( Radford et al. 2021 ) , aims to learn joint representations between images and text such that matching image-text pairs have similar embeddings and non-matches have dissimilar embeddings. Vision-language models developed for CPath have used various sources to curate image-text pairs, including Twitter (e.g. PLIP ( Huang et al. 2023 ) ), Youtube (e.g., Quilt-1M ( Ikezogwo et al. 2023 ) ), and PubMed (e.g., CONCH ( Lu et al. 2024 ) ). KEEP extended traditional vision-language contrastive learning by creating a knowledge graph to enable hierarchical semantic alignment ( Zhou et al. 2024 ) . For self-distillation, the most commonly adopted algorithm is DinoV2 ( Oquab et al. 2024 ) , which combines contrastive data augmentation and masking in a student-teacher framework. This learning paradigm aims to learn representations that capture meaningful features of the underlying data that are stable under transformations. CPath models developed using DinoV2 include UNI ( Chen et al. 2024 ) , Prov-GigaPath ( Xu et al. 2024 ) , and Virchow ( Vorontsov et al. 2024 ; Zimmermann et al. 2024 ) . 2.2 Analysis of CPath foundation models Several studies have evaluated the performance of CPath foundation models on downstream tasks such as disease classification, biomarker prediction, and prognostication ( Wölflein et al. 2024 ; Bilal et al. 2025 ; Neidlinger et al. 2024 ; Bareja et al. 2025 ; Marza et al. 2025 ) . In terms of representation analysis, Kömen et al. 2024 demonstrated that the originating data site can be reliably predicted from the embeddings of several vision-only CPath foundation models, a finding also supported by de Jong et al. 2025 . 2.3 Representational similarity analysis Representational similarity analysis (RSA) is a framework often used in computational neuroscience to compare neural representations across different conditions, such as comparing the responses of neurons to different stimuli ( Kriegeskorte et al. 2008 ) . The core of RSA involves constructing representational dissimilarity matrices (RDMs), which capture pairwise distances between representations across conditions. The RDMs for different systems (e.g., brain regions) can then also be compared to offer insights into similarities and differences between representational structures. 3 Methods 3.1 Models Studied We selected six popular foundation models for analysis (Table 1 ), consisting of three vision-only self-distillation models (UNI (v2), Virchow (v2), Prov-Gigapath) and three vision-language models (CONCH, PLIP, KEEP). We additionally include a Dinov2 model trained on natural images as a baseline ( Oquab et al. 2024 ) . UNI was trained on over 200 million image patches from over 350k slides sourced from Mass General Brigham ( Chen et al. 2024 ) . Virchow was trained on 3.1 million WSIs from the Memorial Sloan Kettering Cancer Center ( Zimmermann et al. 2024 ) . For both UNI and Virchow, we use the version 2 models. Prov-GigaPath was trained on 1.3 billion image patches from 171,189 WSIs from the Providence health system ( Xu et al. 2024 ) . CONCH was trained on over 1.17 million image–caption pairs curated from various sources, including PubMed ( Lu et al. 2024 ) . PLIP was trained on 208,414 image–caption pairs curated from public sources (OpenPath), including Twitter ( Huang et al. 2023 ) . KEEP ( Zhou et al. 2024 ) was trained using OpenPath and Quilt-1M ( Ikezogwo et al. 2023 ) , a dataset of 1M image–caption pairs curated from public sources, including Youtube. All of the studied models use a form of vision transformer (ViT) for the embedder architecture (Table 1 ). Table 1: Summary of models studied. Model Pretraining Architecture Training Data Source UNI Vision ViT-H Mass General Brigham Virchow ViT-H MSKCC Prov-GigaPath ViT-G Providence CONCH Vision-Language ViT-B Mixed PLIP ViT-B OpenPath KEEP ViT-L Quilt-1M, OpenPath Dinov2 Vision (Baseline) ViT-B Natural images 3.2 Dataset We leverage four subsets of TCGA to perform the analysis: breast invasive carcinoma (BRCA), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), and colon adenocarcinoma (COAD) ( Cancer Genome Atlas Research Network et al. 2013 ) .