Current Pathology Foundation Models are unrobust to Medical Center Differences
Edwin D. de Jong, Eric Marcus, Jonas Teuwen · 2025
Read Current Pathology Foundation Models are unrobust to Medical Center Differences by Edwin D. de Jong, Eric Marcus, Jonas Teuwen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18055.txt.
Introduction
Edwin de Jong, Eric Marcus, and Jonas Teuwen show that today's big pathology AI models quietly learn which hospital a slide came from, in a 2025 paper. Stain colors and lab habits leave a signature on tissue images, and a model that leans on it may fail at new hospitals. The team built a robustness index that weighs how much a model favors real biology over site signatures.
All ten public models tested carried a strong site signal. Only one scored above one, meaning biology barely won. Errors in cancer sorting were not random: slides were confused with other classes from the same center. Maps of the model spaces clustered by hospital more than by tissue type. The authors urge robustness checks before clinical use.
Catalog record
| CREATOR | Edwin D. de Jong, Eric Marcus, Jonas Teuwen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Current Pathology Foundation Models are unrobust to Medical Center Differences Edwin D. de Jong, Eric Marcus, Jonas Teuwen Abstract Pathology Foundation Models (FMs) hold great promise for healthcare. Before they can be used in clinical practice, it is essential to ensure they are robust to variations between medical centers. We measure whether pathology FMs focus on biological features like tissue and cancer type, or on the well known confounding medical center signatures introduced by staining procedure and other differences. We introduce the Robustness Index . This novel robustness metric reflects to what degree biological features dominate confounding features. Ten current publicly available pathology FMs are evaluated. We find that all current pathology foundation models evaluated represent the medical center to a strong degree. Significant differences in the robustness index are observed. Only one model so far has a robustness index greater than one, meaning biological features dominate confounding features, but only slightly. A quantitative approach to measure the influence of medical center differences on FM-based prediction performance is described. We analyze the impact of unrobustness on classification performance of downstream models, and find that cancer-type classification errors are not random, but specifically attributable to same-center confounders : images of other classes from the same medical center. We visualize FM embedding spaces, and find these are more strongly organized by medical centers than by biological factors. As a consequence, the medical center of origin is predicted more accurately than the tissue source and cancer type. The robustness index introduced here is provided with the aim of advancing progress towards clinical adoption of robust and reliable pathology FMs. 1 Introduction Pathology Foundation Models (FMs) have quickly become the dominant approach in current pathology AI. Following Campanella’s groundbreaking work that used weakly-supervised learning to scale up machine learning for pathology [ 1 ] and early papers applying Self-Supervised Learning (SSL) in the domain [ 2 , 3 , 4 ] , an impressive and quickly growing series of pathology FMs have become available, with more than ten new such models published last year so far alone: [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Several of these models demonstrate remarkable capabilities and can detect patterns that human pathologists struggle to observe from H&E slides, such as Microsatellite Instability (MSI) and Immunohistochemistry (IHC) biomarkers such as Ki67 and PD-L1 [ 19 , 20 , 8 ] . Pathology foundation models thus hold great potential for healthcare by aiding pathologists, for example in routine or large volume, labor intensive tasks. If this promise is to be realized, it is essential that models can be trusted to provide unbiased estimates of a patient’s condition. A particular obstacle to the adoption of pathology FMs in clinical practice is the sensitivity of machine learning models (ML) to staining variations, caused by differences in the staining procedures used by different labs, the staining fluids, and imaging equipment. It is well known that these image variations can influence pathology ML models, and reduce their ability to generalize to data from laboratories not seen during training [ 21 ] . Staining procedures vary per laboratory, and are thereby associated with specific medical centers. This leads to a clear risk of bias [ 22 ] : if models are sensitive to the medical center from where images originate, then patients from different medical centers will be evaluated differently by such models, possibly leading to different diagnoses and treatment based on irrelevant technical differences between images. To ensure foundation models can be safely introduced into healthcare practice, evaluating and confirming their robustness to image variations that occur in practice is a necessary step. To assess whether current pathology foundation models provide an objective assessment of a patient’s condition, we analyze to what extent medical centers influence the embedding spaces generated by FMs. Our contributions are as follows: • A basic yet effective description for the concept of robustness in medical ML is suggested, based on the distinction between biological features and confounding features. Models can vary in their robustness to image variations such as noise, color differences, augmentations, and variations between medical centers. • We introduce a novel robustness metric: the Robustness Index, measuring the degree to which biological features dominate confounding features in the neighborhood structure of the embedding space induced by the foundation model. • For the first time, a quantitative approach to measure the influence of medical center differences on FM-based prediction performance is described. The approach directly relates prediction errors to same-center confounders: images from the same center as the predicted sample that have a different class, and thereby contribute to incorrect classification. • 10 current pathology foundation models are evaluated on their medical center robustness. It is found that current pathology FMs vary widely in their robustness to medical center variations. • We suggest that the value of an FM is determined by the relation between (A) its ability to characterize relevant biological information, enabling high prediction performance of biological information in downstream tasks, and (B) its robustness, reflected in its insensitivity to irrelevant non-biological variation such as staining and medical center differences. • To gain further insight into the mappings learned by pathology FMs, we project embeddings to a 2D space for visualization using t-SNE [ 23 ] . We find that most foundation models show a clear clustering of medical centers in the embedding space; this shows more clearly than the clustering of biological classes. 2 Related Work In previous work [ 24 ] presented at AMLD 2024, we visualized the embedding space of ViT models trained on TCGA using DINO. It was found that a 2D t-SNE projection of the embedding space was clustered by medical center, forming the inspiration for the current work. In simultaneous work from the TU Berlin BIFOLD group [ 25 ] , batch effects in pathology foundation models were analyzed and shown. [ 26 ] analyzes rotation invariance. Tellez [ 21 ] quantified the effects of data augmentation and stain color normalization in pathology in the context of CNNs, and [ 27 ] evaluated the effect of stain normalization in colorectal tissue classification. [ 28 ] discusses the issue that hospitals are represented in WSI data, and looks into domain generalization for hospital-agnostic image representation learning; [ 29 ] also uses domain adaptation to reduce the influence of staining differences. [ 30 ] draws attention to the existence of medical center signatures in TCGA data specifically, and finds that deep neural networks can predict the acquisition site. A main consequence of unrobustness is that predictive models based on unrobust representations can lead to biased predictions. [ 31 ] studies the impact of medical center signatures on deep learning model accuracy and bias. [ 18 ] notes that embeddings from SSL models tend to cluster by individual WSIs and proposes the EXAONEPath pathology FM to address this, which is included in our evaluation here. While finalizing this paper, a new relevant article discussing the measurement and optimization of robustness in pathology became available [ 32 ] . 3 Robustness for Medical Foundation Models To clarify what is meant by