Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification

Conghao Xiong, Zhengrui Guo, Zhe Xu, Yifei Zhang, Raymond Kai-yu Tong, Si Yong Yeo, Hao Chen, Joseph J. Y. Sung, Irwin King · 2025

Read Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification by Conghao Xiong, Zhengrui Guo, Zhe Xu, Yifei Zhang, Raymond Kai-yu Tong, Si Yong Yeo, Hao Chen, Joseph J. Y. Sung, Irwin King (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15504.txt.

Introduction

Classifying whole slide pathology images from only a few examples tends to overfit, and Conghao Xiong and co-authors argue the deeper cause is geometric. Features from pathology foundation models lie on a crumpled low-dimensional surface, and standard model layers, blind to that shape, can warp it. The team proposes a plug-in part called the Manifold Residual block.

The block splits each layer into two paths. One is a fixed random matrix that anchors the geometry and sharpens the feature spectrum, and the other is a small trainable path that learns only the leftover task-specific change. Tests reach top accuracy with far fewer parameters than rival methods.

Catalog record

CREATORConghao Xiong, Zhengrui Guo, Zhe Xu, Yifei Zhang, Raymond Kai-yu Tong, Si Yong Yeo, Hao Chen, Joseph J. Y. Sung, Irwin King
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification Conghao Xiong, Zhengrui Guo, Zhe Xu, Yifei Zhang, Raymond Kai-yu Tong, Si Yong Yeo, Hao Chen, Joseph J. Y. Sung, Irwin King Abstract Few-shot Whole Slide Image (WSI) classification is severely hampered by overfitting. We argue that this is not merely a data-scarcity issue but a fundamentally geometric problem. Grounded in the manifold hypothesis, our analysis shows that features from pathology foundation models exhibit a low-dimensional manifold geometry that is easily perturbed by downstream models. This insight reveals a key potential issue in downstream multiple instance learning models: linear layers are geometry-agnostic and, as we show empirically, can distort the manifold geometry of the features. To address this, we propose the Manifold Residual (MR) block, a plug-and-play module that is explicitly geometry-aware. The MR block reframes the linear layer as residual learning and decouples it into two pathways: (1) a fixed, random matrix serving as a geometric anchor that approximately preserves topology while also acting as a spectral shaper to sharpen the feature spectrum; and (2) a trainable, low-rank residual pathway that acts as a residual learner for task-specific adaptation, with its structural bottleneck explicitly mirroring the low effective rank of the features. This decoupling imposes a structured inductive bias and reduces learning to a simpler residual fitting task. Through extensive experiments, we demonstrate that our approach achieves state-of-the-art results with significantly fewer parameters, offering a new paradigm for few-shot WSI classification. Code is available in https://github.com/BearCleverProud/MR-Block . 1 Introduction Histopathology is the gold standard for disease diagnosis ( Chen et al., 2024 ; Guo et al., 2024 ; Xiong et al., 2024a ; Xiong et al., 2024b ) , and its computational analysis via Whole Slide Images (WSIs) operates under a uniquely challenging paradigm defined by two constraints. First, the gigapixel nature of WSIs makes Multiple Instance Learning (MIL) the de facto setting, where each slide is represented as a bag of patch features ( Ilse et al., 2018 ; Li et al., 2021 ; Zhang et al., 2022 ) . Second, expert-intensive annotation limits supervision, yielding datasets that are both small (few-shot) and weakly labeled, typically with only slide-level labels ( Lu et al., 2021 ) . This few-shot, weakly supervised setting forces models to identify discriminative patterns from thousands of unlabeled instances using only a handful of slide-level labels ( Xiong et al., 2023 ; Guo et al., 2025b ) , often leading to severe overfitting. To understand the root cause of overfitting, we look beyond the learning algorithm and into the intrinsic structure of the features themselves . Grounded in the manifold hypothesis ( Tenenbaum et al., 2000 ) , we investigate the intrinsic geometry of Camelyon16 ( Litjens et al., 2018 ) under different feature extractors, including CONCH ( Lu et al., 2024 ) , UNI ( Chen et al., 2024 ) , and ResNet-50 ( He et al., 2016 ) (UNI and ResNet-50 results are reported in Appendix B ), and present multi-faceted evidence that these representations lie on a low-dimensional, nonlinear manifold. First, spectral analysis ( Skean et al., 2025 ) reveals a low effective rank of 29.7 ( vs . the 512-dimensional CONCH features), confirming low dimensionality ( Fig. 1 (a)). Second, t-SNE visualizations reveal a distinct cluster topology ( Fig. 1 (b)). Finally, and most critically, tangent space analysis shows a non-flat, distance-dependent growth of geometric drift ( Fig. 1 (d), Original), providing quantitative evidence of intrinsic manifold curvature, thus excluding the pure linear subspace hypothesis. Therefore, we posit that one important contributing factor to overfitting is geometric: while features from pathology foundation models exhibit a low-dimensional manifold geometry, existing MIL models often fail to preserve this fragile structure . This is not incidental but a limitation rooted in their ubiquitous building block: the linear layer. Across MIL models, e.g . ABMIL ( Ilse et al., 2018 ; Lu et al., 2021 ; Li et al., 2021 ) , Transformers ( Shao et al., 2021 ; Xiong et al., 2023 ; Tang et al., 2024 ) , graph-based models ( Chen et al., 2021 ; Li et al., 2024b ) , information-bottleneck ( Zhang et al., 2024 ; Huang et al., 2024 ) and density-based ( Zhu et al., 2024 ) approaches, the linear layer serves as the indispensable component. However, the linear layer is inherently geometry-agnostic, lacking an architectural bias to respect the manifold structure. Our tangent space analysis provides direct, empirical evidence of this destructive tendency: as shown in Fig. 1 (d), a trained linear layer (Vanilla Linear in the figure) severely distorts the intrinsic geometry of the manifold compared to the original (blue line). Consequently, especially under the few-shot setting, these linear layers are prone to learning an overly complex mapping that does not respect the low-rank nature of features and distorts their structure, thereby discarding the geometric priors that the pathology foundation models learned during large-scale pretraining. This points to a fundamental research gap: a lack of mechanisms designed to actively preserve and leverage this crucial, yet fragile, manifold structure. Figure 1: Feature geometric analysis reveals a low-dimensional manifold, which linear layers distort and our method preserves. (a) Spectral analysis confirms low-dimensionality. (b, c) t-SNE shows a cluster topology, and a random transformation preserves it. (d) Tangent space analysis visualizes this preservation: our MR block maintains the manifold curvature, while a linear layer causes distortion. To bridge this research gap, we propose a plug-and-play Manifold Residual (MR) block. The MR block aims to better preserve the geometric structure of the pretrained features from the Pathology Foundation Models (PFMs). It reframes the feature mapping as residual learning and decouples it into two parallel paths: a frozen base path for geometric preservation and a trainable Low-rank Residual Path (LRP) for task-specific adaptation. The base path employs a fixed random matrix with Kaiming uniform initialization that serves as both a geometric anchor and a spectral sharpener. Theoretically, random projection theory shows that such a transformation provides a geometry-preserving embedding that, with high probability, approximately preserves pairwise Euclidean distances ( Johnson et al., 1984 ; Dasgupta & Gupta, 2003 ) and local geodesic structure ( Baraniuk & Wakin, 2009 ) , and acts as a near-isometry on tangent spaces under standard smoothness assumptions ( Papaspiliopoulos, 2020 ) . Empirically, Fig. 1 shows that the geometric anchor preserves the cluster topology (c), retains the original distance-drift profile in tangent space (d), and sharpens the eigenvalue spectrum (a). The LRP employs a bottleneck structure to model the inherently low-rank task-specific residuals, explicitly mirroring the low-rank features. Although LRP alone substantially distorts the geometry ( Fig. 1 (d), LRP), this behavior is expected because it is designed to learn task-specific residuals. The geometric anchor counterbalances this effect ( Fig. 1 (d), Random Uniform). Together, the full MR block ( Fig. 1 (d), MR Block) preserves the manifold geometry while capturing task-specific signals. The final representation is the sum of the two paths, balancing structural stability with adaptivity. The contribution of our paper can be summarized as follows: 1. We provide quantitative evidence that foundation-model features lie on a low-dimensional manifold and identify a common failure mode: linear layers can disrupt this fragile structure.

Read the complete text