MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification Abstract Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels, hindering model generalization. This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification. We first extend the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology image tiles, into a vision-language model by adding adaptors and aligning it with medical text encoders via contrastive learning on 923K image-text pairs. The model is then used to extract visual features and text embeddings from few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike prior methods that combine prompts with frozen features using prefix embeddings or self-attention, we propose multi-granular attention that compares interactions between learnable prompts with individual image patches and groups of them. This approach improves the model’s ability to capture both fine-grained details and broader context, enhancing its recognition of complex patterns across sub-regions. To further improve accuracy, we leverage (unbalanced) optimal transport-based visual-text distance to secure model robustness by mitigating perturbations that might occur during the data augmentation process. Empirical experiments on lung, kidney, and breast pathology modalities validate the effectiveness of our approach; thereby, we surpass several of the latest competitors and consistently improve performance across diverse architectures, including CLIP, PLIP, and Prov-GigaPath integrated PLIP. We release our implementations and pre-trained models at this MGPATH . Anh-Tien Nguyen 1,2,3 Duy Minh Ho Nguyen 6,7,8∗ Nghiem Tuong Diep 8,∗ Trung Quoc Nguyen 8 Nhat Ho 5 Jacqueline Michelle Metsch 1,2 Miriam Cindy Maurer 1,2 Daniel Sonntag 4,8 Hanibal Bohnenberger 1,2 Anne-Christin Hauschild 1,2 1 University of Göttingen, Germany 2 University Medical Center Göttingen, Germany 3 Max Planck Institute for Multidisciplinary Sciences, Germany 4 University of Oldenburg, Germany 5 The University of Texas at Austin † , USA 6 Max Planck Research School for Intelligent Systems (IMPRS-IS), Germany 7 University of Stuttgart, Germany 8 German Research Center for Artificial Intelligence (DFKI), Germany † † ⋆ Equal second contribution 1 Introduction Whole slide imaging (WSI) Niazi et al. 2019 has become essential in modern pathology for capturing high-resolution digital representations of entire tissue samples, enabling easier digital storage, sharing, and remote analysis Pantanowitz et al. 2011 . Unlike conventional methods that depend on examining slides under a microscope, WSI provides faster, detailed structural and cellular insights essential for disease diagnosis across multiple tissue layers, which is particularly valuable in cancer screening Barker et al. 2016 ; Cheng et al. 2021 . Nevertheless, WSIs are massive images, often containing billions of pixels Farahani et al. 2015 ; Song et al. 2023 , making detailed annotations and analysis difficult and expensive. To tackle these challenges, machine learning techniques incorporating few-shot and weakly supervised learning have been developed Madabhushi and Lee 2016 ; Li et al. 2023 ; Lin et al. 2023 ; Ryu et al. 2023 ; Shi et al. 2024 . Among these, multiple instance learning (MIL) and vision-language models (VLMs) have gained particular attention for their ability to effectively manage limited annotations and interpret complex whole-slide pathology images. Figure 1: Unlike previous methods that add prompts at prefix positions or patch-level attention - disrupting structural correlations - our MGPath framework integrates prompts at both regional and individual patch levels (multi-granular attention). In MIL Ilse et al. 2018 ; Xu et al. 2019a ; Li et al. 2023 ; Lin et al. 2023 ; Tang et al. 2023 ; Shi et al. 2023 , each WSI is first divided into smaller patches or instances. These instances are extracted feature embeddings using pre-trained vision encoders before being grouped into a "bag", i.e., a whole slide-level representation for the entire WSI. The MIL model mainly focuses on learning ensemble functions to identify patterns in specific patches, contributing to the overall label prediction for each bag (e.g., cancerous or non-cancerous), hence reducing the need for detailed annotations. Nonetheless, these methods often struggle to select relevant patches due to complex correlations and tissue variability Gadermayr and Tschuchnig 2024 ; Qu et al. 2024b . To overcome those obstacles, VLMs Lu et al. 2023 ; Huang et al. 2023 ; Ikezogwo et al. 2024 ; Shi et al. 2024 have emerged as a promising solution, combining slide-level visual features with textual descriptions to enrich contextual understanding and support predictions in sparse data scenarios with approaches such as zero-shot learning Xu et al. 2024 ; Ahmed et al. 2024 . Specifically, VLMs incorporate multi-scale images Shi et al. 2024 ; Han et al. 2024 , permitting the extraction of global and local WSI features at different resolutions. To adapt the pre-trained vision-language model efficiently, prompt learning Zhou et al. 2022b ; Gao et al. 2024 is employed where learnable prompts are treated as part of the input text to guide the model, and contextual prompts Li and Liang 2021 ; Yao et al. 2024 are integrated into feature embeddings using a self-attention mechanism Vaswani 2017 . Despite their strong classification performance across diverse tasks, these approaches still encounter certain limitations. First , (i) adapting prompt learning with frozen visual features often neglects the hierarchical relationships among learnable prompts and the visual features they interact with - specifically, the multi-granular attention between prompts to individual patches and groups of patches . This limitation lessens the model’s ability to capture interdependence across distinct scales — from fine-grained local features to broader contextual information, leading to less accurate comprehension of complex patterns in pathology images. Second , (ii) many VLMs rely on the CLIP architecture Radford et al. 2021 , which was not explicitly pre-trained on pathology images, thereby limiting its adaptability in few-shot settings, especially when the architecture is primarily frozen and prompt learning is applied. While there exist recent works that have incorporated PLIP Huang et al. 2023 , a model pre-trained on 200k pathology image-text pairs curated from Twitter and showed significant improvements, an open question remains whether scaling pre-training to millions or billions of pathology-specific samples could further boost performance. Lastly , (iii) most VLM models for whole-slide pathology rely on cosine similarity to align visual and textual features. This metric, however, can struggle with multiple text descriptions for sub-regions Chen et al. 2023 and with augmented data perturbations Nguyen et al. 2024b , as it lacks the precision to capture fine-grained alignments between varied image-text pairs. In this work, we present MGPath , a novel VLM method developed to address the challenges in whole-slide pathology classification. Our approach begins by adapting Prov-GigaPath Xu et al. 2024 - one of the largest pre-trained vision models trained on 1.3 billion pathology image patches - into a vision-language model. We accomplish this through contrastive learning with a pre-trained text encoder from the PLIP model Huang et al. 2023 , which was trained on approximately 200K pathology image-text pairs. To strengthen this alignment, we collected an additional 923K image-text pairs from ARCH Gamper and Rajpoot 2021 , PatchGastricADC22 Tsuneki and Kanavati 2022 and Quilt-1M Ikezogwo et al. 2024 and trained adaptor-based cross-alignment Gao et al. 2024 ; Cao et al. 2024 between Prov-GigaPath ’s visual encoder and PLIP ’s text encoder. Crucially, only lightweight adaptors are updated, making this process highly parameter-efficient. To the best of our knowledge, MGPath is the first parameter-efficient vision-language model trained for pathology at this data scale — utilizing 923K image-text pairs compared to the 200K in PLIP , and further benefiting from Prov-GigaPath ’s 1.3 billion sample pre-training. Next, we leverage these pre-trained models for few-shot WSI tasks by introducing multi-granular prompt learning . First, visual embeddings and descriptive text prompts are generated for image patches at different resolutions using large language models, which have been shown to improve performance Han et al. 2024 ; Shi et al. 2024 ; Qu et al. 2024a . Unlike prior methods that concatenate or use basic attention on individual patches Li and Liang 2021 ; Zhou et al. 2022b ; Yao et al. 2024 ; Shi et al. 2024 , our attention integrates learnable prompts with frozen visual features at both fine- and coarse-grained perspectives (Figure 1 ). We represent image patches from each WSI as a spatial graph, using bounding box coordinates to enable region-level aggregation through message passing along local connections. This spatial structure is encoded as tokens within the Key - Value matrices, which interact with Query matrices derived from prompt embeddings. By directing attention from Query to Key-Value matrices across both patch and region levels, our approach effectively captures hierarchical information, enriching feature representation and selectively emphasizing features across diverse tissue areas. Finally, to measure the distance between prompt-fused visual embedding and multiple text prompts, we resort to the optimal transport (OT) method Nguyen et al. 2021 ; Pham et al. 2020 ; Séjourné et al. 2023 ; Chen et al. 2023 ; Dong et al. 2023 ; Nguyen et al. 2024b ; Zhan et al. 2021 , providing flexibility in aligning heterogeneous data distributions. This property is beneficial in few-shot WSI classification when it can (i) handle data augmentation with noise, as OT can adapt to perturbations without losing meaningful structural relationships, and (ii) capture imbalances masses between two modality embeddings when text prompts only describe sub-regions in WSI samples. Through extensive evaluations of three datasets with various architectures ( CLIP-ResNet50 , CLIP-ViTB16 , PLIP , and (Prov-GigaPath)-integrated PLIP ), we observe that MGPath demonstrate consistent improvements over several state-of-the-art MIL and VLM in literature ( 14 competitors). As an example, MGPath with (Prov-GigaPath)-PLIP variant outperforms MSCPT Han et al. 2024 by 5\% in F1 and 8\% in AUC on the TCGA-BRCA dataset. Additionally, it also surpasses two state-of-the-art VLMs models, CONCH Lu et al. 2024 and QUILT Ikezogwo et al. 2024 , by approximately 6% in accuracy on TCGA-BRCA . 2 Related Work 2.1 Large-scale Pre-trained Models for Pathology Recent advancements in large-scale pre-trained models for pathology can be broadly classified into two categories. Vision models , such as Virchow Ikezogwo et al. 2024 , Hibou Nechaev et al. 2024 , UNI Chen et al. 2024 , and Prov-GigaPath Xu et al. 2024 leverage massive pathology image datasets to learn robust visual representations. Among these, Prov-GigaPath stands out as the largest model, trained on 1.3 billion pathology image patches, and excels in resolving complex tissue patterns at high resolution. On the other hand, vision-language models (VLMs) like PLIP Huang et al. 2023 (trained 200K image-text pairs), CONCH Lu et al. 2024 (1.17M), or QUILTNET Ikezogwo et al. 2024 (1M), integrate visual and textual information to enhance contextual understanding and improve pathology slide interpretation. In contrast, our MGPath combines the strengths of both approaches by using a parameter-efficient adaptor to link Prov-GigaPath (the largest pre-trained vision encoder) with a text encoder from VLMs like PLIP or CONCH , leveraging both rich visual features and semantic textual embeddings. Although we use the PLIP text encoder in our experiments due to its common use in baselines, the method can be extended to other larger pre-trained text models. 2.2 Few-shot learning in WSI MIL treats a WSI as a bag of patches and aggregates these instances into a bag of features, with early methods using non-parametric techniques like mean or max pooling. However, since disease-related patches are rare, these methods can overwhelm useful information with irrelevant data. To address this, attention-based methods, graph neural Networks (GNNs), and Transformer-based methods have been introduced Lu et al. 2021 ; Chen et al. 2021 ; Ilse et al. 2018 ; Li et al. 2021 ; Shao et al. 2021 ; Zheng et al. 2022 . In contrast, VLMs have gained popularity through contrastive learning, aligning image-text pairs to enhance performance on a variety of tasks. While collecting large-scale pathology image-text pairs remains challenging, models like MI-Zero, PLIP , and CONCH have been trained on hundreds of thousands to over a million pathology image-text pairs Lu et al. 2023 ; Huang et al. 2023 ; Lu et al. 2024 . Some approaches also integrate multi-magnification images and multi-scale text to mimic pathologists’ diagnostic processes, especially for detecting subtle abnormalities Shi et al. 2024 ; Han et al. 2024 . Our MGPath extends on the VLMs strategy by further amplifying the benefits of using a large pre-trained pathology VLM model and introducing a new parameter-efficient multi-granular prompt learning to adapt these models to few-shot settings. 2.3 Prompt Learning for Vision-Language Adaptations Prompt tuning is proposed to transfer large pre-trained model task-specific downstream tasks and has shown strong results in multimodal models like CLIP. Rather than design a heuristic template, several methods like CoOp Zhou et al. 2022b , CoCoOp Zhou et al. 2022a , or MaPLe Khattak et al. 2023 among others Rao et al. 2022 ; Shu et al. 2022 have allowed models to determine optimal prompts from multiple perspectives, such as domain generalization Ge et al. 2023 ; Yao et al. 2024 , knowledge prototype Zhang et al. 2022b ; Li et al. 2024 , or diversity Lu et al. 2022 ; Shu et al. 2022 . However, these approaches focus on natural images and do not address the unique challenges of whole-slide pathology images, which require multi-scale and structural contextual information. While a few current methods typically integrate prompts with frozen visual features via self-attention Shi et al. 2024 ; Qu et al. 2024a , these approaches might struggle with the complex relationships in WSIs. Our solution introduces multi-granular prompt learning, bridging attention on both individual image patches and spatial groups to better align with the hierarchical structure of WSI data. 3 Methods Figure 2 provides an overview of the key steps in our method. Before diving into the details of each section, we first introduce our PLIP model enhanced by Prov-GigaPath through the use of adaptors. Figure 2: The pipeline of the proposed MGPath method. Low- and high-resolution image patches are processed with large language models to generate visual contextual descriptions (Section 3.2 ). Visual prompts are integrated with frozen features through multi-granular attention at both patch and group-of-patch levels 3.3 . The final output is obtained by aligning visual and text embeddings using optimal transport (Section 3.4 ). 3.1 Bridging Pathology Visual and Text Encoders To leverage Prov-GigaPath ’s extensive pre-trained visual features for pathology, we implement lightweight adaptors that map image patch-level features to an embedding space aligned with the PLIP text encoder. These adaptors allow us to train joint image-text representations with parameter efficiency by updating only the adaptor weights. Given a set of collected pathology image-text pairs \left\{(\mathbf{I}_{i},\mathbf{T}_{i})|\,i=1,2..,N\,\right\} (Sec. 4 ), we denote by E_{I}(.) be a pre-trained vision encoder from Prov-GigaPath , extracting patch-level feature, and E_{T}(.) the pre-trained text encoder from PLIP model. Given a batch size of B samples, the image and text embeddings are computed as \mathbf{x}_{i}=E_{I}(\mathbf{I}_{i})\in\mathbb{R}^{d_{v}},\,\mathbf{t}_{i}=E_{T}(\mathbf{T}_{i})\in\mathbb{R}^{d_{t}} . We then design two trainable adaptors A_{I}(.) and A_{T}(.) , that maps \left(\mathbf{x}_{i},\mathbf{t}_{i}\right) into the same hidden dimension \mathbb{R}^{d} and minimizes the noise contrastive loss Oord et al. 2018 : \mathcal{L}_{con}=\mathbb{E}_{B}\left[-\log\frac{\exp\left(\cos(A_{I}(\mathbf{x}_{i}),A_{T}(\mathbf{t}_{i}))/\tau\right)}{\sum_{j}\exp\left(\cos(A_{I}(\mathbf{x}_{i}),A_{T}(\mathbf{t}_{j}))/\tau\right)}\right], (1) where \cos(.) is the cosine similarity, and \tau denotes for temperature of the softmax function. For parameter efficiency, we train only the adaptors A_{I}(.),A_{T}(.) while keeping the Prov-GigaPath visual encoder and PLIP text encoder frozen. After optimizing Eq.( 1 ), we use the outputs of the adaptors as visual and text embeddings for downstream tasks. Unless otherwise specified, we refer to this model as GigaPath-PLIP . 3.2 Multi-Magnification Descriptive Text Prompts To improve vision-language models (VLMs) for whole-slide image (WSI) analysis, designing effective text prompts is essential. Pathologists typically examine WSIs by first assessing tissue structures at low magnification before zooming in to analyze finer details such as nuclear size and shape. Inspired by this diagnostic workflow and the inherently multi-scale nature of WSIs, recent studies Shi et al. 2024 ; Han et al. 2024 have introduced dual-scale visual descriptive text prompts to guide VLMs, leading to significant improvements in classification performance. Building on this observation, we further extend and refine this strategy to enhance model effectiveness. First, to ensure that generated prompts remain robust across varying WSI magnifications, we design shared prompts that combine both high- and low-scale descriptive elements, treating them as contextual embeddings. Specifically, we leverage the API of a frozen language model (GPT-4) and query it with the prompt as Figure 3 LLM Prompt What visually descriptive features characterize {class name} at both low and high resolutions within the whole-slide image? Please summarize into a single paragraph. Figure 3: LLM template prompt. In the above query, we replace {class name} by specific categories, for e.g., they are invasive ductal carcinoma (IDC) and invasive lobular carcinoma (ILC) in the TCGA-BRCA dataset. Second, at each low/high scale, rather than inserting a single learnable text prompt of length K alongside a frozen contextual prompt from LLMs Shi et al. 2024 ; Han et al. 2024 , we propose using M learnable prompts . This approach aims to capture different sub-regions or structural features within each patch that might be overlooked with only a single prompt. Specifically, we define visual descriptive text prompts for both low and high resolutions as follows: \begin{split}\mathbf{T}_{i}^{(l)}&=\left\{\left([\mathbf{\omega}_{i}^{(l)}]_{1}\,[\mathbf{\omega}_{i}^{(l)}]_{2}\,...[\omega_{i}^{(l)}]_{K}[\mathrm{LLM\,context}]\right)|_{i=1}^{M}\right\}\\ \mathbf{T}_{i}^{(h)}&=\left\{\left([\mathbf{\omega}_{i}^{(h)}]_{(1)}\,[\mathbf{\omega}_{i}^{(h)}]_{2}\,...[\omega_{i}^{(h)}]_{K}[\mathrm{LLM\,context}]\right)|_{i=1}^{M}\right\},\end{split} (2) where [\omega_{i}^{\beta}]_{j},j\in[1,...,K],i\in[1,..,M] are KM trainable textual prompts for each resolution \beta\in\{l,h\} . 3.3 Granularity-aware Visual Prompt Learning We propose to adapt visual prompts to frozen features extracted by a pre-trained vision encoder in the VLM model by taking into account the image patch level and spatial groupings of patches. Specifically, for each WSI W , we denote by \left\{W^{(l)},W^{(h)}\right\} are representations of W at low and high magnification. We define a bag of multiple instances of W as I=\left\{I^{(l)},I^{(h)}\right\} where I^{(l)}\in\mathbb{R}^{N_{l}\times N_{b}\times N_{b}\times 3} , I^{(h)}\in\mathbb{R}^{N_{h}\times N_{b}\times N_{b}\times 3} with N_{l},N_{h} indicate the number of low and high-resolution image patches and N_{b} is the patch size. Following prior works Shi et al. 2024 ; Ilse et al. 2018 ; Lu et al. 2021 ; Han et al. 2024 , we employ a non-overlapping sliding window technique to extract patches I from the WSI. 3.3.1 Patches-based Prompting The frozen image encoder E_{I}(.) (or A_{I}(E_{I}(.)) in case of GigaPath-PLIP ) is used to map patches I into a feature vector H=\{H^{(l)}\in\mathbb{R}^{N_{l}\times d},H^{(h)}\in\mathbb{R}^{N_{h}\times d}\} where d denotes the feature dimension. To effectively consolidate the extensive set of patch features into a final slide-level representation, we introduce a set of learnable visual prompts \mathbf{p}_{v}\in\mathbb{R}^{N_{p}\times d} , which facilitate the progressive merging of patch features in H^{(l)} (similarly for H^{(h)} ) (Figure 2 ). In particular, we formulate \mathbf{p}_{v} as Query and take all features in H^{(l)} as the Keys K^{(l)}_{p} and Values V^{(l)}_{p} in in self-attention Vaswani 2017 . We then associate \mathbf{p}_{v} with patch features as: \mathbf{p}_{v,p}^{(l)}=\operatorname{Normalize}\left(\operatorname{SoftMax}\left(\frac{\mathbf{p}_{v}K_{p}^{(l)^{T}}}{\sqrt{d}}\right)V_{p}^{(l)}\right)+\mathbf{p}_{v}, (3) where \operatorname{Normalize(.)} and \operatorname{Softmax(.)} indicate the layer normalization operator and activation function respectively. Intuitively, Eq.( 3 ) computes the correlations between the visual prompt \mathbf{p}_{v} and all individual feature patches in H^{l} , subsequently grouping patches with high similarity to form fused prompt embeddings. However, since cancerous tissues in WSIs often appear as large, contiguous regions of adjacent image patches, this motivates the introduction of spatial patch group-based attention. 3.3.2 Spatial Patch Group-based Prompting We build spatial correlations for multiple instances in I by using their image patch coordinates inside each WSI W . In particular, taken I^{(l)}=\left\{I^{(l)}_{1},\,I^{(l)}_{2},...,I^{(l)}_{N_{l}}\right\} with their corresponding extracted features in H^{(l)}=\left\{H^{(l)}_{1},H^{(l)}_{2},...,H^{(l)}_{N_{l}}\right\} , we construct a graph G^{(l)}=(V^{(l)},E^{(l)}) to capture regional tissue structures where the set of vertices V^{(l)}=I^{(l)} , and E^{(l)}\in\{0,1\}^{N_{l}\times N_{l}} is the set of edges. Edges in E^{(l)} can be defined by linking inner patches to their K-nearest neighbors based on the coordinates. We define the node-feature embedding as X^{(l)}=H^{(l)}\in\mathbb{R}^{N_{l}\times d} that associates each vertex v^{(l)}_{i} with its feature node x_{i}^{(l)}=H^{(l)}_{i} . We next design a trainable message-passing network g_{\epsilon}(.) based on the graph attention layer (GAT) Veličković et al. 2017 to capture the feature representation of each node and its local neighbors. The message passing of the GAT layer is formulated as: \begin{split}\alpha_{i,j}&=\frac{\exp{\left(\sigma(a_{s}^{T}\Theta_{s}\,x_{i}^{(l)}+a_{t}^{T}\Theta_{t}\,x_{j}^{(l)})\right)}}{\sum_{k\in\mathcal{N}(i)\cup\{i\}}\exp({\sigma(a_{s}^{T}\Theta_{s}\,x_{i}^{(l)}+a_{t}^{T}\Theta_{t}\,x_{k}^{(l)}))}}\\ x_{i}^{(l)^{\prime}}&=\alpha_{i,i}\Theta_{s}\,x_{i}^{(l)}+\sum_{j\in\mathcal{N}(i)}\alpha_{i,j}\Theta_{t}\,x_{j}^{(l)},\end{split} (4) where x_{i}^{(l)^{\prime}} is aggregated features of x_{i}^{(l)} with its local region after GAT layer, \sigma(.) is the LeakyReLU activation function, \mathcal{N}(i) denote the neighboring nodes of the i -th node, \alpha_{i,j} are the attention coefficients and a_{s},a_{t},\Theta_{s},\Theta_{t} are weight parameters of g_{\epsilon}(.) . After doing a message passing by g_{\epsilon}(.) , the graph of patch-image features G^{(l)} is updated to G^{(l)^{\prime}} , where each node now represents a super-node that encapsulates its corresponding feature region. We then squeeze all feature nodes in G^{(l)^{\prime}} as a vector H^{(l)}_{gr} and treat them as another Keys K_{gr}^{(l)} and Values V_{gr}^{(l)} for region-level features. Similar to Eq.( 3 ), we associate prompt \mathbf{p}_{v} with those group-level features: \mathbf{p}_{v,gr}^{l}=\operatorname{Normalize}\left(\operatorname{SoftMax}\left(\frac{\mathbf{p}_{v}K_{gr}^{(l)^{T}}}{\sqrt{d}}\right)V_{gr}^{(l)}\right)+\mathbf{p}_{v}. (5) The final output of our multi-granular is computed as: \mathbf{p}_{v}^{(l)}=(1-\alpha)\cdot\mathbf{p}_{v,p}^{(l)}+\alpha\cdot\mathbf{p}_{v,gr}^{(l)}, (6) which interpolates between image patches and spatial patch groups. 3.4 Optimal Transport for Visual-Text Alignment Given descriptive text prompts \mathbf{T}^{(l)} and \mathbf{T}^{(h)} (Eq.( 2 )) and visual prompt-guided slide features \mathbf{p}_{v}^{(l)} and \mathbf{p}_{v}^{(h)} (Eq.( 6 )) for low and high resolutions, our goal is to maximize the similarity between slide and text embeddings for each class c . Rather than relying on cosine distance, as in prior works Zhou et al. 2022b ; Zhou et al. 2022a ; Zhao et al. 2024 ; Qu et al. 2024a ; Qu et al. 2024a ; Singh and Jaggi 2020 , we propose using optimal transport (OT)-based distance to capture a more nuanced cross-alignment between visual and text domains. Although OT has been explored for prompt learning in natural images and multi-modal learning Kim et al. 2023 ; Chen et al. 2023 ; Nguyen et al. 2024a ; Séjourné et al. 2023 , we are the first to adapt it for whole-slide imaging (WSI), effectively handling the alignment of multi-magnification patches to capture rich structural details across scales. Recap OT: Given two sets of points (features), we can represent the corresponding discrete distributions as follows: \bm{\mu}=\sum^{M}_{i=1}p_{i}\delta_{f_{i}},\quad\bm{\nu}=\sum^{N}_{j=1}q_{j}\delta_{g_{i}}, (7) where \delta_{f} and \delta_{g} represent Dirac delta functions centered at \bm{f} and \bm{g} , respectively, and M and N indicate the dimensions of the empirical distribution. The weight vectors \bm{p}=\{p_{i}\}^{M}_{i=1} and \bm{q}=\{q_{i}\}^{N}_{j=1} lie within the M and N -dimensional simplex, respectively, meaning they satisfy \sum_{i=1}^{M}p_{i}=1 and \sum_{j=1}^{N}q_{j}=1 . The discrete optimal transport problem can then be expressed as: \displaystyle\bm{T}^{\ast}=\underset{\bm{T}\in\mathbb{R}^{MXN}}{\arg{\min}}\sum^{M}_{i=1}\sum^{N}_{j=1}\bm{T}_{ij}\bm{C}_{ij} \displaystyle\textrm{s.t.}\quad\bm{T}\bm{1}^{N}=\bm{\mu},\quad\bm{T}^{\top}\bm{1}^{M}=\bm{\nu}. (8) where \bm{T}^{\ast} is denoted as the optimal transport plan, which is optimized to minimize the total distance between the two probability vectors, \bm{C} is the cost matrix which measures the distance between \bm{f}_{i} and \bm{g}_{j} . We then define the OT distance between \bm{\mu} and \bm{\nu} as: d_{\mathrm{OT}}(\bm{\mu},\bm{\nu})=\langle\bm{T}^{\ast},\bm{C}\rangle. (9) Objective functions: Given the visual prompt-guided slide features \mathbf{p}_{v}^{(l)}\in\mathbb{R}^{N_{p}\times d} in Eq.( 6 ) and the descriptive text prompts \mathbf{T}^{(l)} in Eq.( 2 ), we compute the textual embedding for \mathbf{T}^{(l)} as \mathbf{p}_{t}^{(l)}=E_{T}(\mathbf{T}^{(l)})\in\mathbb{R}^{M\times d} . We next denote \mathbf{T}^{(l)}_{c} as the input text prompts, \left(\mathbf{p}_{t}^{(l)}\right)_{c} as the extracted textual embedding, and \left(\mathbf{p}_{v}^{(l)}\right)_{c} as the visual prompt-guided slide features associated with class c . We then aim to minimize the distance between \mathbf{T}^{(l)}_{c} and \left(\mathbf{p}_{v}^{(l)}\right)_{c} , indicated as d_{\mathrm{OT}}\left(\mathbf{T}^{(l)}_{c},\left(\mathbf{p}_{v}^{(l)}\right)_{c}\right) in the paper, by computing optimal transport distance between \left(\mathbf{p}_{t}^{(l)}\right)_{c} and \left(\mathbf{p}_{v}^{(l)}\right)_{c} . Specifically, we treat \left(\mathbf{p}_{t}^{(l)}\right)_{c}\rightarrow\bm{F}=\left\{\bm{f}_{i}|_{i=1}^{M}\right\} and \left(\mathbf{p}_{v}^{(l)}\right)_{c}\rightarrow\bm{G}=\left\{\bm{g}_{j}|_{j=1}^{N_{p}}\right\} and compute the cost matrix \bm{C} as \bm{C}=\left(\bm{1}-\bm{F}^{T}\,\bm{G}\right)\in\mathbb{R}^{M\times N_{p}} , which used to compute \bm{T}^{\ast} in Eq. ( 8 ) for estimate optimal transport distance defined in Eq. ( 9 ). Following the same procedure, we can also compute d_{\mathrm{OT}}\left(\mathbf{T}^{(h)}_{c},\left(\mathbf{p}_{v}^{(h)}\right)_{c}\right) at high-resolution image patches. Then, the prediction probability is written as: \mathrm{P}_{c}=\frac{\exp(2-\sum_{k\in\{l,h\}}\,d_{\mathrm{OT}}\left(\mathbf{T}^{(k)}_{c},\left(\mathbf{p}_{v}^{(k)}\right)_{c}\right))}{\sum_{c^{\prime}=1}^{C}\exp(2-\sum_{k\in\{l,h\}}\,d_{\mathrm{OT}}\left(\mathbf{T}^{(k)}_{c^{\prime}},\left(\mathbf{p}_{v}^{(k)}\right)_{c}\right))}, (10) where \lambda_{k} controls contribution of each-resolution. Finally, we can train the model with the cross-entropy as: \mathcal{L}_{class}=\operatorname{Cross}(\mathrm{P},\mathrm{GT}), (11) with \operatorname{Cross}(.) be the cross-entropy and \mathrm{GT} denotes slide-level ground-truth. The details for solvers of Eq.( 9 ) and a relaxed version with unbalanced optimal transport are presented in Sections ( D ) and ( D.1 ) in Appendix. Intuitively, using OT, in this case, offers several key advantages over cosine similarity. Pathology images exhibit complex, heterogeneous patterns that can be described from multiple perspectives. OT models these relationships as a distribution, enabling a more holistic alignment that handles variability and incomplete details while reducing noise from irrelevant prompts. This enhances the model’s ability to generalize to unseen or complex disease cases. 4 Experiments 4.1 Settings Datasets for contrastive learning. PatchGastricADC22 Tsuneki and Kanavati 2022 consists of approximately 262K patches derived from WSI of H&E-stained gastric adenocarcinoma specimens, each paired with associated diagnostic captions collected from the University of Health and Welfare, Mita Hospital, Japan. QUILT-1M Ikezogwo et al. 2024 includes approximately 653K images and one million pathology image-text pairs, gathered from 1,087 hours of educational histopathology videos presented by pathologists on YouTube. ARCH Gamper and Rajpoot 2021 is a pathology multiple-instance captioning dataset containing pathology images at the bag and tile level. However, our work focuses on tile-level images from all datasets for our contrastive training strategy. In total, we collected approximately 923K images from these datasets. Downstream tasks. For the classification task, the proposed method was evaluated in three datasets from the Cancer Genome Atlas Data Portal The Cancer Genome Atlas () TCGA : TCGA-NSCLC , TCGA-RCC , and TCGA-BRCA . We followed the ViLa-MIL Shi et al. 2024 experimental settings for TCGA-NSCLC and TCGA-RCC , randomly selecting proportions for training, validation, and testing. For TCGA-BRCA , we adapted the training and testing slide ID from MSCPT Han et al. 2024 . The detailed description is included in the appendix section. Implementation Details. We followed the ViLa-MIL preprocessing pipeline for tissue region selection and patch cropping. To integrate our attention module with CLIP50 and PLIP , we extracted tile-level embeddings from their frozen vision encoders (1024-dimensional for CLIP50 and 512 for PLIP ). We used the visual encoder of Prov-GigaPath to produce 1536-dimensional embeddings. To align it with PLIP ’s frozen text encoder, we developed two MLP-based adaptors that project both encoders into a shared feature space during a contrastive learning process, using datasets outlined in Section 4 . Table 1: Comparison of methods on TCGA-BRCA with few-shot settings. Results are shown for AUC, F1, and Accuracy (ACC). FVM denotes for foundation vision-language models. CLIP ImageNet Pretrained Methods # Param. TCGA-BRCA AUC F1 ACC Max-pooling 197K 60.42 \pm 4.35 56.40 \pm 3.58 68.55 \pm 6.54 Mean-pooling 197K 66.64 \pm 4.21 60.70 \pm 2.78 71.73 \pm 3.59 ABMIL Ilse et al. 2018 461K 69.24 \pm 3.90 61.72 \pm 3.36 72.77 \pm 3.15 CLAM-SB Lu et al. 2021 660K 67.80 \pm 5.14 60.51 \pm 5.01 72.46 \pm 4.36 CLAM-MB Lu et al. 2021 660K 60.81 \pm 4.87 55.48 \pm 4.96 67.31 \pm 4.19 TransMIL Shao et al. 2021 2.54M 65.62 \pm 3.20 60.75 \pm 4.04 67.52 \pm 4.16 DSMIL Li et al. 2021 462K 66.18 \pm 3.08 59.35 \pm 3.18 67.52 \pm 1.56 RRT-MIL Tang et al. 2024 2.63M 66.33 \pm 4.30 61.14 \pm 5.93 71.21 \pm 8.94 CoOp Zhou et al. 2022b 337K 68.86 \pm 4.35 61.64 \pm 2.40 71.08 \pm 3.22 CoCoOp Zhou et al. 2022a 370K 69.13 \pm 4.27 61.48 \pm 2.62 72.41 \pm 1.87 Metaprompt Zhao et al. 2024 360K 69.12 \pm 4.46 63.39 \pm 4.38 74.65 \pm 7.20 TOP Qu et al. 2024a 2.11M 69.74 \pm 3.14 63.39 \pm 4.62 74.41 \pm 5.27 ViLa-MIL Shi et al. 2024 2.77M 72.25 \pm 6.16 62.04 \pm 2.38 75.01 \pm 6.14 MSCPT Han et al. 2024 1.35M 74.56 \pm 4.54 65.59 \pm 1.85 75.82 \pm 2.38 MGPath (ViT) 592K 74.96 \pm 6.98 64.60 \pm 5.39 77.10 \pm 2.39 FVM CONCH Lu et al. 2024 110M 84.11 \pm 15.44 65.63 \pm 10.81 73.24 \pm 8.89 QUILT Ikezogwo et al. 2024 63M 73.48 \pm 10.57 63.78 \pm 8.72 73.26 \pm 10.13 PLIP Pathology Pretrained Max-pooling 197K 66.50 \pm 2.74 61.50 \pm 2.88 71.57 \pm 4.82 Mean-pooling 197K 71.62 \pm 2.41 66.34 \pm 2.96 74.45 \pm 2.49 ABMIL Ilse et al. 2018 461K 72.41 \pm 4.25 63.04 \pm 3.62 74.09 \pm 4.38 CLAM-SB Lu et al. 2021 660K 72.34 \pm 6.17 65.51 \pm 3.28 76.16 \pm 4.36 CLAM-MB Lu et al. 2021 660K 73.41 \pm 3.76 66.11 \pm 1.94 77.88 \pm 2.30 TransMIL Shao et al. 2021 2.54M 74.98 \pm 6.01 67.50 \pm 6.00 77.04 \pm 6.14 DSMIL Li et al. 2021 462K 71.44 \pm 2.72 64.48 \pm 1.64 75.26 \pm 2.28 RRT-MIL Tang et al. 2024 2.63M 71.21 \pm 6.46 64.15 \pm 1.38 75.92 \pm 5.10 CoOp Zhou et al. 2022b 337K 71.53 \pm 2.45 64.84 \pm 2.40 74.22 \pm 5.02 CoCoOp Zhou et al. 2022a 370K 72.65 \pm 4.63 66.63 \pm 3.55 66.98 \pm 3.35 Metaprompt Zhao et al. 2024 360K 74.86 \pm 4.25 65.03 \pm 1.81 77.88 \pm 3.22 TOP Qu et al. 2024a 2.11M 76.13 \pm 6.01 66.55 \pm 1.72 78.58 \pm 5.30 ViLa-MIL Shi et al. 2024 2.77M 74.06 \pm 4.62 66.03 \pm 1.81 78.12 \pm 4.88 MSCPT Han et al. 2024 1.35M 75.55 \pm 5.25 67.46 \pm 2.43 79.14 \pm 2.63 MGPath 592K 79.02 \pm 6.43 68.25 \pm 4.42 79.65 \pm 1.72 MGPath (PLIP-G) 5.35M 87.36 \pm 1.85 73.13 \pm 3.49 79.56 \pm 4.77 To implement spatial attention, we use a Graph Attention Network (GAT) to model spatial relationships between WSI patches. Each tile-level embedding serves as a node, connected to its left, right, top, and bottom neighbors, ensuring local spatial dependencies are captured. We then integrate spatial patch group-based attention \mathbf{p}_{v,gr} into patch-based attention \mathbf{p}_{v,p} using Equation 6 . The hyperparameter \alpha (0 to 1) controls the balance between spatial context and prototype-based guidance. 4.2 Comparison to State-of-the-Art We compare our MGPath with state-of-the-art multi-instance learning methods, including Maxpooling , Mean- pooling , ABMIL Ilse et al. 2018 , CLAM Lu et al. 2021 , TransMIL Shao et al. 2021 , DSMIL Li et al. 2021 , GTMIL Zheng et al. 2022 , DTMIL Zhang et al. 2022a , RRT-MIL Tang et al. 2024 and IBMIL Lin et al. 2023 , and vision-language methods, including CoOp Zhou et al. 2022b , CoCoOp Zhou et al. 2022a , Metaprompt Zhao et al. 2024 , TOP Qu et al. 2024a , ViLa-MIL Shi et al. 2024 , MSCPT Han et al. 2024 , QUILT Ikezogwo et al. 2024 , CONCH Lu et al. 2024 . Among these, QUILT and CONCH are foundation VLMs. We provide different versions for our MGPath including CLIP backbone ReNet-50 ( CLIP50 ) for TCGA-NSCLC and TCGA-RCC and ViT-16 backbone for TCGA-BRCA . We also provide a version using PLIP backbone, as well as GigaPath-PLIP , which was pre-trained on the Pathology dataset. 4.3 Results on Few-shot and Zero-shot Settings. MGPath with CLIP and PLIP backbones outperform several competitive MIL and VLM methods. As shown in Tables 2 and 1 , our MGPath , based on CLIP50 and PLIP , outperforms several baseline models and achieves significant improvements over other VLMs with similar architectures, such as ViLa and MSCPT. The performance gain is particularly notable with the PLIP backbone. For example, on TCGA-BRCA using CLIP (ViT), MGPath achieves an accuracy of 77.10%, compared to 75.82% for MSCPT and 75.01% for ViLA-MIL. Additionally, with the PLIP backbone, MGPath surpasses MSCPT and ViLa-MIL by margins of approximately 3.5% to 5%. GigaPath-PLIP is a strong pre-trained VLM. We validated our whole-slide vision foundation model, pre-trained on 1.3 billion pathology images, using the PLIP text encoder. By incorporating pathology-specific features from Prov-GigaPath Xu et al. 2024 , the integrated MGPath (PLIP-G) model demonstrated strong performance across multiple metrics on the TCGA-NSCLC , TCGA-RCC , and TCGA-BRCA datasets. When compared to other foundation VLMs such as CONCH and QUILT , our model consistently outperforms them. For example, we achieve a 3% improvement in AUC over CONCH on both the TCGA-BRCA and TCGA-NSCLC datasets. Table 2: Comparison of methods on TCGA-NSCLC, and TCGA-RCC datasets with few-shot settings. Results are shown for AUC, F1, and Accuracy (ACC). Methods # Param. TCGA-NSCLC TCGA-RCC AUC F1 ACC AUC F1 ACC Max-pooling 197K 53.0 \pm 6.0 45.8 \pm 8.9 53.3 \pm 3.4 67.4 \pm 4.9 46.7 \pm 11.6 54.1 \pm 4.8 Mean-pooling 197K 67.4 \pm 7.2 61.1 \pm 5.5 61.9 \pm 5.5 83.3 \pm 6.0 60.9 \pm 8.5 62.3 \pm 7.4 ABMIL Ilse et al. 2018 461K 60.5 \pm 15.9 56.8 \pm 11.8 61.2 \pm 6.1 83.6 \pm 3.1 64.4 \pm 4.2 65.7 \pm 4.7 CLAM-SB Lu et al. 2021 660K 66.7 \pm 13.6 59.9 \pm 13.8 64.0 \pm 7.7 90.1 \pm 2.2 75.3 \pm 7.4 77.6 \pm 7.0 CLAM-MB Lu et al. 2021 660K 68.8 \pm 12.5 60.3 \pm 11.1 63.0 \pm 9.3 90.9 \pm 4.1 76.2 \pm 4.4 78.6 \pm 4.9 TransMIL Shao et al. 2021 2.54M 64.2 \pm 8.5 57.5 \pm 6.4 59.7 \pm 5.4 89.4 \pm 5.6 73.0 \pm 7.8 75.3 \pm 7.2 DSMIL Li et al. 2021 462K 67.9 \pm 8.0 61.0 \pm 7.0 61.3 \pm 7.0 87.6 \pm 4.5 71.5 \pm 6.6 72.8 \pm 6.4 GTMIL Zheng et al. 2022 N/A 66.0 \pm 15.3 61.1 \pm 12.3 63.8 \pm 9.9 81.1 \pm 13.3 71.1 \pm 15.7 76.1 \pm 12.9 DTMIL Zhang et al. 2022a 986.7K 67.5 \pm 10.3 57.3 \pm 11.3 66.6 \pm 7.5 90.0 \pm 4.6 74.4 \pm 5.3 76.8 \pm 5.2 IBMIL Lin et al. 2023 N/A 69.2 \pm 7.4 57.4 \pm 8.3 66.9 \pm 6.5 90.5 \pm 4.1 75.1 \pm 5.2 77.2 \pm 4.2 ViLa-MIL Shi et al. 2024 8.8M/47M 74.7 \pm 3.5 67.0 \pm 4.9 67.7 \pm 4.4 92.6 \pm 3.0 78.3 \pm 6.9 80.3 \pm 6.2 CONCH ( Lu et al. 2024 ) 110M 89.46 \pm 10.2 78.5 \pm 9.31 78.78 \pm 9.1 88.08 \pm 4.59 78.21 \pm 4.2 71.67 \pm 19.4 QUILT Ikezogwo et al. 2024 63M 79.66 \pm 13.19 72.30 \pm 13.35 72.42 \pm 13.24 96.92 \pm 1.6 78.46 \pm 5.55 86.34 \pm 1.56 MGPath (CLIP) 1.6M/39M 77.2 \pm 1.3 70.9 \pm 2.0 71.0 \pm 2.1 92.1 \pm 2.8 76.5 \pm 5.2 81.7 \pm 2.9 MGPath (PLIP) 592K 83.6 \pm 4.5 76.41 \pm 4.8 76.5 \pm 4.8 94.7 \pm 1.6 78.6 \pm 4.9 83.6 \pm 3.5 MGPath (PLIP-G) 5.35M 93.02 \pm 2.99 84.64 \pm 4.75 84.77 \pm 4.67 98.2 \pm 0.31 88.33 \pm 3.41 91.72 \pm 1.74 Table 3: Zero-shot classification performance on TCGA-NSCLC, TCGA-RCC, and TCGA-BRCA datasets. Metrics reported include balanced accuracy (B-Acc) and weighted F1-score (W-F1). Zero-shot TCGA-NSCLC TCGA-RCC TCGA-BRCA Average B-Acc W-F1 B-Acc W-F1 B-Acc W-F1 B-Acc W-F1 QuiltNet 61.3 56.1 59.1 51.8 51.3 40.1 57.23 49.33 CONCH 80.0 79.8 72.9 69.1 64.0 61.2 72.3 70.03 PLIP 70.0 68.5 50.7 46.0 64.7 63.8 61.8 59.43 PLIP-G (Our) 72.7 72.6 81.3 81.4 70.0 69.9 74.67 74.63 GigaPath-PLIP achieves competitive performance in zero-shot tasks. We evaluate the zero-shot capabilities of our model on three datasets and compare its performance against foundation VLMs such as CONCH, QUILT, and PLIP. The results, summarized in Table 3 , show that the proposed VLM model achieves the best average performance across datasets, followed by CONCH and PLIP. This consistent top-tier performance across multiple benchmarks underscores the robustness and generalizability of our model. 4.4 Ablation Studies PLIP enhanced Prov-GigaPath. We validate the use of Prov-GigaPath and PLIP under the following settings: (i) using full vision-language PLIP model; (ii) combining Prov-GigaPath with PLIP through the MLP layer which was pre-trained on the large-scale dataset; (iii) integrating Prov-GigaPath with PLIP through adaptor layers which were randomly initialized; (iv) utilizing Prov-GigaPath and an adaptor layer to map to the class output and only train MLP and last FFN layer of slide encoder; (v) only using Prov-GigaPath and an MLP layer to map to the class output and train MLP, the query matrix of last layer and the last FFN layer of slide encoder. Table 6 shows that using Prov-GigaPath combined with PLIP boosts the final performance compared to only using PLIP or Prov-GigaPath . Table 4: Ablation studies on multi-granular (M-Gran), ratio combines two attention levels ( \alpha ), and message passing network types. Configurations TCGA-NSCLC AUC F1 ACC MGPath (CLIP) 76.2 \pm 2.2 69.0 \pm 3.5 69.3 \pm 2.8 - w/o M-Gran (CLIP) 74.6 \pm 2.2 67.8 \pm 2.4 67.8 \pm 2.5 MGPath (PLIP-G) 91.7 \pm 3.6 84.2 \pm 4.6 84.4 \pm 4.5 - w/o M-Gran (PLIP-G) 90.6 \pm 4.5 82.4 \pm 5.7 82.5 \pm 5.7 MGPath , \alpha=0.2 76.2 \pm 2.2 69.0 \pm 3.5 69.3 \pm 2.8 - \alpha=0.5 73.7 \pm 3.1 67.4 \pm 2.6 67.8 \pm 2.7 - \alpha=0.8 72.2 \pm 5.2 66.4 \pm 5.5 66.8 \pm 5.2 TCGA-RCC MGPath (CLIP) 92.1 \pm 2.8 76.5 \pm 5.2 81.7 \pm 2.9 - w/o M-Gran (CLIP) 91.6 \pm 3.5 72.3 \pm 6.4 80.2 \pm 4.4 MGPath (PLIP-G) 98.1 \pm 0.6 85.7 \pm 1.1 89.9 \pm 2.0 - w/o M-Gran (PLIP-G) 98.1 \pm 0.6 85.0 \pm 4.0 89.3 \pm 3.0 Table 5: Ablation studies on adaptor learning for Prov-GiGaPath and PLIP . PLIP-G denotes for mixed version between Prov-GiGaPath and PLIP. Methods # Param. TCGA-NSCLC AUC F1 ACC MGPath (PLIP) 592K 83.6 \pm 4.5 76.41 \pm 4.8 76.5 \pm 4.8 MGPath (PLIP-G) 5.35M 91.7 \pm 3.6 84.2 \pm 4.6 84.4 \pm 4.5 MGPath Random Adaptors 5.35M 91.4 \pm 4.2 82.8 \pm 5.7 83.0 \pm 5.6 GiGaPath Tuning (MLP + last FFN) 4.7M 62.7 \pm 3.5 64.66 \pm 5.3 52.8 \pm 3.4 GiGaPath Tuning (MLP + last Q-ViT) 5.8M 83.1 \pm 6.9 74.3 \pm 7.5 75.8 \pm 6.1 Table 6: Contribution of OT and multiple descriptive text prompts Methods TCGA-NSCLC AUC F1 ACC MGPath (OT, 4 text prompts) 76.2 \pm 2.2 69.0 \pm 3.5 69.3 \pm 2.8 MGPath (OT, 2 text prompts) 77.2 \pm 1.3 70.9 \pm 2.0 71.0 \pm 2.1 MGPath (Cosine, 2 text prompts) 75.8 \pm 3.7 68.3 \pm 4.5 68.4 \pm 4.5 TCGA-RCC MGPath (OT, 4 text prompts) 92.1 \pm 2.8 76.5 \pm 5.2 81.7 \pm 2.9 MGPath (OT, 2 text prompts) 92.1 \pm 2.6 75.6 \pm 3.9 80.4 \pm 2.4 MGPath (Cosine, 4 text prompts) 91.8 \pm 2.8 75.9 \pm 4.3 80.5 \pm 2.6 Multi-Granular Prompt Learning. In Table 6 , we show the performance of MGPath with and without multi-granular ( M-Gran ) for CLIP (row 1 and 2) and PLIP-G (row 3 and 4) on TCGA-NSCLC dataset. It shows that using M-Gran improves the final performance of MGPath . This also happens on TCGA-RCC dataset. Table 6 also shows the impact of ratio when combining attention with graph and attention no graph on TCGA-NSCLC . It shows that with a ratio of 0.2/0.8 (0.2 for spatial attention obtained from graph structure and 0.8 for prototype-guided attention), MGPath achieves the highest performance. OT as Alignment between Contextual Prompts. Table 6 validates the use of OT in our MGPath on TCGA-NSCLC and TCGA-RCC . We see that using OT helps to boost the performance of MGPath (rows 1 and 2) compared to the use of cosine (rows 3 and 4). It also shows that the number of prompt vectors depends on each dataset. In the appendix, we also run with another version using unbalanced optimal transport (UoT). We observe that both UoT and OT provide good alignment quality, with UoT slightly outperforming OT. However, this advantage comes at the cost of increased running time. 4.5 Discussion While we demonstrate significant improvements in few-shot and zero-shot WSI classification across several settings, this paper does not explore other important challenges. For example, how can we scale the current attention mechanism to handle even larger image patches (e.g., using Flash Attention Dao et al. 2022 ), or extend the model from classification to tumor segmentation tasks Khened et al. 2021 . Additionally, the potential for extending GiGaPath to integrate with other large-scale VLM models, such as CONCH Lu et al. 2024 , remains unexplored. 5 Conclusion High-resolution WSI is crucial for cancer diagnosis and treatment but presents challenges in data analysis. Recent VLM approaches, which utilize few-shot and weakly supervised learning, have shown promise in handling complex whole-slide pathology images with limited annotations. However, many overlook the hierarchical relationships between visual and textual embeddings, ignoring the connections between global and local pathological details or relying on non-pathology-specific pre-trained models like CLIP . Additionally, previous metrics lack precision in capturing fine-grained alignments between image-text pairs. To address these gaps, (i) we propose MGPath , which integrates Prov-GigaPath with PLIP , cross-aligning them with 923K domain-specific image-text pairs. (ii) Our multi-granular prompt learning approach captures hierarchical tissue details effectively, (iii) while OT-based visual-text distance ensures robustness against data augmentation perturbations. Extensive experiments on three cancer subtyping datasets demonstrate that MGPath achieves state-of-the-art results in WSI classification. We expect that this work will pave the way for combining large-scale domain-specific models with multi-granular prompt learning and optimal transport to enhance few-shot learning in pathology. 6