Enabling Progressive Whole-slide Image Analysis with Multi-scale Pyramidal Network Shuyang Wu, Yifu Qiu, Ines P. Nearchou, Sandrine Prost, Jonathan A. Fallowfield, Hakan Bilen, Timothy J. Kendall Abstract Multiple-instance Learning (MIL) is commonly used to undertake computational pathology (CPath) tasks, and the use of multi-scale patches allows diverse features across scales to be learned. Previous studies using multi-scale features in clinical applications rely on multiple inputs across magnifications with late feature fusion, which does not retain the link between features across scales while the inputs are dependent on arbitrary, manufacturer-defined magnifications, being inflexible and computationally expensive. In this paper, we propose the Multi-scale Pyramidal Network (MSPN) , which is plug-and-play over attention-based MIL that introduces progressive multi-scale analysis on WSI. Our MSPN consists of (1) grid-based remapping that uses high magnification features to derive coarse features and (2) the coarse guidance network (CGN) that learns coarse contexts. We benchmark MSPN as an add-on module to 4 attention-based frameworks using 4 clinically relevant tasks across 3 types of foundation model, as well as the pre-trained MIL framework. We show that MSPN consistently improves MIL across the compared configurations and tasks, while being lightweight and easy-to-use. 1 Introduction Multiple-instance learning (MIL) is the standard formulation to solve computational pathology (CPath) problems in a weakly-supervised manner. MIL overcomes high computational costs imposed by gigapixel whole-slide images (WSIs) by identifying the most salient instances ( 16 ) . Typically, WSIs are background removed and tiled into patches, each patch is encoded with a feature encoder, and then these patch-level encodings are concatenated to form slide-level features. MIL is recently used to solve challenging tasks such as classification of biomarkers and mutational signatures ( 12 ; 11 ; 28 ) , and prognosis prediction ( 8 ; 36 ; 20 ; 42 ) . WSIs are usually pyramidal files in which images acquired at multiple rigid resolutions defined by slide scanner hardware are stored. Pathologists assess tissue using non-oil immersion objective lenses of varying magnification from 1.25 \times to 40 \times , allowing them to appreciate differing but equally important features ( 29 ) ; high magnifications allow fine cellular features to be resolved whilst lower magnifications allow essential global tissue architecture to provide context ( 32 ; 5 ) . In CPath, most studies that use single-scale images tile patches at a high magnification equivalent to objective lenses of 20 \times or 40 \times ( 25 ; 8 ; 7 ; 22 ) , while the combined utilisation of the images at 5 \times and 10 \times magnification occurs in multi-scale studies. Figure 1 : Different schemes for multi-scale MIL. (a), (b) Either concatenation or cross-scale attention rely on multiple inputs ( 10 ) , and magnifications are fixed in preprocessing. (c) The proposed MSPN consists of multiple CGNs that derive coarser guidances with adjustable scales from high magnification features. However, current multi-scale approaches fail to leverage the full informational value of pyramidal images and being inefficient. Our work seeks to address these deficiencies. Attention-based methods typically handle multi-scale data through concatenation (Figure 1 a, 27 ; 40 ; 23 ) or cross-scale attention (Figure 1 b, 14 ; 10 ). This necessitates separate feature extraction for each resolution as rigidly and arbitrarily defined by scanner hardware. Similarly, transformer-based approaches like HAG-MIL ( 37 ) employ multiple integrated attention transformers (IATs) to integrate representations for each resolution, with associated high computational cost. Thus, the ability of models to use information provided at additional and user-defined scales in the manner a pathologist does is restricted. Further, even when information from the limited manufacturer-predetermined scales is available, there is no assurance that all scales contribute to model performance nor that the link between features evident at different scales is retained, which differs from a pathologist that usually assess cases using diverse lens range available to ensure safe practice. Finally, these existing methods results in either inflexibility of multiple rigid inputs or complex and expensive architecture that such inefficiencies do not fit the current state where foundation models (FMs) already provide good feature representations to boost lightweight models ( 35 ; 6 ; 34 ; 38 ; 24 ; 3 ) . We conceptualise a method to address these limitations by omitting predetermined multi-scale inputs at different magnifications, instead deriving multi-scale information at user-defined magnifications using only the high magnification input. In previous studies, knowledge distillation has been used to pre-train/fine-tune foundation models and align the features extracted from high magnification patches with low magnification patches ( 5 ; 33 ) . Going further, we explore if the use of high magnification features from foundation models can be directly aggregated to represent low magnification features, to the improve performance of MIL downstream tasks without distillation and extra supervision. In this paper, we introduce the Multi-scale Pyramidal Network (MSPN), a lightweight add-on module for attention-based MIL frameworks that allows flexible and progressive multi-scale training (Figure 1 c, 2 ). The MSPN consists of a sequence of residual connected Coarse Guidance Networks (CGNs) where each CGN serves the following functions: (1) creation of coarse-grids using high magnification coordinates, (2) remapping the patch-level features into each grid, (3) aggregation of features in each grid, and (4) generation of coarse guidances using convolution. The magnification of coarse guidances can be flexibly determined by modifying the grid size in each CGN. We demonstrate the efficacy of MSPN using 4 attention-based frameworks as main baselines, namely ABMIL ( 16 ) , DSMIL ( 21 ) , CLAM-SB, and CLAM-MB ( 25 ) , by comparing different implementations: their original implementations, multi-scale with concatenation, multi-scale with cross-scale attention ( 10 ) , and multi-scale with MSPN. ABMIL pre-trained on FEATHER-24K is further compared ( 30 ) , also following the above settings. Performance is benchmarked on 4 tasks (3 biomarker predictions and 1 prognosis prediction) across 2 high quality real-world datasets using features from 3 different pre-trained foundation models. We show that MSPN introduces consistent performance improvements by an average of 2.5% under the diverse combinations of MIL frameworks, foundation models, and multi-scale methods while staying lightweight compared to the current multi-scale methods. Our main contributions are: (1) introducing grid-based remapping, which leverages the high magnification features to create coarse feature map that retains spatial information; (2) proposing CGN for coarse guidance generation using the created coarse features maps and MSPN as a plug-and-play module that consists of multiple CGNs under different scales for progressive multi-scale WSI analysis; and (3) evaluating our method in various settings that demonstrate its robustness with respect to different foundation models and the pre-trained MIL models whilst being lightweight and easy-to-use. 2 Related Work 2.1 MIL in CPath MIL is the typical formulation to solve current CPath challenges. With the emergence of pre-trained specialised foundation models, the overall performance of light weight models ( 16 ; 21 ; 25 ) has improved considerably ( 3 ) . Meanwhile, a previous study has shown that MIL is transferable ( 30 ) , demonstrating the potential for applying a pre-trained MIL on downstream tasks. However, many of the current MIL frameworks accept input from only one magnification which causes inflexibility when multi-scale input is demanded for greatest informational yield. A generalisable multi-scale module for MIL in the foundation model and transferable MIL era has not yet been explored. 2.2 Multi-scale MIL in CPath The leveraging of multi-scale information was explored before MIL became the typical formulation for CPath, with multiple convolutional neural networks (CNNs) used to process input from different scales ( 4 ; 1 ) . This demonstrated that information from different magnifications provided diverse and useful feature representations. Despite its early exploration, the usage of multi-scale learning is still little examined in recent CPath studies, especially those using MIL ( 10 ; 23 ) . In MIL, multi-scale CPath is designed mainly under two schemes, either (1) concatenating multi-scale features or (2) cross-scale attention. The concatenation variation is used as an easy method of late fusion due to its technical simplicity ( 27 ; 23 ) ; similar to the application of multi-scale learning under CNN, a separate module is required to process features under different magnifications. Alternatively, cross-scale attention represents another way of earlier feature fusion (Figure 1 b) that maintains separate MIL networks for encoding and then applies attention to weight importance among the scales ( 10 ) . HAG-MIL ( 37 ) adopted a similar scheme but uses multiple integrated attention transformers (IATs) to integrate instance representations into a bag representation for each resolution. However, these methods require multiple inputs or multiple transformers, resulting in much higher computational costs. TransMIL ( 31 ) applied the pyramid position encoding generator (PPEG) to encode features under different granularities by processing the reshaped slide-level features with multi-size kernels. Although this provides features under different receptive fields, it improves the adaptability of positional encoding ( 18 ; 31 ; 9 ) when the number of patch tokens vary between different WSIs rather than providing a multi-scale design allowing coarse-level feature representations to be explicitly learned and interpreted ( 10 ) . Moreover, the PPEG restores features into a square map for convolution, distorting the architectural relationships present in the tissue. As such, TransMIL technically learns features with multi-kernels but does not represent a multi-scale architecture. To address these limitations, we propose a plug-and-play module that requires only the highest magnification input, the same data used for training any MIL, to derive spatially reasonable coarse feature maps that enable simple convolutional operation to reduce complexity, while the number of scales and the field-of-view of each scale can be flexibly chosen, and notably learns from coarser scale to finer scale in a progressive manner. Figure 2 : Overview of the Multi-scale Pyramidal Network (MSPN) in the universal MIL pipeline. (1) The MSPN consists of k coarse guidance networks (CGN) with residual connections and each CGN serves to generate guidances at different coarser scales. (2) The coarse grids for guidances are determined by the selected field-of-view (FOV) using coordinates in the original high magnification, retaining the aspect ratio. No extra pre-processing is needed since grid-based remapping is used for mapping high magnification features into the proper coarse grids. (3) Features in each grid are aggregated to form a coarse feature map for the CGN, and a sigmoid gate is learned to score the high magnification patches using the coarse feature map. 3 Methodology 3.1 Formulation of MIL MIL is commonly formulated as a binary classification problem. Given a bag with N instances that X=\{x_{1},x_{2},...,x_{N}\} , MIL is trained to predict the bag-level label Y\in\{0,1\} with instance-level labels \{y_{1},y_{2},...y_{N}\} unknown. Y=\begin{cases}0,&\text{iff}\sum_{n}y_{n}=0,\\ 1,&\text{otherwise.}\end{cases} (1) Practically, MIL can further be extended to perform multi-class classification in tasks such as subtyping ( 31 ; 19 ) and prognosis prediction ( 7 ; 8 ) . The current CPath pipeline typically includes two steps: (1) frozen feature extraction and (2) downstream task training. During feature extraction, a foundation model g is used to extract features \mathbf{h}_{n} for each instance, where \mathbf{h}_{n}=g(x_{n})\in\mathbb{R}^{1\times D} with D denotes the feature dimension. The instance-level features are concatenated to form the bag-level feature \mathbf{H}=\{\mathbf{h}_{1},\mathbf{h}_{2},...\mathbf{h}_{n}\}\in\mathbb{R}^{N\times D} . In downstream task training, MIL is applied by aggregating the bag-level features into a bag representation for classification that \varphi(f(\mathbf{H})) , where f denotes the MIL aggregator and \varphi is a classifier with fully connected layers. Our key innovation is to enable flexible and progressive multi-scale learning, by adding our multi-scale proposal network between these two steps to create the multi-scale bag-level feature representation. 3.2 Grid-based Remapping The aim for grid-based remapping is to avoid the need of multi-inputs, and instead utilise the high magnification features \mathbf{H} to derive a coarse feature map that retains the spatial and contextual information. The height and width of the WSI and the patch coordinates are used to create coarser grids and the features inside the grids are aggregated to form coarse-level representations. For a WSI in height and width of H and W containing a bag of patches with coordinates \mathbf{c}=(\mathbf{x},\mathbf{y})\in\mathbb{R}^{N\times 2} and x_{n}\in[0,W],y_{n}\in[0,H] , grid-based remapping is applied to create coarse feature maps using the high magnification bag-level feature \mathbf{H} and the normalised patch coordinates \mathbf{c^{\prime}}=(\mathbf{x^{\prime}},\mathbf{y^{\prime})}\in\mathbb{R}^{N\times 2} that x_{n}^{\prime},y_{n}^{\prime}\in[0,1] . Firstly, a field-of-view (FOV) s is selected to create the pseudo grid. For example, an s=1024 mimics the coarser tile size of 1024\times 1024 , and a grid with H^{\prime}\times W^{\prime} where H^{\prime}=\lceil\frac{H}{s}\rceil and W^{\prime}=\lceil\frac{W}{s}\rceil is created, preserving the WSI’s aspect ratio. The grid coordinates are denoted as (\mathbf{x^{g}},\mathbf{y^{g}}),x_{n}^{g}\in[0,W^{\prime}],y_{n}^{g}\in[0,H^{\prime}] . Subsequently, each \mathbf{h}_{n} is remapped into the grid according to their coordinates. In this step, we index the grid region in a row-major order, which builds a sequence of \mathbf{idx}\in\mathbb{R}^{N} that maps each high magnification patch to its related coarser grid. Next, intra-grid aggregation is applied to obtain the coarse feature map \mathbf{M}\in\mathbb{R}^{1\times D\times H^{\prime}\times W^{\prime}} . The pseudo code of grid-based remapping is described in Appendix A.1 . The theoretical soundness is derived from the formulation of MIL. Given the simplest scenario that meanpooling is adopted as the MIL aggregator, the bag-level representation \mathbf{b} can be written as \mathbf{b}=f_{mean}(\mathbf{H})\in\mathbb{R}^{1\times D} . Considering the m^{th} grid as a small bag X_{m}^{\prime}=\{x_{1},x_{2},...,x_{J}\} where J denotes the number of instances in the grid and J