MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
Doanh C. Bui, Ba Hung Ngo, Hoai Luan Pham, Khang Nguyen, Maï K. Nguyen, Yasuhiko Nakashima · 2025
Read MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images by Doanh C. Bui, Ba Hung Ngo, Hoai Luan Pham, Khang Nguyen, Maï K. Nguyen, Yasuhiko Nakashima (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13099.txt.
Introduction
Doanh C. Bui and five co-authors teach pathology models new cancer tasks without forgetting old ones. Whole slide images are gigabytes each, so retraining a unified model per task is expensive. MergeSlide treats lifelong learning as model merging: each new task is defined with class-aware prompts, fine-tuned briefly on a vision-language pathology foundation model. It is then merged into the shared model, limiting interference between tasks.
At test time the system may not know which task an image belongs to. Their prompt-aligned inference first identifies the likely task from task-level prompts, then applies that task's class prompts for the final call. Across a stream of six cancer datasets from TCGA, MergeSlide beat rehearsal-based continual learning and zero-shot vision-language baselines. It targets hospital pipelines where new tasks arrive over time.
Catalog record
| CREATOR | Doanh C. Bui, Ba Hung Ngo, Hoai Luan Pham, Khang Nguyen, Maï K. Nguyen, Yasuhiko Nakashima |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images Doanh C. Bui, Ba Hung Ngo, Hoai Luan Pham, Khang Nguyen, Maï K. Nguyen, Yasuhiko Nakashima Abstract Lifelong learning on Whole Slide Images (WSIs) aims to train or fine-tune a unified model sequentially on cancer-related tasks, reducing the resources and effort required for data transfer and processing, especially given the gigabyte-scale size of WSIs. In this paper, we introduce MergeSlide, a simple yet effective framework that treats lifelong learning as a model merging problem by leveraging a vision-language pathology foundation model. When a new task arrives, it is: 1 defined with class-aware prompts, 2 fine-tuned for a few epochs using an MLP-free backbone, and 3 merged into a unified model using an orthogonal continual merging strategy that preserves performance and mitigates catastrophic forgetting. For inference under the class-incremental learning (CLASS-IL) setting, where task identity is unknown, we introduce Task-to-Class Prompt-aligned (TCP) inference. Specifically, TCP first identifies the most relevant task using task-level prompts and then applies the corresponding class-aware prompts to generate predictions. To evaluate MergeSlide, we conduct experiments on a stream of six TCGA datasets. The results show that MergeSlide outperforms both rehearsal-based continual learning and vision-language zero-shot baselines. Code and data are available at https://github.com/caodoanh2001/MergeSlide . 1 1 footnotetext: Corresponding author. 1 Introduction Whole Slide Images (WSIs) provide a comprehensive view of tissue at the cellular level, which is essential for cancer diagnosis and prognosis using digital computational pathology tools and protocols [ 41 ] . In recent years, multiple-instance learning (MIL) models [ 22 , 30 , 35 , 48 ] have become the main approach for WSI-related tasks such as tumor classification and cancer subtyping. However, with the rapid adoption of WSIs and the emergence of new cancer-related tasks, developing computational pathology tools has become increasingly challenging [ 16 ] . Due to their gigabyte-scale size, training deep unified multi-purpose models for WSIs is time-consuming, as it often requires substantial resources for data transfer, preprocessing, feature extraction, and model storage [ 36 ] . Moreover, data privacy concerns across institutions restrict data sharing, hindering the development of multi-purpose models that leverage diverse WSIs from multiple sources. Figure 1 : (a) Rehearsal-based methods ; (b) Task-specific fine-tuning and merging in MergeSlide ; MergeSlide first trains an MLP-free model offline using frozen class-aware prompt embeddings. Then, the weights are merged task-by-task. Figure 2 : Performance comparison of MergeSlide with other continual learning methods. (a) Accuracy as tasks accumulate. (b) Accuracy vs. Forgetting. (c) Accuracy vs. Backward Transfer, both after the final task. An effective solution could be to develop a single core model or tool that is continuous, adaptive, and scalable as new tasks are introduced. A naive approach might be fine-tuning a baseline model task-by-task, but this often leads to catastrophic forgetting [ 32 , 2 ] , which is a phenomenon where a model trained on a new task forgets or performs poorly on previously learned tasks. To tackle this, traditional lifelong learning [ 12 ] (also known as continual learning) approaches are possible [ 19 , 16 ] . These methods are generally categorized into regularization-based and rehearsal-based techniques. Regularization-based models [ 25 , 34 , 24 , 47 , 1 ] typically incorporate mechanisms to prevent the model from deviating too far from previous tasks when learning new ones. In contrast, rehearsal-based models [ 7 , 4 , 5 , 19 ] relax this constraint by allowing the model to revisit a small buffer containing samples from previous tasks. As an early attempt to apply lifelong learning to WSIs, ConSlide [ 19 ] introduced a breakup-and-recognize strategy to diversify buffered WSIs. Later on, with the emergence of pathology vision-language foundation models (VLMs) [ 29 , 20 , 28 , 13 ] that align images with diagnostic texts, [ 18 ] demonstrated that such models can effectively boost downstream WSI performance through vision-text interaction. Building on this, [ 16 ] employed class prompts as query prototypes to enhance lifelong learning using foundation models. Nonetheless, we argue that these models have certain drawbacks: Rehearsal-based methods often outperform others but \raisebox{-0.9pt}{1}⃝ pose a risk of data leakage as well as additional memories due to the use of buffers. \raisebox{-0.9pt}{2}⃝ The implementation of these models typically assumes that tasks have the same number of classes, and that the maximum logits for classification must be predefined regardless of the number of incoming tasks , which is impractical. In practice, the number of classes should be defined flexibly by domain experts rather than constrained by the method. Finally, \raisebox{-0.9pt}{3}⃝ these models are sensitive to the order of tasks ; that is, their performance is not guaranteed to remain consistent when the task order changes, as their training schemes heavily rely on interactions with previous parameters or WSIs stored in a buffer. In another line of research, model merging methods [ 21 , 43 , 37 ] have shown that it is possible to merge parameters from individually trained models across different tasks by leveraging task vectors [ 21 ] , which measure the differences between each task and a well-pretrained base model that captures general knowledge. This direction is promising, as it minimizes or avoids interference with parameters across tasks during training. Nonetheless, its performance relative to continual learning models and its applicability to computational pathology remain underexplored, due to the previous lack of pathology foundation models. In this paper, we introduce MergeSlide , a framework that enables efficient lifelong learning for whole slide image (WSI) analysis. An overview of MergeSlide and its comparison with rehearsal-based continual learning methods are shown in Fig. 1 . Simply put, we treat lifelong learning on WSIs as a model merging process. When a new task emerges, we: 1 design class-aware prompts to effectively describe the target cancer subtypes and obtain their embeddings using a pathology vision–language model ; 2 fine-tune for a few epochs using an MLP-free backbone with randomly sampled patches , making it less time-consuming to develop a unified, multi-purpose WSI model. The resulting task-specific weights are then 3 incrementally merged into a unified model that retains the ability to handle all previously seen cancer-related tasks , eliminating the need for a rehearsal buffer. For inference, the backbone is used solely for slide embedding, and predictions are obtained via dot-product similarity with task-level and class-aware prompt embeddings. Thanks to this design, MergeSlide addresses the following disadvantages: \raisebox{-0.9pt}{1}⃝ MergeSlide reduces the time required to develop a unified, multi-purpose giga-scale WSI model: since fine-tuning is performed individually for each task and then merged into a unified model, it avoids repeatedly training the entire model from scratch when new tasks appear. \raisebox{-0.9pt}{2}⃝ MergeSlide preserves data privacy: it does not require a rehearsal buffer, thereby avoiding potential data leakage. Furthermore, one might argue that with the development of pathology VLMs, zero-shot learning on WSIs [ 31 , 29 , 23 ] could outperform continual learning approaches simply by defining effective prompt prototypes for new tasks. However, in this study, we demonstrate that \raisebox{-0.9pt}{3}⃝ MergeSlide, with just a few finetuning epochs on a pre-trained foundation pathology model, significantly outperforms the zero-shot learning approach , while still preserving both computational efficiency and