Multimodal Prototype Alignment for Semi-supervised Pathology Image Segmentation

Mingxi Fu 1 ∗ , Fanglei Fu 1 ∗ , Xitong Ling 1 ∗ , Huaitian Yuan 1 , Tian Guan 1 † , Yonghong He 1 † , Lianghui Zhu 1 † · 2025

Read Multimodal Prototype Alignment for Semi-supervised Pathology Image Segmentation by Mingxi Fu 1 ∗ , Fanglei Fu 1 ∗ , Xitong Ling 1 ∗ , Huaitian Yuan 1 , Tian Guan 1 † , Yonghong He 1 † , Lianghui Zhu 1 † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19574.txt.

Introduction

Outlining regions in pathology slides takes expert hand-labeling, which is slow and costly, and fuzzy borders make the task harder. Mingxi Fu and co-authors present MPAMatch in this 2025 paper, a method that learns from a few labeled slides plus many unlabeled ones.

The method compares pixels against two kinds of prototypes, one built from images and one from text, giving guidance on both structure and meaning. It also swaps in a pathology-trained backbone for feature extraction. Across four gland and cancer data sets, the system beat leading rivals. The authors say text-guided supervision sharpens border decisions where labels are scarcest.

Catalog record

CREATORMingxi Fu 1 ∗ , Fanglei Fu 1 ∗ , Xitong Ling 1 ∗ , Huaitian Yuan 1 , Tian Guan 1 † , Yonghong He 1 † , Lianghui Zhu 1 †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19574

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Prototype Alignment for Semi-supervised Pathology Image Segmentation Mingxi Fu 1 ∗ , Fanglei Fu 1 ∗ , Xitong Ling 1 ∗ , Huaitian Yuan 1 , Tian Guan 1 † , Yonghong He 1 † , Lianghui Zhu 1 † Abstract Pathological image segmentation faces numerous challenges, particularly due to ambiguous semantic boundaries and the high cost of pixel-level annotations. Although recent semi-supervised methods based on consistency regularization (e.g., UniMatch) have made notable progress, they mainly rely on perturbation-based consistency within the image modality, making it difficult to capture high-level semantic priors, especially in structurally complex pathology images. To address these limitations, we propose MPAMatch—a novel segmentation framework that performs pixel-level contrastive learning under a multimodal prototype-guided supervision paradigm. The core innovation of MPAMatch lies in the dual contrastive learning scheme between image prototypes and pixel labels, and between text prototypes and pixel labels, providing supervision at both structural and semantic levels. This coarse-to-fine supervisory strategy not only enhances the discriminative capability on unlabeled samples but also introduces the text prototype supervision into segmentation for the first time, significantly improving semantic boundary modeling. In addition, we reconstruct the classic segmentation architecture (TransUNet) by replacing its ViT backbone with a pathology-pretrained foundation model (Uni), enabling more effective extraction of pathology-relevant features. Extensive experiments on GLAS, EBHI-SEG-GLAND, EBHI-SEG-CANCER, and KPI show MPAMatch’s superiority over state-of-the-art methods, validating its dual advantages in structural and semantic modeling. Fig. 1: Overall process: (a) The framework of Transunet+UNI. (b) The framework of semi-supervised segmentation method. (c) Multimodal prototype learning method. (d) The description of prototype loss calculation. I INTRODUCTION With the continuous advancement of artificial intelligence in the medical field, deep learning has become a key approach for automatic analysis of whole slide images (WSIs). Among these applications, tissue semantic segmentation is one of the core tasks in computational pathology, widely used in scenarios such as disease-assisted diagnosis [ 1 ] and cancer subtyping [ 2 ] . Early models like FCN [ 3 ] and U-Net [ 4 ] demonstrated strong performance on biomedical images, while subsequent methods such as DeepLab [ 5 ] further improved segmentation accuracy through dilated convolutions and multi-scale context awareness. In recent years, the introduction of Transformers has enabled better integration of local and global features such as TransUNet [ 6 ] and Swin-Unet [ 7 ] . However, most existing approaches rely on fully supervised learning paradigms. In pathology, pixel-level annotation requires extensive efforts from pathologists, which greatly limits their feasibility in real clinical scenarios [ 8 ] . To address the challenge of limited annotated data in computational pathology, semi-supervised learning (SSL) has demonstrated remarkable effectiveness by leveraging both scarce labeled and abundant unlabeled samples [ 9 ] . Most state-of-the-art SSL methods adopt the consistency regularization framework, which encourages the model to produce invariant predictions under perturbations of the same input [ 10 , 11 , 12 ] . FixMatch [ 13 ] innovatively combines pseudo-labeling with consistency regularization to construct pseudo-supervised objectives, while UniMatch [ 14 ] further builds upon this by introducing feature-level perturbations and dual-stream augmentation strategies, achieving strong performance in semantic segmentation of natural images. However, conventional SSL segmentation methods often struggle to handle the intrinsic intra-class heterogeneity prevalent in pathological images, limiting their ability to accurately capture structural boundaries [ 15 ] . Prototype learning has emerged as a promising solution by enhancing feature space structuring and promoting global contour awareness across whole histological slides. For instance, Yu et al. [ 16 ] incorporated prototype learning into weakly supervised WSIs analysis via clustering-based prototype construction, substantially improving structural modeling. Liu et al. [ 17 ] further integrated prototype learning with attention mechanisms to quantify instance–prototype similarity, enabling interpretable instance-level predictions. Nevertheless, current prototype-based approaches remain largely constrained to the instance level, with limited exploration of alignment with pixel-level representations in pathology. Nevertheless, most of these semi-supervised learning methods rely solely on visual modality information, overlooking the rich semantic information in pathological text descriptions. In recent years, pre-trained vision-language models such as CLIP [ 18 ] and its pathology-adapted variants (e.g., PLIP [ 19 ] , CONCH [ 20 ] ) have demonstrated great potential in incorporating textual semantic supervision. However, the text encoders in CLIP typically rely on static templates, which are insufficiently flexible to capture task-specific semantics. To enhance the task adaptability of language representations, prompt learning methods such as CoOp [ 21 ] have been proposed to optimize textual embeddings via learnable context prompts. These methods enable class-level semantic guidance without fine-tuning the vision-language backbone and have achieved notable success in tasks like few-shot classification. To address the above limitations in pathological image segmentation, we propose a multimodal prototype and pixel-level label contrastive fusion framework. By constructing cross-modal semantic and visual prototypes, our method enables coarse-grained supervision to enhance global morphological structure awareness and improve segmentation performance. Specifically, we first modify the TransUNet architecture by replacing its original ViT encoder with UNI [ 22 ] , a vision foundation model pretrained on large-scale pathology images, to better capture complex tissue morphology. In handling unlabeled data, we design a prototype-guided prediction module. On the visual side, we perform online clustering to dynamically generate structural prototypes for each class, which are fused with decoder features to refine segmentation. On the textual side, we prompt a large language model (LLM) to generate descriptive phrases for each prototype class, and extract semantic embeddings using a multimodal pathology foundation model (CONCH [ 20 ] ) to form text-guided prototypes. To enable learnability and task adaptability, we further incorporate the CoOp mechanism, allowing the text prototypes to be optimized in an end-to-end manner during training. The main contributions of this paper are as follows: (1) We propose a multimodal prototype-guided semi-supervised segmentation framework that integrates both structural and semantic information by incorporating visual and textual prototypes, thereby improving segmentation accuracy. (2) We design a contrastive learning mechanism that combines coarse-grained (prototype-level) and fine-grained (pixel-level) supervision, and introduce a prototype-based loss function to enhance inter-class separability. (3) In the encoder-decoder architecture of the segmentation model, we replace the ViT module in TransUNet with UNI [ 22 ] , to better capture fine-grained structures such as glandular regions and tissue boundaries in pathological images. II RELATED WORK II-A Semi-supervised segmentation in histopathological image Semi-supervised learning (SSL) enables models to leverage abundant unlabeled data alongside limited annotations, effectively enhancing performance while reducing the dependence on costly pixel-level labels. In recent years, SSL methods have been widely applied in computer vision [ 23 , 24 , 25 , 26 , 27 ] . Two common SSL strategies include consistency regularization [ 28 , 29 , 30 ] and self-training [ 31 , 32 ] . Due to the scarcity and high cost of manual annotation in computational pathology, SSL offers a practical solution for pathological images segmentation. Rashmi et al. [ 33 ] proposes a hybrid strategy for tissue semantic segmentation that combines pseudo-label generation and consistency regularization. Sun et al. [ 34 ] proposes a classification-guided semi-supervised framework that leverages multi-expert collaboration and pseudo-labeling to enhance performance in breast cancer pathological image segmentation. Han et al. [ 35 ] proposed a two-stage semi-supervised segmentation framework that first generates

Read the complete text