Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation

Qing Xu, Wenting Duan, Zhen Chen (🖂) · 2025

Read Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation by Qing Xu, Wenting Duan, Zhen Chen (🖂) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06740.txt.

Introduction

Tissue regions and cell nuclei in pathology slides are segmented together, each task helping the other, by Qing Xu and two co-authors in a 2025 medical imaging paper. Analyzing tumor tissue and nuclei shapes guides cancer study, yet earlier methods handled the two tasks separately and missed how they inform each other.

Their framework, Co-Seg, introduces mutual prompting. A region-aware prompt encoder supplies high-quality hints about semantic regions and individual nuclei. A mutual prompt mask decoder then uses cross-guidance so both tasks keep consistent context while computing their masks. On the PUMA dataset, the system surpassed leading methods at semantic, instance, and panoptic segmentation of tumor tissue and nuclei alike. Code is released publicly. Working on both scales at once, the authors argue, gives a fuller picture of the tumor environment.

Catalog record

CREATORQing Xu, Wenting Duan, Zhen Chen (🖂)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation Qing Xu, Wenting Duan, Zhen Chen (🖂) Abstract Histopathology image analysis is critical yet challenged by the demand of segmenting tissue regions and nuclei instances for tumor microenvironment and cellular morphology analysis. Existing studies focused on tissue semantic segmentation or nuclei instance segmentation separately, but ignored the inherent relationship between these two tasks, resulting in insufficient histopathology understanding. To address this issue, we propose a Co-Seg framework for collaborative tissue and nuclei segmentation. Specifically, we introduce a novel co-segmentation paradigm, allowing tissue and nuclei segmentation tasks to mutually enhance each other. To this end, we first devise a region-aware prompt encoder (RP-Encoder) to provide high-quality semantic and instance region prompts as prior constraints. Moreover, we design a mutual prompt mask decoder (MP-Decoder) that leverages cross-guidance to strengthen the contextual consistency of both tasks, collaboratively computing semantic and instance segmentation masks. Extensive experiments on the PUMA dataset demonstrate that the proposed Co-Seg surpasses state-of-the-arts in the semantic, instance and panoptic segmentation of tumor tissues and nuclei instances. The source code is available at https://github.com/xq141839/Co-Seg . 1 Introduction Medical image segmentation plays a crucial role in clinical applications and has received extensive attention in the research [ 7 , 29 , 10 , 5 ] . Particularly, histopathology image analysis is challenged by the delineation of tissue regions and further separation of individual nuclei within the regions to assess tissue subtypes and tumor grading [ 36 ] . These requirements have led to two significant segmentation tasks in the field, i.e. , tissue semantic segmentation and nuclei instance segmentation. For medical semantic segmentation, existing methods [ 26 , 35 , 24 , 34 ] rely on hierarchical encoding and decoding to enhance multi-scale awareness for precise mask generation. In particular, ViT-based [ 1 ] and Mamba-based [ 17 ] architectures model long-range dependencies to gain global contexts of target regions. For instance segmentation, a series of works [ 6 , 30 , 2 , 10 ] adopted different distance proxy maps to improve the boundary understanding of instances. Recent SAM-based methods [ 3 , 21 , 29 , 5 , 29 , 4 , 33 , 16 , 19 ] have revealed impressive medical semantic and instance segmentation performance by manually providing corresponding task prompts. Despite the advancements, existing methods [ 1 , 31 , 18 , 32 , 8 ] usually focus on the optimization of tissue semantic or nuclei instance segmentation tasks, which can only be supervised by the isolated knowledge. In fact, both tasks are highly correlated as they aim to achieve the adequate perception and understanding of histopathology images. For example, accurately identifying nuclei provides valuable cues for understanding the underlying tissue structures, while tissue segmentation can aid in localizing nuclei. Therefore, this strong interdependence motivates us to develop a collaborative approach that integrates tissue and nuclei segmentation to advance state-of-the-art histopathology image analysis. To overcome this bottleneck, we propose Co-Seg , a collaborative tissue and nuclei segmentation framework that allows semantic and instance segmentation tasks to mutually enhance each other. As illustrated in Fig. 1 (c), the Co-Seg, based on a co-segmentation paradigm, improves segmentation mask quality by capturing contextual dependencies between the two tasks. Specifically, we first devise the region-aware prompt encoder (RP-Encoder) to extract high-quality semantic and instance prompts from the target region of both tasks, guiding segmentation decoding. We further introduce a mutual prompt mask decoder (MP-Decoder), which jointly computes semantic and instance maps by leveraging cross-guidance. This collaborative learning approach achieves contextual consistency, reducing prediction errors in both tasks. Experimental results on melanoma tissue and nuclei segmentation tasks demonstrate that our Co-Seg achieves remarkable performance over state-of-the-art segmentation methods. Figure 1: Comparison of our Co-Seg and existing works in tissue and nuclei segmentation. (a) Two independent networks for tissue and nuclei segmentation. (b) A shared image encoder but separated task decoders for tissue and nuclei segmentation. (c) Our Co-Seg leverages mutual prompts for collaborative tissue and nuclei segmentation. Figure 2: (a) The overview of the proposed Co-Seg framework for collaborative tissue and nuclei segmentation, consisting of (b) RP-Encoder and (c) MP-Decoder. Co-Seg fully exploits complementary information by leveraging mutual prompts. 2 Methodology As elaborated in Fig. 2 , we devise the Co-Seg framework with the co-segmentation paradigm, achieving the mutual optimization between tissue and nuclei segmentation in medical imaging. To accomplish this, we design the RP-Encoder to provide prior constraints to guide both tasks, and the MP-Decoder to cooperatively generate semantic and instance maps through bidirectional information interactions. This integrated approach enables the two tasks to enhance each other, leveraging mutual prompts for improved segmentation accuracy. 2.1 Co-Segmentation Paradigm Existing tissue [ 9 , 13 , 22 ] and nuclei [ 23 , 20 , 3 ] segmentation methods decouple the parameter space of semantic and instance segmentation, disrupting their interdependencies, and degrading the performance. To address this issue, we propose the co-segmentation paradigm that leverages closed-loop bidirectional interaction to realize dual-task collaborative optimization. Specifically, our goal is to train a model f_{\Theta}:x\rightarrow\{y_{1},y_{2}\} , where \Theta=\{\theta_{1},\theta_{2}\} represents the parameters for semantic segmentation \theta_{1} and instance segmentation \theta_{2} , with y_{1} and y_{2} as the corresponding segmentation masks. To model the interdependence between the two tasks, we define their joint prediction of y_{1} and y_{2} as: p(y_{1},y_{2}|x,\theta_{1},\theta_{2})=p(y_{1}|x,\theta_{1})\cdot p(y_{2}|y_{1},x,\theta_{2})=p(y_{2}|x,\theta_{2})\cdot p(y_{1}|y_{2},x,\theta_{1}). (1) Note that Eq. ( 1 ) reveals the mutual dependency between semantic and instance segmentation through intertwined conditional probabilities. It states that p(y_{1},y_{2}|x,\theta_{1},\theta_{2}) , the joint probability of obtaining both segmentations given the image and model parameters, can be decomposed in two symmetrical ways: • p(y_{1}|x,\theta_{1})\cdot p(y_{2}|y_{1},x,\theta_{2}) suggests that once the semantic segmentation y_{1} is known, it directly influences the prediction of the instance segmentation y_{2} . • Similarly, p(y_{2}|x,\theta_{2})\cdot p(y_{1}|y_{2},x,\theta_{1}) implies that knowing the instance segmentation y_{2} affects the outcome of the semantic segmentation y_{1} . Both expressions are mathematically equivalent because they describe the same joint probability from different perspectives, highlighting that each segmentation task provides crucial context that enhances the accuracy of the other. This reciprocal relationship underscores the need for a collaborative approach, where both segmentation tasks are optimized together rather than in isolation, leveraging the full context available from each task to improve overall segmentation results. Translating this relationship to the gradient calculations for \theta_{1} and \theta_{2} , we have: \nabla_{\theta_{i}}\ell_{i}=-\nabla_{\theta_{i}}\mathbb{E}\left[\log p(y_{i}|x,\theta_{i})\right]-\nabla_{\theta_{i}}\mathbb{E}\left[\log p(y_{j}|y_{i},x,\theta_{j})\right]+\nabla_{\theta_{i}}I(y_{1};y_{2}|x), (2) for i,j\in\{1,2\},i\neq j where \nabla_{\theta}I(y_{1};y_{2}|x)\approx\mathbb{E}\left[\nabla_{\theta}\log\frac{p(y_{1},y_{2}|x)}{p(y_{1}|x)p(y_{2}|x)}\right] . This term captures the mutual information between y_{1} and y_{2} , ensuring that both segmentation tasks benefit from shared feature learning. The first term of Eq. ( 2 ) represents the main segmentation loss gradient, while the second term incorporates the implicit gradient of task interdependencies, modeling the complementarity between \theta_{1} and \theta_{2} . This leads to the following optimization update rule: \theta_{1}\leftarrow\theta_{1}-\eta\nabla_{\theta_{1}}\ell_{1},\quad\theta_{2}\leftarrow\theta_{2}-\eta\nabla_{\theta_{2}}\ell_{2}, (3) where \eta is the learning rate. In this way, the co-segmentation paradigm breaks the barrier of isolating gradient flows from each other in multi-task learning, enabling our Co-Seg framework to achieve the bidirectional optimization of tissue and nuclei segmentation. 2.2 Region-aware Prompt Encoder The proposed co-segmentation paradigm requires establishing implicit interdependencies between tissue and nuclei segmentation tasks. To achieve this, we introduce the RP-Encoder that leverages mask prompts to capture task-specific target regions as prior

Read the complete text