Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models

Yankai Jiang, Peng Zhang, Donglin Yang, Yuan Tian, Hai Lin, Xiaosong Wang · 2025

Read Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models by Yankai Jiang, Peng Zhang, Donglin Yang, Yuan Tian, Hai Lin, Xiaosong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02753.txt.

Introduction

A segmentation model trained on one body region usually fails on another, so hospitals end up with many narrow tools. Six researchers present DiffuGTS in 2025, a system meant to outline tumors in new regions without fresh training. It builds attention maps from text prompts over a frozen medical diffusion model. Those maps flag tissue that looks abnormal. The model then repaints flagged areas as if they were healthy, and the difference between the two versions sharpens the outline.

Tests on four datasets covering seven tumor types beat current methods across several zero-shot settings. The authors argue that the hidden features inside large diffusion models already make strong learners for this task.

Catalog record

CREATORYankai Jiang, Peng Zhang, Donglin Yang, Yuan Tian, Hai Lin, Xiaosong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models Yankai Jiang, Peng Zhang, Donglin Yang, Yuan Tian, Hai Lin, Xiaosong Wang Abstract We explore Generalizable Tumor Segmentation, aiming to train a single model for zero-shot tumor segmentation across diverse anatomical regions. Existing methods face limitations related to segmentation quality, scalability, and the range of applicable imaging modalities. In this paper, we uncover the potential of the internal representations within frozen medical foundation diffusion models as highly efficient zero-shot learners for tumor segmentation by introducing a novel framework named DiffuGTS . DiffuGTS creates anomaly-aware open-vocabulary attention maps based on text prompts to enable generalizable anomaly segmentation without being restricted by a predefined training category list. To further improve and refine anomaly segmentation masks, DiffuGTS leverages the diffusion model, transforming pathological regions into high-quality pseudo-healthy counterparts through latent space inpainting, and applies a novel pixel-level and feature-level residual learning approach, resulting in segmentation masks with significantly enhanced quality and generalization. Comprehensive experiments on four datasets and seven tumor categories demonstrate the superior performance of our method, surpassing current state-of-the-art models across multiple zero-shot settings. Codes are available at https://github.com/Yankai96/DiffuGTS . 1 Introduction Generalizable tumor segmentation (GTS) represents a fundamental challenge within medical image analysis Antonelli et al. 2022 ; Liu et al. 2023 ; Chen et al. 2024 ; Jiang et al. 2024a ; Wu et al. 2024c , stemming from both the diversity of tumor types and the variability across imaging modalities. Current AI models for multi-tumor segmentation Liu et al. 2023 ; Chen et al. 2023a ; Ye et al. 2023 ; Zhao et al. 2023 ; Gao 2024 rely heavily on comprehensively annotated training data, limiting their ability to generalize beyond a restricted set of categories. This makes it challenging to address unseen diseases in clinical scenarios where the available training data may not adequately represent the diversity of real-world cases. Figure 1 : We propose DiffuGTS , a novel framework that utilizes and extends the capabilities of a frozen medical foundational diffusion model for advanced zero-shot tumor segmentation across various anatomical regions and imaging modalities. (a) DiffuGTS employs descriptions of both normal and abnormal categories to generate open-vocabulary text-attribution attention maps (b) for anomaly segmentation through cross-modal feature interactions. Furthermore, DiffuGTS leverages the frozen diffusion model to refine anomaly segmentation masks (c) by synthesizing pseudo-healthy equivalents and applying pixel-level and feature-level residual learning, significantly surpassing the performance of existing zero-shot lesion segmentation methods Jiang et al. 2024a ; Huang et al. 2024 ; Jiang et al. 2024b . With development of vision-language models ( e.g . , CLIP Radford et al. 2021 ), some methods Jiang et al. 2024a ; Huang et al. 2024 ; Jiang et al. 2024b have paved the way for unseen tumor segmentation through the zero-shot generalization ability of vision-language alignment between segmentation regions and text prompts. However, in medical imaging, the visual cues of tumors are often subtle and ambiguous. Without a large amount of high-quality image-text pairs for training, segmenting unseen tumor categories using text prompts alone is highly challenging. Furthermore, the vision-language alignment process based on contrastive learning is not necessarily optimal for pixel-level segmentation, as the training objective is not directly optimized for spatial and relational understanding. As a result, these methods are often limited in segmentation quality. Another promising direction towards GTS is tumor synthesis Hu et al. 2023 ; Chen et al. 2024 ; Wu et al. 2024c , which enables label-free tumor segmentation by creating artificial yet realistic medical images. However, current tumor synthesis methods are unable to encompass all tumor types, as simulating tumors with irregular shapes or those not encountered during diffusion model training poses significant challenges Wu et al. 2024c ; Guo et al. 2024 . Consequently, achieving GTS relying solely on synthetic medical data remains an intricate problem. Motivated by the above challenges, we take an innovative approach towards GTS. Our key insight is that, despite the challenges in simulating tumors, a medical foundation diffusion model (MFDM) trained on large-scale data is capable of learning and understanding rich, diverse anatomical structures and organ-specific knowledge Guo et al. 2024 . Moreover, this valuable knowledge is already embedded within its internal representations. Therefore, instead of synthesizing data for model training, we uncover the potential of pre-trained MFDMs as highly efficient semantic feature extractors and demonstrate that their internal image representations can be repurposed, through carefully designed strategies, as effective zero-shot learners for the GTS task. To this end, we propose DiffuGTS , a novel framework with strong zero-shot capabilities that leverages frozen MFDMs for generalizable tumor segmentation. We utilize the cross-modal interactions between the internal visual representations of frozen MFDMs and text prompts for anomaly detection to construct a set of novel anomaly-aware open-vocabulary attention (AOVA) maps to achieve zero-shot tumor detection. The AOVA maps repurpose and recalibrate the diverse anatomical knowledge from MFDMs for unseen anomaly segmentation, improving efficiency and enabling generalization to unseen image modalities. Furthermore, we leverage the generative capabilities of frozen MFDMs to refine the anomaly segmentation masks derived from AOVA maps. We first adopt a training-free latent space inpainting strategy that transforms pathological regions into pseudo-healthy equivalents, conditioned on the anomaly segmentation masks. Then, a novel pixel-level and feature-level residual learning approach generates refined segmentation masks by identifying discrepancies between the original pathological regions and their corresponding pseudo-healthy equivalents, enabling substantial advancements in zero-shot tumor segmentation performance. Extensive experiments across multiple datasets validate the superiority of DiffuGTS ( Fig. 1 ), significantly surpassing previous SOTA methods under various challenging zero-shot settings. In a nutshell, our work offers: (1) an effective and efficient framework with novel designs capable of segmenting unseen tumors across diverse anatomical regions and imaging modalities in a zero-shot manner; (2) superior performance: comparisons with various methods across four datasets demonstrate that our method establishes a new state-of-the-art for generalizable tumor segmentation; (3) in-depth analysis: multiple visualizations imply the strong zero-shot capabilities of AOVA maps for anomaly detection and the efficacy of mask refinement with frozen MFDMs, while key ablation experiments demonstrate the effectiveness of our strategies. 2 Related Work Zero-Shot 3D Medical Image Segmentation. Existing zero-shot 3D medical image segmentation methods fall primarily into two categories: SAM Kirillov et al. 2023 -based methods ( Wald et al. 2023 ; Yamagishi et al. 2024 ; Aleem et al. 2024 ; Shaharabany and Wolf 2024 ) and methods based on vision-language alignment Jiang et al. 2024a ; Jiang et al. 2024b . SAM-based 3D medical image segmentation methods ( Wald et al. 2023 ; Yamagishi et al. 2024 ; Aleem et al. 2024 ; Shaharabany and Wolf 2024 ) demonstrate promising zero-shot performance in segmenting certain organs, particularly larger organs with clear boundaries. However, they primarily focus on organ segmentation and have not been evaluated or proven effective when confronted with unseen lesions that have less defined structures. Recently, some methods Jiang et al. 2024a ; Jiang et al. 2024b , achieve competitive language-driven zero-shot tumor segmentation performance by matching mask proposals with text descriptions through contrastive learning. However, relying only on weak supervision from text descriptions, these methods are limited to compromised zero-shot performance. DiffuGTS takes a huge step further by leveraging frozen

Read the complete text