Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework

Mengmeng Zhang Xingyuan Dai Yicheng Sun Jing Wang, Yueyang Yao Xiaoyan Gong Fuze Cong Feiyue Wang Yisheng Lv · 2025

Read Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework by Mengmeng Zhang Xingyuan Dai Yicheng Sun Jing Wang, Yueyang Yao Xiaoyan Gong Fuze Cong Feiyue Wang Yisheng Lv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02854.txt.

Introduction

Medical image tools often need a person to point at the region of interest before they can outline it. Mengmeng Zhang and co-authors present a framework from 2025 that removes that step for medical scans. Their method builds on the Segment Anything Model, a general image model that normally depends on prompts. The new system learns to prompt itself, working through layers of the image in stages.

Rather than simple location prompts, the model learns abstract prompts that carry richer meaning about the scan. Tests cover tasks such as outlining polyps and skin lesions, and the system holds up across varied scan types. It also works well on data it was not trained on, beating earlier methods by up to about fourteen percent on some hard test sets.

Catalog record

CREATORMengmeng Zhang Xingyuan Dai Yicheng Sun Jing Wang, Yueyang Yao Xiaoyan Gong Fuze Cong Feiyue Wang Yisheng Lv
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework Mengmeng Zhang Xingyuan Dai Yicheng Sun Jing Wang, Yueyang Yao Xiaoyan Gong Fuze Cong Feiyue Wang Yisheng Lv Abstract Although the Segment Anything Model (SAM) is highly effective in natural image segmentation, it requires dependencies on prompts, which limits its applicability to medical imaging where manual prompts are often unavailable. Existing efforts to fine-tune SAM for medical segmentation typically struggle to remove this dependency. We propose Hierarchical Self-Prompting SAM (HSP-SAM), a novel self-prompting framework that enables SAM to achieve strong performance in prompt-free medical image segmentation. Unlike previous self-prompting methods that remain limited to positional prompts similar to vanilla SAM, we are the first to introduce learning abstract prompts during the self-prompting process. This simple and intuitive self-prompting framework achieves superior performance on classic segmentation tasks such as polyp and skin lesion segmentation, while maintaining robustness across diverse medical imaging modalities. Furthermore, it exhibits strong generalization to unseen datasets, achieving improvements of up to 14.04% over previous state-of-the-art methods on some challenging benchmarks. These results suggest that abstract prompts encapsulate richer and higher-dimensional semantic information compared to positional prompts, thereby enhancing the model’s robustness and generalization performance. All models and codes will be released upon acceptance. 1 Introduction Medical image segmentation is critical for clinical diagnosis and treatment planning by enabling precise delineation of anatomical structures and lesions. Due to the traditionally time-consuming and labor-intensive manual annotation by radiologists, there has been an increasing demand for efficient and precise automated annotation solutions. Recently, researchers have made significant progress in the field of medical image segmentation wang2022medical ; chen2024towards ; jiang2024zept . Traditional task-specific small models often require more specialized and intricate designs to adapt to different tasks, significantly increasing the difficulty of deployment and integration with other tasks. The emergence of general-purpose segmentation models, with their powerful generalization capabilities combined with the pre-training and fine-tuning paradigm, allows these large models to be easily adapted to various downstream tasks. Segmentation Anything Module (SAM) sam as a general-purpose segmentation model for natural images, demonstrates exceptional interactive segmentation capabilities due to its training on the SA-1B dataset. However, SAM’s reliance on prompts poses challenges in medical segmentation. Since vanilla SAM relies on positional prompts, whereas medical imaging fundamentally focuses on lesion detection. An ideal medical segmentation system must automate both lesion localization and segmentation. Some work has attempted to apply SAM to medical image segmentation through fine-tuning, but many studies still rely on manual prompts ma2024segment ; Med-SA . In addition, some methods attempt to compensate for the loss of performance without manual prompts huang2024segment ; ma2024segment by introducing data-specific structures self-prompt_sam ; masksam ; MA-SAM , which in turn limit the segmentation framework’s applicability across diverse imaging modalities. Figure 1 : Previous Methods vs. HSP-SAM. (a)illustrates interactive segmentation methods that rely on manual prompt inputs. (b)shows prior self-prompting approaches still constrained by positional prompts, lacking generalization. (c)presents our proposed self-prompting method that generates abstract task-guided prompts, achieving strong generalizability across modalities. We argue that such adaptations reveal a fundamental distrust in the original SAM image encoder’s ability to generalize across medical imaging tasks. In practice, to achieve network-level inference speed, SAM trains a highly capable image encoder, and constructs extremely lightweight prompt and mask decoders. This design ensures that SAM’s image encoder possesses strong feature extraction and generalization capabilities but makes SAM heavily rely on manual prompt inputs. Therefore, the main obstacle in transferring SAM to medical imaging tasks stems from its strong dependence on manual prompts rather than limitations in its image encoder’s feature extraction capability. We observe that SAM relies on manual prompt inputs to specify segmentation targets, as its original design uses point, box, and mask prompts to provide explicit localization information. However, we raise doubts about this prompting method. Is positional information the only means to guide SAM’s segmentation? We believe this is not the case. At its core, SAM requires task guidance, and positional prompts are only one possible form, which may not be the most effective. While positional prompts offer the most direct form of task guidance, it does not align with the requirements of batch automated segmentation. In automated scenarios, guiding the model through generalized task descriptions—rather than providing target-specific positional prompts—is a more suitable and scalable solution. Considering the current challenges of adapting SAM to the medical domain, we propose Hierarchical Self-prompting SAM(HSP-SAM) to generate abstract task-guided prompts by self-prompting. This enables a prompt-free, generalizable segmentation framework for medical imaging. Fig. 1 illustrates the difference between methods relying on positional prompts and our proposed abstract self-prompting approach. Specifically, we design the corresponding prompts for the image encoder and mask decoder, forming Q&A prompt pairs. These paired prompts are derived through mapping transformations, guiding the encoder to discover key features and assisting the decoder in interpreting critical information. Through this mechanism, the Q&A prompt pairs move beyond focusing on individual segmentation targets within a single input and instead capture a broader understanding of the overall segmentation task, providing more generalized task guidance for the model. Additionally, we designed a U-shaped structure to fuse high and low-dimensional features, enhancing the model’s segmentation performance. The overall architecture of HSP-SAM is depicted in Fig. 2. We conducted extensive experiments across diverse medical imaging modalities. The results demonstrate that HSP-SAM achieves superior robustness and generalization, consistently outperforming other SAM-based medical segmentation models and even surpassing some specialized models on challenging datasets. In summary, our contributions are as follows. • Our findings suggest that SAM depends primarily on "task-guided prompts" rather than solely on "positional prompts." • We find that abstract task-guided prompts, compared to positional prompts, enable the model to achieve a higher-level understanding of segmentation tasks, leading to improved generalization performance. • We propose a self-prompting approach based on Q&A prompt pairs to generate abstract task-guidance prompts, enabling the prompt-free transfer of SAM to medical image segmentation. 2 Related Work Our method builds upon prompt learning and incorporates elements from traditional segmentation models. We also conduct extensive comparisons with SAM-based medical segmentation methods. Therefore, this section briefly reviews prompt learning, traditional segmentation approaches, and SAM-based segmentation methods. 2.1 Prompt Learning Prompt Learning was initially proposed in the NLP field ding2021openprompt ; gao2020making ; jiang2020can and later adopted in vision-language (V-L) and vision-only models zhou2022conditional ; yan2023prompt ; wang2024mcpl ; wang2024vilt . Originally, Prompt Learning refers to the method of manually designing or automatically learning prompts during fine-tuning to help text prompts, serving as instructions, better assist the text encoder in understanding tasks. This approach primarily involves replacing part of the original input text prompt yang2024text ; coop or adding additional prompts to the existing text prompt Wang_2022_CVPR ; lei2024prompt . As a Parameter Efficient Fine-Tuning (PEFT) method, prompt Learning is characterized by its simplicity and efficiency, leading to its gradual adoption in other domains. Moreover, the role of prompt learning continues to expand. For instance, MaPLe khattak2023maple facilitates the alignment between the text and vision branches in V-L models through prompt learning, enabling CLIP to efficiently align text and images in downstream few-shot tasks. Inspired by MaPLe, our work also employs prompt learning to establish Q&A prompt pairs between the encoder and decoder, thereby helping HSP-SAM adaptively discover abstract

Read the complete text