PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation
Yiheng Zhong †, Zihong Luo †, Chengzhi Liu, Feilong Tang, Zelin Peng, Ming Hu, Yingzhen Hu, Jionglong Su, Zongyuan Ge 🖂, Imran Razzak 🖂 · 2025
Read PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation by Yiheng Zhong †, Zihong Luo †, Chengzhi Liu, Feilong Tang, Zelin Peng, Ming Hu, Yingzhen Hu, Jionglong Su, Zongyuan Ge 🖂, Imran Razzak 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18227.txt.
Introduction
Yiheng Zhong and co-authors present a 2025 paper on outlining organs in medical scans with the Segment Anything Model. That general model segments images well, but its skill drops on medical scans. Pairing text with images can guide it, yet coarse text and the gap between photos and scans weaken the guidance.
Their method, PG-SAM, uses fine-grained text from a medical language model to align the two inputs. An aligner sharpens the shared prior before the model draws its masks. The decoder then fuses features at several levels and refines the mask in steps. It can also learn without prompts. The design aims at accurate multi-organ masks for clinical use, where errors carry real weight.
Catalog record
| CREATOR | Yiheng Zhong †, Zihong Luo †, Chengzhi Liu, Feilong Tang, Zelin Peng, Ming Hu, Yingzhen Hu, Jionglong Su, Zongyuan Ge 🖂, Imran Razzak 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation Yiheng Zhong †, Zihong Luo †, Chengzhi Liu, Feilong Tang, Zelin Peng, Ming Hu, Yingzhen Hu, Jionglong Su, Zongyuan Ge 🖂, Imran Razzak 🖂 Abstract Segment Anything Model (SAM) demonstrates powerful zero-shot capabilities; however, its accuracy and robustness significantly decrease when applied to medical image segmentation. Existing methods address this issue through modality fusion, integrating textual and image information to provide more detailed priors. In this study, we argue that the granularity of text and the domain gap affect the accuracy of the priors. Furthermore, the discrepancy between high-level abstract semantics and pixel-level boundary details in images can introduce noise into the fusion process. To address this, we propose Prior-Guided SAM ( PG-SAM ), which employs a fine-grained modality prior aligner to leverage specialized medical knowledge for better modality alignment. The core of our method lies in efficiently addressing the domain gap with fine-grained text from a medical LLM. Meanwhile, it also enhances the priors’ quality after modality alignment, ensuring more accurate segmentation. In addition, our decoder enhances the model’s expressive capabilities through multi-level feature fusion and iterative mask optimizer operations, supporting unprompted learning. We also propose a unified pipeline that effectively supplies high-quality semantic information to SAM. Extensive experiments on the Synapse dataset demonstrate that the proposed PG-SAM achieves state-of-the-art performance. Our code is released at https://github.com/logan-0623/PG-SAM . † † footnotetext: † Equal contribution. † † footnotetext: 🖂 Corresponding author. 1 Introduction Multi-organ segmentation is a core task in medical image analysis, aiming to accurately separate multiple organs. Segment Anything Model (SAM) [ 21 ] demonstrating its broad application potential [ 13 , 16 , 19 , 24 , 25 , 31 , 33 ] . The success of SAM relies on precise prompts. However, traditional SAM methods are time-consuming, rely on domain expertise, and are prone to human error [ 12 , 23 , 32 , 37 ] . To address these challenges, recent research has focused on prompt-free methods to offer simpler and more efficient segmentation solutions [ 8 , 14 , 34 , 36 , 30 ] . These methods leverage prior information to aid the decoder in better segmentation. Inspired by multimodal learning, recent methods exploit textual information to generate priors for enhancing segmentation [ 1 , 5 , 18 , 23 ] . However, these approaches overlook the fact that the semantic representations derived from visual–language models (VLMs) are largely abstract and non-pixel-level, which can introduce noise. Moreover, the granularity of text descriptions influences the quality of the priors; coarse descriptions result in poor alignment with image features, thereby undermining segmentation accuracy [ 23 ] . In this work, we propose that narrowing the gap between semantic information and pixel-level boundary information can improve segmentation and mitigate noise. Text-visual alignment methods have shown potential in guiding SAM. Prompt learning-based methods [ 11 ] generate visual descriptions by aligning with CLIP [ 27 ] , providing valuable text-visual information. However, these methods suffer from text granularity limitations, which compromise modality alignment. Furthermore, the absence of a dedicated segmentation stage makes capturing fine-grained image details challenging, as illustrated in Fig. 1 (a). In contrast, TP-DRSeg [ 23 ] directly aligns explicit text with CLIP and generates priors to assist SAM, offering finer-grained information. However, it relies on ophthalmology texts verified by experts and faces alignment issues due to the VLM’s reliance on natural-image training, as shown in Fig. 1 (b). Meanwhile, SEG-SAM [ 18 ] employs LLMs to provide text information, leveraging cross-attention to calculate similarity and more efficiently introduce semantic information into SAM, as depicted in Fig. 1 (c). However, it still faces granularity issues and does not fully exploit the zero-shot capability of VLM, potentially affecting its generalization. Figure 1 : Comparison of PG-SAM with other methods: (a) Issues with text granularity. (b) Fine-grained explicit text relies on manual verification and suffers from alignment problems. (c) Faces text granularity issues and lacks the zero-shot capabilities of VLM. (d) Our pipeline improves modality alignment through VLM fine-tuning, providing fine-grained text and the most comprehensive improves. To this end, we propose Prior-Guided SAM (PG-SAM), an efficient pipeline that provides domain-adapted, fine-grained priors and mitigates domain gap issues, as shown in Fig. 1 (d). Specifically, we introduce a fine-grained modality prior aligner that leverages medical LLMs to merge the fluency of large-scale models with the domain-specific expertise of medical professionals, thereby excelling in complex medical scenarios and delivering more detailed, specific semantic information to CLIP. Furthermore, we fine-tune CLIP with Low-Rank Adaptation (LoRA) [ 15 ] for the medical domain, providing more accurate semantic priors. While these priors complement the embeddings of SAM, image features of CLIP focus on abstract semantics, lacking pixel-level details, which may cause simple fusion methods to blur SAM’s embeddings and hinder the retention of fine-grained features. To address this, we design a novel decoder that enhances feature extraction through multi-level fusion, reducing detail loss from noise and promoting knowledge sharing between the CLIP and SAM. Finally, by leveraging an iterative mask optimizer, we dynamically fine-tune the mask weights for each category, enhancing feature expression and enabling better discrimination of small organ details. Extensive experiments on the Synapse dataset demonstrate that the proposed PG-SAM achieves state-of-the-art performance. Overall, our contributions are threefold: (1) We propose a fine-grained modality prior aligner that combines high-level semantics and visual information to generate high-quality priors for all categories; (2) We introduce a novel decoder, which improves mask quality through multi-level feature fusion and a mask fine-tuner; (3) We provide a unified pipeline that simplifies the process while enriching prompt-free methods with medical knowledge. Experimental results demonstrate that our method improves multi-organ segmentation, outperforming state-of-the-art performance on the Synapse dataset [ 22 ] . 2 Methodology 2.1 Overview Fig. 2 illustrates the overview of our method, consisting of three coordinated key components: a Fine-Grained Modality Prior Aligner , as described in Section 2.2 , a Multi-level Feature Fusion , detailed in Section 2.3 , and a Iterative Mask Optimizer , explained in Section 2.4 . PG-SAM first generates fine-grained text descriptions for each image, combining them to generate semantic priors, referred to as the Semantic Guide Matrix, to assist the decoding process. Then the Multi-level Feature Fusion module facilitate knowledge sharing between the text-guided explicit prior and multi-level visual features. Finally, in the iterative mask optimizer, candidate masks are provided for each category via mask tokens, and a mask refiner optimizes these segmentation details. 2.2 Fine-Grained Modality Prior Aligner Figure 2 : Overview of PG-SAM. (a) Illustrates the process by which the fine-grained modality prior aligner generates the Semantic Guide Matrix G ; (b) For multi-level feature fusion, G is integrated with the feature map after multi-level sampling to preserve more detailed features; (c) It outlines the iterative mask optimizer, which dynamically learns convolution kernel parameters via a Hypernetwork and refines the final mask using a dedicated refiner. Figure 3 : Comparison of textual prompts and corresponding heatmaps generated by the Base LLM (left) and the Medical LLM (right) for anatomical images of the spleen , stomach , and pancreas . The Medical LLM provides clinically precise descriptions, yielding more focused and detailed semantic guidance, as demonstrated by the sharper