SGP-SAM: Self-Gated Prompting for Transferring 3D Segment Anything Models to Lesion Segmentation

Zixuan Tang and Shen Zhao · 2026

Read SGP-SAM: Self-Gated Prompting for Transferring 3D Segment Anything Models to Lesion Segmentation by Zixuan Tang and Shen Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22825.txt.

Introduction

Yao and three colleagues adapt a three-dimensional Segment Anything style model to lesion segmentation, where small, irregular targets and extreme foreground-background imbalance defeat direct transfer. Their SGP-SAM framework adds a gating unit that switches on multi-scale fusion only when features need it, plus a Zoom Loss weighting lesion-focused supervision. On liver and brain tumor benchmarks it beats strong SAM-Med3D transfer baselines, improving liver mDice by 7.3 percent over fine-tuning.

Catalog record

CREATORZixuan Tang and Shen Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17015
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22825

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SGP-SAM: Self-Gated Prompting for Transferring 3D Segment Anything Models to Lesion Segmentation Zixuan Tang and Shen Zhao Abstract Large segmentation foundation models such as the Segment Anything Model (SAM) have reshaped promptable segmentation in natural images, and recent efforts have extended these models to medical images and volumetric settings. However, directly transferring a 3D SAM-style model to lesion segmentation remains challenging due to (i) weak spatial representational capacity for small, irregular targets in intermediate features, and (ii) extreme foreground–background imbalance in 3D volumes. We propose SGP-SAM , a self-gated prompting framework for efficient and effective transfer to 3D lesion segmentation. Our key component, the Self-Gated Prompting Module (SGPM) , performs conditional multi-scale spatial enhancement: a lightweight multi-channel gating unit predicts whether the current features require additional multi-scale fusion, and only then activates a Multi-Scale Feature Fusion Block to enrich spatial context. To further address small-lesion learning, we design a Zoom Loss that up-weights lesion-focused supervision by combining Dice and a voxel-balanced focal term. Experiments on MSD Liver Tumor and MSD Brain Tumor (enhancing tumor) show consistent gains over strong transfer baselines based on SAM-Med3D. On MSD Liver Tumor, SGP-SAM improves mDice by 7.3% over fine-tuning. 1 Introduction Promptable segmentation has recently advanced rapidly due to foundation models such as SAM [ 12 ] and its video extension SAM 2 [ 20 ] . In parallel, medical foundation segmentation models (e.g., MedSAM [ 15 ] , MedSAM2 [ 16 ] , and Medical SAM 2 [ 28 ] ) have demonstrated promising generalization across modalities and anatomies [ 1 , 19 ] . Despite this progress, transferring promptable segmentation to 3D lesion segmentation remains difficult [ 1 ] . Lesions (e.g., liver tumors and enhancing brain tumors) typically occupy a very small portion of a volume and exhibit diverse, irregular morphologies; both properties amplify ambiguity under sparse prompts and challenge intermediate representations in a transformer image encoder [ 27 , 22 , 1 ] . Figure 1: Lesion segmentation in medical images is highly challenging due to the small proportion of lesions and their complex structures. The area within the red box represents liver tumor lesions which occupy a small proportion in CT images, while the area within the red box represents the complex-shaped brain enhancing tumor region. Existing medical SAM adaptations mainly follow two directions. One line focuses on improving 2D medical segmentation by pretraining and then applying per-slice prompting or fine-tuning. Although effective in some settings, slice-wise processing does not fully leverage 3D spatial continuity and may require heavy interaction for volumetric annotation [ 23 , 1 , 9 ] . Another line modifies the architecture to process 3D volumes directly (e.g., SAM-Med3D [ 23 ] ) and explores parameter-efficient transfer (e.g., Med-SA [ 26 ] , 3DSAM-adapter [ 5 ] ). However, lesion segmentation still suffers from two practical issue: (1) intermediate features may be spatially insufficient for capturing small and complex targets, especially in early-to-mid encoder layers; (2) extreme class imbalance makes training unstable, causing the model to overfit easy background voxels [ 3 , 1 , 25 , 27 ] . To address these issues, we propose SGP-SAM , a self-gated prompting strategy that improves spatial representation only when needed. Our SGPM is inserted into the 3D image encoder and learns to decide whether a feature map should undergo additional multi-scale fusion. This conditional enhancement enriches spatial context for hard cases while avoiding unnecessary computation on easy cases [ 11 , 8 , 13 ] . In addition, we introduce Zoom Loss to strengthen lesion-focused supervision by combining Dice loss [ 18 ] with a voxel-balanced focal term [ 14 ] and a lesion-size-dependent reweighting. Our contributions: • We propose SGP-SAM , a self-gated prompting framework for transferring 3D SAM-style models to lesion segmentation with improved spatial representation and lesion-focused learning. • We design the SGPM , where multi-channel gating predicts when to activate a Multi-Scale Feature Fusion Block for conditional spatial enhancement inside the 3D image encoder. • We introduce Zoom Loss , which combines Dice Loss and voxel-balanced focal supervision with lesion-size-aware weighting, effectively handling the challenge of small lesion areas in 3D medical images by prioritizing their learning and mitigating the impact of easy background pixels. 2 Related Work 2.1 Promptable segmentation foundation models SAM [ 12 ] introduces a prompt-based paradigm that supports points, boxes, and masks. SAM 2 [ 20 ] extends this idea to images and videos with a streaming memory design. These models enable interactive segmentation with strong generalization on natural images, inspiring medical adaptations [ 15 , 1 ] . 2.2 Medical and 3D extensions of SAM MedSAM [ 15 ] improves medical generalization through large-scale medical data curation and training. For 3D medical imaging, several approaches adapt SAM via architectural changes or parameter-efficient tuning. SAM-Med3D [ 24 ] extends SAM-style components to volumetric data. Med-SA [ 26 ] proposes adapter-based transfer and introduces designs such as space-depth transpose for 3D adaptation. 3DSAM-adapter [ 5 ] addresses holistic 2D-to-3D adaptation for promptable tumor segmentation. Beyond SAM-style adaptation, recent 3D foundation models such as SegVol [ 4 ] and VISTA3D [ 7 ] scale up 3D training and unify automatic and interactive workflows. 2.3 Conditional computation and gating Conditional computation aims to allocate extra capacity to hard inputs while keeping efficiency. Gating and reparameterization tools (e.g., Gumbel-Softmax [ 11 ] ) are widely used to enable differentiable discrete decisions. Motivated by this idea, we introduce gating inside the 3D image encoder to selectively apply multi-scale feature fusion only when intermediate features are predicted to be spatially insufficient. 3 Method 3.1 Overview Figure 2 illustrates SGP-SAM. We build upon a 3D SAM-style backbone (e.g., SAM-Med3D [ 24 ] ) consisting of (i) a 3D image encoder, (ii) a prompt encoder, and (iii) a mask decoder. Given a 3D medical volume \mathbf{X}\in\mathbb{R}^{H\times W\times D} , the image encoder outputs latent features \mathbf{F}\in\mathbb{R}^{H_{d}\times W_{d}\times D_{d}\times C_{d}} . The prompt encoder converts point prompts into sparse embeddings \mathbf{S} and dense embeddings \mathbf{D} , and the mask decoder predicts the final mask \hat{\mathbf{Y}}\in[0,1]^{H\times W\times D} . Our key modification is the Self-Gated Prompting Module (SGPM) , inserted at the beginning of selected transformer blocks in the 3D image encoder. SGPM predicts a gate value and conditionally activates a Multi-Scale Feature Fusion Block (MSFB) to enhance spatial representation. Figure 2: SGP-SAM pipeline. SGPM is inserted into the 3D image encoder. A multi-channel gating unit predicts whether to activate MSFB for multi-scale spatial enhancement. Zoom Loss strengthens lesion-focused supervision. 3.2 Self-Gated Prompting Module (SGPM) SGPM contains: (i) a Multi-Channel Self-Gating Unit that estimates whether intermediate features need additional spatial enhancement, and (ii) an MSFB that performs multi-scale fusion when activated. Multi-Channel Self-Gating Unit Given \mathbf{F}\in\mathbb{R}^{H_{d}\times W_{d}\times D_{d}\times C_{d}} , we compute dimension-wise summary vectors by mean pooling along complementary axes: \displaystyle\mathbf{f}_{X}(x) \displaystyle=\frac{1}{\prod_{i\neq X}D_{i}}\sum_{i\neq X}\mathbf{F}[i_{1},\ldots,i_{X},\ldots,i_{N}]\in\mathbb{R}^{D_{X}}, (1) where X\in\{H,W,D,C\} represents the dimension being aggregated, \mathbf{f}_{X}(x) is the aggregated feature for dimension X , \mathbf{F}[h,w,d,c] is the data over all dimensions, \prod_{i\neq X}D_{i} is the product of the sizes of all dimensions except X , and \mathbb{R}^{D_{X}} is the feature dimension for the corresponding aggregated feature. Each summary is mapped to a scalar key by a small predictor (we

Read the complete text