Adapting a Segmentation Foundation Model for Medical Image Classification

Pengfei Gu 1 , Haoteng Tang 1 , Islam A. Ebeid 2 , Jose A. Nunez 1 , Fabian Vazquez 1 , Diego Adame 1 , Marcus Zhan 3 , Huimin Li 4 , Bin Fu 1 , Danny Z. Chen 5 · 2025

Read Adapting a Segmentation Foundation Model for Medical Image Classification by Pengfei Gu 1 , Haoteng Tang 1 , Islam A. Ebeid 2 , Jose A. Nunez 1 , Fabian Vazquez 1 , Diego Adame 1 , Marcus Zhan 3 , Huimin Li 4 , Bin Fu 1 , Danny Z. Chen 5 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06217.txt.

Introduction

Pengfei Gu and co-authors adapt a segmentation foundation model for medical image classification in this 2025 paper. The model, SAM, was built to outline objects in images with little prompting. Turning it to diagnosis-style classification is less explored. Their framework uses SAM's image encoder as a frozen feature extractor. Those features carry spatial and context detail from the outline training. A new spatial module then reshapes them for class decisions.

Freezing the encoder avoids heavy retraining costs. The approach borrows SAM's broad visual knowledge for a new task. Medical images differ from daily photos, so adaptation is the crux. The authors report the design and its tests in the full text. Hospitals hold many scans that need fast, cheap triage. A reusable foundation model could serve many such tasks. The work joins a push to reuse big vision models in medicine.

Catalog record

CREATORPengfei Gu 1 , Haoteng Tang 1 , Islam A. Ebeid 2 , Jose A. Nunez 1 , Fabian Vazquez 1 , Diego Adame 1 , Marcus Zhan 3 , Huimin Li 4 , Bin Fu 1 , Danny Z. Chen 5
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adapting a Segmentation Foundation Model for Medical Image Classification Pengfei Gu 1 , Haoteng Tang 1 , Islam A. Ebeid 2 , Jose A. Nunez 1 , Fabian Vazquez 1 , Diego Adame 1 , Marcus Zhan 3 , Huimin Li 4 , Bin Fu 1 , Danny Z. Chen 5 Abstract Recent advancements in foundation models, such as the Segment Anything Model (SAM), have shown strong performance in various vision tasks, particularly image segmentation, due to their impressive zero-shot segmentation capabilities. However, effectively adapting such models for medical image classification is still a less explored topic. In this paper, we introduce a new framework to adapt SAM for medical image classification. First, we utilize the SAM image encoder as a feature extractor to capture segmentation-based features that convey important spatial and contextual details of the image, while freezing its weights to avoid unnecessary overhead during training. Next, we propose a novel Spatially Localized Channel Attention (SLCA) mechanism to compute spatially localized attention weights for the feature maps. The features extracted from SAM’s image encoder are processed through SLCA to compute attention weights, which are then integrated into deep learning classification models to enhance their focus on spatially relevant or meaningful regions of the image, thus improving classification performance. Experimental results on three public medical image classification datasets demonstrate the effectiveness and data-efficiency of our approach. I Introduction Medical image classification is a critical problem in computer-aided diagnosis, requiring high accuracy and robustness. While deep learning (DL) models, such as ResNet [ 1 ] , SENet [ 2 ] , and Swin Transformer v2 [ 3 ] , have achieved remarkable success in this domain, they often face several challenges: (1) Capturing spatially localized features crucial for distinguishing subtle anatomical variations; (2) neglecting valuable segmentation-based information that may provide further insights into the spatial structures of medical images. The Segment Anything Model (SAM) [ 4 ] , a foundation model originally designed for natural image segmentation tasks, offers the potential to bridge this gap with its powerful image encoder that excels at capturing rich features from diverse visual patterns. A number of recent studies have explored customizing SAM for medical image segmentation, by either directly applying SAM [ 5 , 6 , 7 , 8 , 9 ] or fine-tuning SAM for medical images [ 10 , 11 , 12 ] . However, SAM has yet to be fully adapted for medical image classification. SAMAug-C [ 13 ] proposed a new augmentation method that leverages SAM for augmenting raw image inputs to improve medical image classification. However, this method did not fully explore SAM’s capability to capture spatially localized features that provide valuable segmentation-based information. To address this challenge, we propose a new framework to effectively adapt SAM for medical image classification. This work addresses two pivotal questions regarding medical image classification: (I) How to efficiently utilize SAM to extract segmentation-based features? (II) How to effectively integrate such features into DL classification models to enhance their performance? For question (I), we propose to use SAM’s image encoder as a feature extractor to capture spatially localized features enriched with segmentation-based information. Specifically, we extract feature maps from the patch embedding, the first, middle, and last blocks of the Transformer blocks, and the Convolution (Conv) block. These extracted feature maps are then incorporated into different stages of the DL classification models. The weights of the SAM image encoder are frozen during model training, enabling efficient adaptation. For question (II), we introduce a novel attention mechanism called Spatially Localized Channel Attention (SLCA). SLCA is applied to compute the attention weights of the captured features from the SAM image encoder, which are then used to enable the intermediate features of the DL classification models to focus on what is locally (spatially) meaningful. This attention mechanism helps effective incorporation of the extracted features into the DL classification models, guiding them to focus on spatially meaningful regions and leading to more accurate and interpretable predictions. Fig. 1: An overview of our framework. In [C,H,W] , C represents the number of channels, and H and W represent the height and width of a feature map, respectively. For simplicity, the BN and ReLU that follow each Conv layer are not shown. In summary, our main contributions are as follows: (1) We efficiently capture segmentation-based features using the SAM image encoder as a frozen feature extractor. (2) We introduce the SLCA mechanism to compute spatially localized attention weights, effectively incorporating the extracted segmentation-based features into DL classification models, enhancing their focus on spatially relevant regions in input images and improving their performance. (3) We demonstrate the effectiveness and data efficiency of our new approach with extensive experiments on three public medical image datasets. II Methodology Fig. 1 (a) shows the pipeline of our method. Our framework consists of two branches: a DL classification branch (e.g., ResNet152 [ 1 ] ) and a SAM-based feature extraction branch. On the SAM-based feature extraction branch, a frozen SAM image encoder is used to extract features from different blocks (Section II-A ), which are passed to SLCAs (Section II-B ) to compute the attention weights. The computed attention weights are integrated into the corresponding stages of the DL classification branch via a residual connection to help the model focus on spatially relevant local regions in the image (Section II-C ). II-A SAM Image Encoder: A Frozen Feature Extractor The SAM image encoder, a Vision Transformer (ViT) [ 14 ] , is pre-trained with Masked Auto-Encoders [ 15 ] using over 1 billion tasks in 11 million images. It accepts an input image of any size and reshapes it to size 1024\times 1024 . Then the image is converted to sequential patch embeddings with patch size 16\times 16 and embedding size 256. Through 32 Transformer blocks with window attention and a Conv block, the output of the image encoder is of size 64\times 64\times 256 , where 256 is the channel number. The outputs of patch embedding, Transformer blocks, and Conv block are all of size 64\times 64\times 256 . In this paper, to efficiently leverage the SAM image encoder as a feature extractor to extract segmentation-based features of the input image: (I) The weights of the SAM image encoder are frozen during the training stage to reduce computational overhead. (II) There are three different scales of pre-trained image encoders: ViT-h, ViT-l, and ViT-b. We use ViT-h in all our experiments, considering the trade-off between real-time performance and accuracy. By doing so, we ensure that SAM’s pre-trained segmentation capabilities are preserved without the need for additional fine-tuning. II-B Spatially Localized Channel Attention (SLCA) The features extracted by the SAM image encoder contain important spatial information of the image (see Fig. 3 ). It is crucial to utilize these features to enable the DL classification model to focus on spatially relevant regions in the image [ 16 , 17 , 18 , 19 ] . Hence, a pertinent question is: How to effectively utilize the features obtained by the SAM image encoder to enable the DL classification model to focus on spatially meaningful local regions? To address this question, we propose a new attention mechanism called Spatially Localized Channel Attention (SLCA), which enables the model

Read the complete text