Annotation-Efficient Task Guidance for Medical Segment Anything

Tyler Ward, Abdullah-Al-Zubaer Imran · 2024

Read Annotation-Efficient Task Guidance for Medical Segment Anything by Tyler Ward, Abdullah-Al-Zubaer Imran (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08575.txt.

Introduction

Tyler Ward and Abdullah-Al-Zubaer Imran cut the labeling cost of medical image segmentation in a 2024 paper. Outlining organs in scans guides many treatment choices. Training models to do it normally needs huge sets of hand-drawn outlines, which are slow and costly to make. Their method, SAM-Mix, pairs a general segmentation engine with a helper classifier.

The classifier paints rough heat maps of where the target lies. Those maps steer the segmentation branch while it learns from few labels. On a public liver scan set, the method trained on only fifty labeled slices, a tiny share of the data. It still beat the best rival model on the standard overlap score, with ninety percent fewer training rounds. On a new-domain test, its lead grew much larger. The authors release their code for reuse.

Catalog record

CREATORTyler Ward, Abdullah-Al-Zubaer Imran
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08575

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Annotation-Efficient Task Guidance for Medical Segment Anything Tyler Ward, Abdullah-Al-Zubaer Imran Abstract Medical image segmentation is a key task in the imaging workflow, influencing many image-based decisions. Traditional, fully-supervised segmentation models rely on large amounts of labeled training data, typically obtained through manual annotation, which can be an expensive, time-consuming, and error-prone process. This signals a need for accurate, automatic, and annotation-efficient methods of training these models. We propose SAM-Mix , a novel multitask learning framework for medical image segmentation that uses class activation maps produced by an auxiliary classifier to guide the predictions of the semi-supervised segmentation branch, which is based on the SAM framework. Experimental evaluations on the public LiTS dataset confirm the effectiveness of SAM-Mix for simultaneous classification and segmentation of the liver from abdominal computed tomography (CT) scans. When trained for 90% fewer epochs on only 50 labeled 2D slices, representing just 0.04% of the available labeled training data, SAM-Mix achieves a Dice improvement of 5.1% over the best baseline model. The generalization results for SAM-Mix are even more impressive, with the same model configuration yielding a 25.4% Dice improvement on a cross-domain segmentation task. Our code is available at https://github.com/tbwa233/SAM-Mix . † † address: Department of Computer Science University of Kentucky, Lexington, KY 40506, USA Figure 1 : Illustration of the proposed SAM-Mix multitask learning framework. SAM-Mix combines a ResNet-38 classifier for GradCAM generation, automated prompt generation through mask and bounding box extraction, and a SAM-based segmentation pipeline. Low-rank adaptation (LoRA) is used with a rank ( r ) of 8 to make SAM more parameter-efficient. SAM-Mix uses classification-guided attention to produce accurate segmentation masks ( m_{1} - m_{3} ) with corresponding confidence scores ( s_{1} - s_{3} ). 1 Introduction Segmentation of targets such as organs, lesions, and tumors is one of the most important tasks in the downstream medical imaging pipeline, playing a vital role in enhancing the diagnostic and monitoring capabilities of clinicians. Traditional U-Net [ 1 ] -based architectures have long been one of the most popular and reliable methods for performing medical image segmentation, and have demonstrated strong performance across a variety of imaging modalities. However, these methods are not without their faults and can struggle with issues like the vanishing gradient problem, lack of understanding of global context in large medical images, and high computational complexity and required computing power. Recently, the development of foundation models for segmentation such as the Segment Anything Model (SAM) [ 2 ] has opened up new pathways for addressing some of the challenges associated with traditional fully-supervised models. Due to the vast amount of data it was trained on, SAM and its variants have demonstrated strong generalization ability in both zero-shot and few-shot settings. One of the defining characteristics of SAM is that it is a promptable segmentation model, supporting a variety of different prompt formats, including points [ 3 ] , bounding boxes [ 4 ] , text [ 5 ] , and masks [ 6 ] . There has been much focus on the best way to efficiently and accurately prompt SAM-based methods. Fully automated prompt generation is underexplored and has proven largely ineffective when attempted, due to the high task variability across segmentation tasks, particularly in domain-specific fields such as medical imaging [ 7 ] . As such, the literature surrounding SAM largely focuses on manual or semi-automated prompting, although such prompt generation methods carry several of the same flaws as the manual annotation of training data, in that they can be time-consuming to generate. Additionally, it is not always desirable to increase the parameters in a foundation model such as SAM, as this can lead to issues such as overfitting. A potential solution for the issues present in manual or semi-automated prompting of SAM lies in semi-supervised learning (SSL). In SSL, a large number of unlabeled samples are paired with a limited number of labeled samples to learn more efficiently than in a fully-supervised setting. Furthermore, it has been shown that incorporating SSL in a multitask learning framework, where multiple tasks such as segmentation and classification are trained together so that information from one task can be used to improve the other [ 8 ] . Such multitask learning can improve model generalizability. As an example, in one semi-supervised multi-task model, MultiMix [ 9 ] , a learnable connection was created between an auxiliary classification task and a segmentation branch, where saliency information obtained from the classification branch was used to improve the segmentation predictions. In this case, the use case was a binary segmentation problem to determine the presence of pneumonia from chest X-ray images. Similar approaches are frequently employed in weakly supervised semantic segmentation (WSSS) models, aiming to produce segmentations based on image-level class labels. Class activation maps (CAMs) are popular in WSSS frameworks and have been incorporated with SAM both for post-processing and zero-shot inference [ 10 , 11 ] . However, these methods are vulnerable to noise in the CAMs. Recently proposed S2C [ 12 ] attempts to use SAM to address these vulnerabilities within CAM-based WSSS models. S2C attempts to leverage information from SAM to improve the quality of CAMs. While deemed effective, the inverse of this process, using CAMs to enhance the output of SAM, is underexplored. Additionally, S2C’s training time is a major downside, as it relies on the computationally expensive process of running SAM’s automatic mask generator across the input dataset before the training. Inspired by S2C [ 12 ] andMultiMix [ 13 ] , we propose SAM-Mix , a novel multitask learning framework that leverages an innovative task guidance from image-level labels to enhance SAM-based medical image segmentation. Through the connection between these two branches, SAM-Mix offers improved explainability and segmentation accuracy for the specific needs of medical imaging. Our specific contributions are summarized as: • A novel connection between CAMs and SAM, enhancing the segmentation mask predictions through auxiliary classification-guided attention. • Our SAM-Mix model demonstrates impressive performance even when trained on as few as 5 segmentation-labeled abdominal computed tomography (CT) slices. Additionally, we demonstrate SAM-Mix’s strong generalizability in a cross-domain evaluation. • Thorough experimentation demonstrating our model’s superiority over fully-supervised and semi-supervised models in segmenting liver from the public LiTS and TotalSegmentator datasets. 2 Methods: SAM-Mix To formulate the problem, we assume a data distribution p(\mathit{X},\mathit{Y}) over \mathit{D} where \mathit{X}=\{x_{1},x_{2},\dots,x_{N}\} is a set of abdominal CT slices and \mathit{Y}=\{y_{1},y_{2},\dots,y_{N}\} is the set of corresponding ground truth segmentation maps. A network \mathit{G}_{\phi} is created with parameters \phi such that \mathit{G}_{\phi}(\mathit{X})\rightarrow\mathit{Y} . To aid in segmentation, an auxiliary classification task is manufactured by creating a set of class labels \mathit{C} such that \mathit{C}=\{\max(\mathit{y}_{i})\} where \mathit{y}_{i}=\{0,1\}^{256\times 256} for i=1,\ldots,N . Auxiliary Classification: For the classification branch of our SAM-Mix model, we first define a classifier R based on a ResNet38 [ 14 ] backbone. The primary role of this classifier is to predict class label for each input image while also providing feature maps for GradCAM generation. The classifier consists of an encoder C_{E} and a classification head. The encoder is responsible for extracting a feature map F_{cls}\in\mathbb{R}^{D\times h\times w} from the input image X\in\mathbb{R}^{1\times H\times W} . F_{cls}=C_{E}(X). (1) After obtaining the feature maps from the last convolution layer (

Read the complete text