BiSeg-SAM: Weakly-Supervised Post-Processing Framework for Boosting Binary Segmentation in Segment Anything Models

1 st Encheng Su, 2 nd Hu Cao ∗, 3 rd Alois Knoll · 2025

Read BiSeg-SAM: Weakly-Supervised Post-Processing Framework for Boosting Binary Segmentation in Segment Anything Models by 1 st Encheng Su, 2 nd Hu Cao ∗, 3 rd Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01452.txt.

Introduction

Encheng Su, Hu Cao, and Alois Knoll boost a broad vision model for medical outlining in a 2025 paper. Drawing exact borders around polyps and skin sores guides cancer care, but doctor drawn borders cost dear, so labels stay coarse.

Their add on tunes the broad model with a small local feature net. A smart box tool turns coarse masks into prompts the model can use, fixing the gap between rough labels and fine output. A scale matching loss and a detail polish step, fed by a few true borders, sharpen edges. Tests on five polyp sets and one skin set top past methods.

Catalog record

CREATOR1 st Encheng Su, 2 nd Hu Cao ∗, 3 rd Alois Knoll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01452

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BiSeg-SAM: Weakly-Supervised Post-Processing Framework for Boosting Binary Segmentation in Segment Anything Models Thanks: ∗ Hu Cao is the corresponding author of this work ([email protected]) 1 st Encheng Su, 2 nd Hu Cao ∗, 3 rd Alois Knoll Abstract Accurate segmentation of polyps and skin lesions is essential for diagnosing colorectal and skin cancers. While various segmentation methods for polyps and skin lesions using fully supervised deep learning techniques have been developed, the pixel-level annotation of medical images by doctors is both time-consuming and costly. Foundational vision models like the Segment Anything Model (SAM) have demonstrated superior performance; however, directly applying SAM to medical segmentation may not yield satisfactory results due to the lack of domain-specific medical knowledge. In this paper, we propose BiSeg-SAM, a SAM-guided weakly supervised prompting and boundary refinement network for the segmentation of polyps and skin lesions. Specifically, we fine-tune SAM combined with a CNN module to learn local features. We introduce a WeakBox with two functions: automatically generating box prompts for the SAM model and using our proposed Multi-choice Mask-to-Box (MM2B) transformation for rough mask-to-box conversion, addressing the mismatch between coarse labels and precise predictions. Additionally, we apply scale consistency (SC) loss for prediction scale alignment. Our DetailRefine module enhances boundary precision and segmentation accuracy by refining coarse predictions using a limited amount of ground truth labels. This comprehensive approach enables BiSeg-SAM to achieve excellent multi-task segmentation performance. Our method demonstrates significant superiority over state-of-the-art (SOTA) methods when tested on five polyp datasets and one skin cancer dataset. The code for this work is open-sourced and available at https://github.com/suencgo/BiSeg-SAM. I Introduction Accurate segmentation is essential for early diagnosis and treatment planning in medical imaging, particularly for colorectal cancer (CRC) and skin cancer. Automated segmentation methods significantly enhance the accuracy and efficiency of lesion detection. CRC has a high global incidence and mortality rate [ 21 ] , while melanoma, a type of skin cancer, poses a major threat to public health. These challenges underscore the need for precise lesion identification to enable early diagnosis and effective treatment. Convolutional Neural Networks (CNNs), such as U-Net [ 19 ] , MSNet [ 29 ] , CaraNet [ 17 ] , UNet++ [ 30 ] , and UACANet [ 14 ] , have brought substantial advancements to segmentation. Their ability to learn complex features has greatly enhanced segmentation accuracy and reliability. Transformer-based models have achieved significant progress in the field of computer vision [ 10 , 5 , 4 ] . In medical image segmentation, many excellent methods are proposed, such as Polyp-PVT [ 7 ] and Swin-Unet [ 12 ] . However, these methods heavily rely on large, precisely annotated datasets, which are costly and time-consuming to acquire. The advent of large models like SAM [ 15 ] and its variants, including polyp-SAM [ 16 ] , Poly-SAM++ [ 3 ] , and BA-SAM [ 23 ] , has set new benchmarks in performance. Yet, the high computational demands of these models and the uncertainty of outcomes highlight the ongoing challenge of requiring high-quality data. To address these limitations, we introduce BiSeg-SAM, a weakly supervised learning network tailored for SAM, optimizing segmentation tasks with limited annotations for binary classification in colorectal and skin cancer imaging. Unlike prior works that employed scribbles for supervision, our approach utilizes weak box supervision within SAM for medical image segmentation. We enhance the weak box supervision method by adapting bounding boxes based on the foreground center points, which has shown promising results. Our Adaptively Global-Local Module incorporates a CNN block to capture local features, improving the model’s ability to process fine details in the image. Additionally, our DetailRefine Module refines coarse segmentation predictions, thereby improving boundary precision and overall segmentation accuracy. While weak supervision reduces the labeling costs by allowing the use of imprecisely labeled data, during testing, we observed that the bounding box generation method, M2B in WeaklyPolyp [ 26 ] , introduced excessive noise when handling multiple foreground objects. This often resulted in large non-foreground areas being included, leading to suboptimal performance on complex binary classification data. Fig. 1: The pipeline for our proposed BiSeg-SAM. It includes: (1) an SAM and CNN Module that integrates local detail information into SAM; (2) an automatic box prompting mechanism via the WeakBox Module; and (3) the DetailRefine Module, aimed at learning clear edge information and richer image features. The main contributions of this work can be summarized as follows: • Introducing BiSeg-SAM, the weakly supervised method designed for SAM in medical binary classification segmentation, enabling automated prompting for the SAM model. • Enhancing bounding box generation and improving dataset generalization through MM2B transformation, which effectively addresses the transition between multiple and single foregrounds. This results in better segmentation performance for tasks involving multiple foreground objects or irregular and small foregrounds. • Integrating coarse learning and fine detail learning with a new training approach. This method initially captures main features and later focuses on boundaries and details, enhancing the model’s ability to accurately segment complex and varied scenarios in binary segmentation tasks. II Method Fig. 1 depicts the entire pipeline of BiSeg-SAM, which consists of three modules. The input image is first resized and processed by the shared weights of the SAM and CNN Module to obtain segmentation results at different scales. These results are then fed into the WeakBox Module to generate coarse masks, which are further processed by the DetailRefine Module to produce the final output with enhanced boundary details. The technical details of these three modules are as follows: Fig. 2: The architecture of the SAM and CNN Module. The image encoder from the SAM is frozen and integrated with a pre-trained CNN block. The output from the CNN block is concatenated with the frozen image encoder’s output. The concatenated features are then passed through a gate before being fed into the mask decoder. Additionally, the bounding box generated during the MM2B block is used as a prompt for the B. II-A Adaptively Global-Local Module This module integrates a CNN block structured on the principles of ResNet-18 [ 11 ] , designed to preprocess the input image and enhance localized features. To adapt the SAM model for medical image segmentation and address its limitations in extracting local features, we propose an improved CNN block. This module (see Fig. 2 ) not only extracts and concatenates local features from the images but also enhances the extraction and integration of these features by introducing a learnable weight coefficient \alpha [ 6 ] to better balance the fusion of these two sets of features. Specifically, let the input image be x\in\mathbb{R}^{H\times W\times C} , where H and W represent the spatial dimensions, and C denotes the number of channels. The input image is processed simultaneously by the CNN block and the SAM encoder. Feature concatenation and weight allocation are then utilized to obtain the feature map: Fig. 3: First, the center point is judged to determine whether it is in the foreground or background, clarifying the number of foreground objects. If it is a single foreground image, the bounding box is directly generated. If it is a multiple foreground image, the maximum and minimum values are used to generate a bounding box that encompasses all

Read the complete text