π π β π \mathbf{S^{4}M} : Boosting Semi-Supervised Instance Segmentation with SAM
Heeji Yoon, Heeseong Shin, Eunbeen Hong, Hyunwook Choi, Hansang Cho, Daun Jeong, Seungryong Kim Β· 2025
Read π π β π \mathbf{S^{4}M} : Boosting Semi-Supervised Instance Segmentation with SAM by Heeji Yoon, Heeseong Shin, Eunbeen Hong, Hyunwook Choi, Hansang Cho, Daun Jeong, Seungryong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05301.txt.
Introduction
Heeji Yoon and co-authors present S4M, a method that pairs a strong general segmentation model with training on few labeled images, in a 2025 paper. The task is to outline each distinct object in an image, yet labeled data are scarce. Teacher-student setups help, but their guessed labels can be poor when labels are few.
The general model, SAM, outlines well but does not know object classes and can split objects too far. The authors add a teaching step that passes on SAM's sharp outlines while keeping class sense, plus a step that cleans guessed labels. Fresh image mixes built from those labels add more gains. Tests and side studies back the claim of top scores.
Catalog record
| CREATOR | Heeji Yoon, Heeseong Shin, Eunbeen Hong, Hyunwook Choi, Hansang Cho, Daun Jeong, Seungryong Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05301
Browse the full library Β· About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
π π β π \mathbf{S^{4}M} : Boosting Semi-Supervised Instance Segmentation with SAM Heeji Yoon, Heeseong Shin, Eunbeen Hong, Hyunwook Choi, Hansang Cho, Daun Jeong, Seungryong Kim Abstract Semi-supervised instance segmentation poses challenges due to limited labeled data, causing difficulties in accurately localizing distinct object instances. Current teacher-student frameworks still suffer from performance constraints due to unreliable pseudo-label quality stemming from limited labeled data. While the Segment Anything Model (SAM) offers robust segmentation capabilities at various granularities, directly applying SAM to this task introduces challenges such as class-agnostic predictions and potential over-segmentation. To address these complexities, we carefully integrate SAM into the semi-supervised instance segmentation framework, developing a novel distillation method that effectively captures the precise localization capabilities of SAM without compromising semantic recognition. Furthermore, we incorporate pseudo-label refinement as well as a specialized data augmentation with the refined pseudo-labels, resulting in superior performance. We establish state-of-the-art performance, and provide comprehensive experiments and ablation studies to validate the effectiveness of our proposed approach. * * footnotetext: These authors contributed equally. $\dagger$ $\dagger$ footnotetext: Corresponding author. 1 Introduction Instance segmentationβsimultaneously detecting objects and delineating their pixel-level boundariesβis fundamental to applications such as autonomous driving and medical imaging [ 60 , 58 ] . Although fully-supervised methods [ 19 , 6 , 7 , 24 ] have achieved impressive accuracy, their dependence on extensive annotated datasets limits scalability due to the labor-intensive nature of pixel-level labeling. Consequently, recent work [ 51 , 3 , 5 , 16 , 28 ] has explored semi-supervised learning (SSL) approaches that additionally leverage unlabeled images. Typically, these methods generate pseudo-labels using a teacher network trained on a limited labeled dataset, which subsequently guides the training of a student network. However, the restricted availability of labeled data renders the pseudo-labels error-prone, thereby reducing the benefits of incorporating unlabeled images in the teacher-student framework. To address this, recent studies have proposed several strategies, including noise filtering [ 51 ] , the use of auxiliary information (e.g., depth maps [ 5 ] ), and dedicated training stages to stabilize the teacherβstudent framework [ 16 , 3 ] . Despite these advances, the inherent limitations imposed by the small-scale labeled data continue to produce pseudo-labels with significant errors, ultimately impeding the performance of the student network. Figure 1 : Analysis on pseudo-labels by the teacher in a teacher-student framework for semi-supervised instance segmentation. (a) Bottleneck analysis revealing that the primary limitation lies in mask quality rather than classification. Note that class accuracy (CA) is computed on matched pairs with IoU > 0.5, and segmentation quality (SQ) is measured by the standard segmentation quality metric from panoptic quality [ 32 ] . (b) Example failure cases with correct, confident class prediction but inaccurate masks. Recently, large-scale vision foundation models [ 38 , 20 , 39 , 52 ] , pretrained on web-scale datasets, have demonstrated exceptional performance across diverse tasks and exhibit strong generalization without task-specific fine-tuning [ 1 , 2 , 59 ] . In particular, the Segment Anything Model (SAM) [ 33 , 42 ] has gathered significant attention as a prompt-driven segmentation foundation model, capable of predicting fine-grained masks at any granularity, from whole objects to parts and sub-parts, using geometric prompts such as points and bounding boxes. Through training on an unprecedented scale of images, SAM has demonstrated its efficacy and generalization capabilities in various domains [ 29 , 43 , 4 ] , as well as its application in diverse tasks [ 53 , 41 ] . These advancements motivate our exploration of SAM to enhance instance segmentation through knowledge distillation, pseudo-label enhancement, and data augmentation, which are fundamental elements in semi-supervised learning. However, it still faces challenges in directly applying SAM to instance segmentation due to its class-agnostic design [ 33 , 49 ] : instance segmentation inherently requires both mask and class predictions, yet SAM is not designed to generate class-conditioned masks. In this work, we integrate SAM into the semi-supervised instance segmentation framework to address challenges associated with limited labeled data. We first examine the deficiencies of existing semi-supervised approaches [ 3 ] by visualizing pseudo-labels generated by their teacher networks, thereby establishing a basis for incorporating SAM. As illustrated in Fig. 1 , while these networks reliably identify classes, they often fail in precise localization by grouping multiple instances into a single mask, which we refer to as under -segmentation. Although it might appear straightforward to use SAM to separate instances within these pseudo-labels, its class-agnostic design frequently produces masks that capture only fine-grained segments of an object rather than the object as a whole, resulting in over -segmentation [ 46 ] . This underscores the necessity for carefully balancing under- and over-segmentation when integrating SAM into the semi-supervised instance segmentation framework. In this regard, we argue that it is crucial to identify what and what not to learn from SAM for tackling the under- and over-segmentation problem, and propose a novel framework for distilling SAM to improve the teacher and student networks. In specific, we first improve the teacher network trained on a small amount of label data by a novel knowledge distillation objective, which can effectively acquire the fine-grained localization capabilities of SAM while avoiding over-segmentation or hindering semantic recognition. We further enhance our framework by propagating the strong segmentation capability of SAM through pseudo-label refinement and an augmentation strategy designed for instance segmentation. Figure 2 : Overall pipeline of the proposed framework, \mathbf{S^{4}M} . We propose \mathbf{S^{4}M} , a semi-supervised instance segmentation framework that effectively leverages SAM knowledge through three key approaches. First, we improve the teacher network through structural distillation, which distills SAMβs inherent spatial understanding. Then, as the student learns from unlabeled images, we apply pseudo-label refinement based on SAMβs strong segmentation capability, and further enhance training with instance-aware augmentation, ARP, which leverages the improved pseudo-labels. Our framework for boosting S emi- S upervised Instance S egmentation with S AM, or \mathbf{S^{4}M} , establishes state-of-the-art performance in all benchmarks, demonstrating the effectiveness of our approach. We further provide detailed ablations and analysis on our methodology, as well as qualitative comparison with baselines. Our main contributions can be summarized as follows: β’ We incorporate Segment Anything Model (SAM) into the semi-supervised instance segmentation framework, and present our explorations for improving the teacher-student framework with SAM. β’ We carefully design our framework to fully leverage SAM through structural distillation, pseudo-label refinement and data augmentation while avoiding potential drawbacks of directly adopting SAM. β’ We establish state-of-the-art performance across benchmarks, and provide thorough ablations and analysis to validate our approach. 2 Related Work Semi-supervised instance segmentation. Dominant approaches to semi-supervised instance segmentation [ 50 , 16 , 28 , 3 , 5 ] have been based on student-teacher pseudo-labeling, where teacher model generates pseudo-labels for unlabeled images, which are then used by student model for training. In this framework, applying weak and strong data augmentations for generating and predicting pseudo-labels, respectively, effectively utilizes unlabeled data, as proposed in FixMatch [ 47 ] . The semi-supervised instance segmentation task was first introduced by Noisy Boundaries [ 50 ] , which proposed a noise-tolerant mask head to filter noisy student predictions. Polite teacher [ 16 ] employs EMA teacher while filtering out pseudo-labels