From Specialist to Generalist: Unlocking SAM’s Learning Potential on Unlabeled Medical Images

Vi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam Hoang-Thien Nguyen , Tianyang Wang , Xingjian Li , Min Xu · 2026

Read From Specialist to Generalist: Unlocking SAM’s Learning Potential on Unlabeled Medical Images by Vi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam Hoang-Thien Nguyen , Tianyang Wang , Xingjian Li , Min Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17934.txt.

Introduction

Vi Vu and co-authors ask whether a strong general vision model can learn medicine from unlabeled scans. Their 2026 preprint is about the Segment Anything Model, or SAM. SAM generalizes well on plain images, but medical images differ a lot from its training world. Labels for scans are scarce, and light fine-tuning cannot use the unlabeled bulk. Older models like U-Net, by contrast, shine in semi-supervised medical learning.

The team explores using that U-Net strength to guide a tuned SAM. In effect, the specialist helps the generalist learn from scans no one has marked. If it works, hospitals could adapt SAM with far less hand labeling. The work speaks to the wider push for foundation models in care.

Catalog record

CREATORVi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam Hoang-Thien Nguyen , Tianyang Wang , Xingjian Li , Min Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Specialist to Generalist: Unlocking SAM’s Learning Potential on Unlabeled Medical Images Vi Vu, Thanh-Huy Nguyen, Tien-Thinh Nguyen, Ba-Thinh Lam Hoang-Thien Nguyen , Tianyang Wang , Xingjian Li , Min Xu Abstract Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM , a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM’s adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist–generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM . † † address: 1 Carnegie Mellon University, USA 2 Industrial University of Ho Chi Minh City, Vietnam 3 University of North Carolina at Charlotte, USA 4 University of Alabama at Birmingham, USA 1 Introduction Foundation models have reshaped image segmentation through large-scale pre-training, with the Segment Anything Model (SAM) [ 7 ] showing impressive prompt-driven zero-shot performance. However, SAM struggles in medical imaging due to the significant domain gap, scarce annotations, and the instability of tuning a large model with limited labels. Parameter-Efficient Fine-Tuning (PEFT) reduces computation but still relies heavily on labeled data, leaving the abundant unlabeled medical images underutilized. Meanwhile, conventional architectures such as U-Net have shown, over years of semi-supervised research, that unlabeled data can be a powerful asset. Techniques like consistency regularization and pseudo-labeling allow these models to extract reliable structures from unannotated images. Interestingly, although U-Nets excel at learning from unlabeled data, SAM does not naturally inherit this ability. SemiSAM+ [ 17 ] prompts a frozen SAM using a specialist. KnowSAM [ 4 ] distills SAM into lighter networks. CPC-SAM [ 10 ] enforces cross-prompt consistency between dual SAM decoders. These methods demonstrate various uses of SAM but also introduce architectural complexity or rely solely on SAM’s predictions. It leaves open the question of whether a conventional model could meaningfully help a PEFT SAM learn from unlabeled data. This observation motivates our approach. We propose SC-SAM: Specialist Co-training with SAM , a specialist-generalist collaboration framework that leverages the complementary strengths of U-Net and SAM. Instead of treating SAM as a primary supervisor, we invert the direction: U-Net first learns from both labeled and unlabeled data using standard semi-supervised strategies, and its predictions are then transformed into point-based prompts and pseudo-labels that guide SAM during PEFT. In parallel, SAM generates refined masks that regularize and stabilize U-Net’s training. This forms a bidirectional co-training loop in which U-Net contributes structural domain knowledge, and SAM serves as a high-level semantic regularizer. The two models progressively align, allowing SAM to benefit directly from unlabeled information. Across various medical image segmentation benchmarks, SC-SAM achieves consistent improvements over existing semi-supervised SAM variants and even surpasses strong medical foundation models such as MedSAM and SAM-Med-2D. Our results point to a simple but underexplored principle: Cooperation between conventional specialists and modern generalist models can unlock new pathways for scalable, label-efficient medical image segmentation. Figure 1: Overview of different techniques for incorporating SAM in semi-supervised settings: a) PEFT-SAM, b) Dual-SAM, c) SP-SAM, and d) SC-SAM (Ours) 2 Methodology In semi-supervised segmentation, the training set consists of a labeled set D_{l}=\{(X_{i}^{l},Y_{i}^{l})\}_{i=1}^{N} , and a larger unlabeled set D_{u}=\{(X_{j}^{u})\}_{j=1}^{M} ( M\gg N ). Each image X\in\mathbb{R}^{C\times H\times W} has C=3 for RGB or C=1 for grayscale inputs, and each labeled sample X_{i}^{l} is paired with a ground-truth mask Y_{i}^{l} . We next outline existing techniques for incorporating SAM in semi-supervised settings: PEFT-SAM, Dual-SAM, and SP-SAM, and show how their limitations give rise to our method, SC-SAM. PEFT-SAM: Parameter-efficient Fine-tuning of SAM on labeled set. To adapt SAM on the limited labeled data, we apply PEFT by introducing learnable Adapter layers [ 16 ] into each Transformer block of its image encoder \mathcal{E} , while prompt encoder \mathcal{P} and mask decoder \mathcal{D} are kept trainable with negligible additional parameters as outlined in Fig. 1 (a). Point prompts are sampled from the ground-truth masks of the labeled set and, together with the encoder’s image embeddings, are fed into the decoder to obtain the predictions: \text{points}=\mathrm{Sample}(Y^{l}) P_{SAM}^{l}=\mathcal{D}\big(\mathcal{E}(X^{l}),\mathcal{P}(\text{points})\big), where \mathrm{Sample}(\cdot) randomly selects five foreground and five background points from the ground truth. The supervised loss \mathcal{L}_{\mathrm{seg}} is the mean of Dice and Cross-Entropy losses: \mathcal{L}_{\text{sup}}=\mathcal{L}_{seg}\big(\mathrm{softmax}(P_{SAM}^{l}),Y^{l}\big) Dual-SAM: Leveraging unlabeled data with Dual-branch SAM. Miao et al. [ 10 ] introduced a dual-decoder SAM framework, where each decoder generates prompts and provides cross-supervision to the other. We denote \mathcal{D}_{1} and \mathcal{D}_{2} as two distinct learnable decoders (see Fig. 1 (b)). Phase 1: P^{l,u}_{1,2}=\mathcal{D}_{1,2}\big(\mathcal{E}(X^{u,l})\big) \text{points}_{1,2}=\mathrm{Sample}\big(P^{l,u}_{1,2}\big) Phase 2: P^{l,u}_{1}=\mathcal{D}_{1}\big(\mathcal{E}(X^{u,l}),\text{points}_{2}\big),\quad P^{l,u}_{2}=\mathcal{D}_{2}\big(\mathcal{E}(X^{u,l}),\text{points}_{1}\big) The pseudo-label \hat{P}=\arg\max(P) is obtained from the model prediction for computing the unsupervised loss: \mathcal{L}_{\text{sup}}=\mathcal{L}_{seg}(P^{l}_{1},Y^{l})+\mathcal{L}_{seg}(P^{l}_{2},Y^{l}) \mathcal{L}_{\text{unsup}}=\mathcal{L}_{seg}(P^{u}_{1},\hat{P}^{u}_{2})+\mathcal{L}_{seg}(P^{u}_{2},\hat{P}^{u}_{1}) SP-SAM: Specialist-guided Prompting for SAM fine-tuning. Fig. 1 (c) represents the work of Huang et al. [ 4 ] , which integrates two specialists (or conventional networks) UNet [ 12 ] \mathcal{S}_{1} and VNet [ 11 ] \mathcal{S}_{2} , together with a learnable fusion module \mathcal{F} that fuses their predicted masks to generate mask prompts for SAM: P^{l,u}_{1,2}=S_{1,2}(X^{l,u}) \text{mask}=\mathcal{F}\big(P^{l,u}_{1},P^{l,u}_{2}\big) P^{l,u}_{\text{SAM}}=\mathcal{D}\big(\mathcal{E}(X^{l,u}),\mathcal{P}(\text{mask})\big) Through knowledge distillation, SAM supervises the specialists and allows them to exploit the unlabeled set to produce more robust prompts for SAM fine-tuning. \mathcal{L}_{\text{sup}}=\mathcal{L}_{seg}(P^{l}_{1},Y^{l})+\mathcal{L}_{seg}(P^{l}_{2},Y^{l})+\mathcal{L}_{seg}(P^{l}_{\text{SAM}},Y^{l}) \mathcal{L}_{\text{unsup}}=\mathcal{L}_{KD}(P^{u}_{1},\hat{P}^{u}_{\text{SAM}})+\mathcal{L}_{KD}(P^{u}_{2},\hat{P}^{u}_{\text{SAM}}), where L_{\text{KD}} is the Kullback–Leibler divergence loss. SC-SAM: Specialist Co-training with SAM. PEFT-SAM focuses on fine-tuning SAM using limited labeled data or adopt zero or few-shot tuning, leaving the unlabeled set under-exploited. Dual-SAM leverages unlabeled data through dual-branch cross-supervision; however, under significant domain shift in semi-supervised medical image segmentation, generalist models like SAM are prone to overconfident predictions, leading both branches to step into coupling problem and converge to similar local minima. In contrast, specialist networks (e.g., UNet) trained from scratch can correct SAM’s high-confidence false predictions. Unlike SP-SAM , which exclusively exploits specialist strengths only for robust prompting, our goal is to collaboratively train generalist and specialist networks to form a bi-directional co-training loop (see Fig. 1 (d)). P_{\text{UNet}}^{l,u}=\mathcal{S}(X^{l,u}) (1) \text{points}=\text{Sample}(P_{\text{UNet}}^{l,u}) (2) P_{\text{SAM}}^{l,u}=\mathcal{D}\big(\mathcal{E}(X^{l}),\mathcal{P}(\text{points})\big) (3) However, naive co-training is unstable due to the disparity in convergence behaviors: SAM shows superior early-stage performance to serve as an effective regularizer for UNet but also suffer from early overfitting, while UNet converges slowly yet ultimately adapts better to the target domain. Directly transferring supervision signals from UNet to SAM causes noisy unlabeled signals dominate in the early unsupervised phase and corrupt SAM. We thereby design a sigmoid ramp-up strategy as follows: \mathcal{L}_{\text{sup}}=\mathcal{L}_{\text{seg}}\!\left(P_{\text{UNet}}^{l},Y^{l}\right)+\mathcal{L}_{\text{seg}}\!\left(P_{\text{SAM}}^{l},Y^{l}\right) (4) \mathcal{L}_{\text{unsup}}=\mathcal{L}_{\text{seg}}\!\left(P_{\text{UNet}}^{u},\hat{P}_{\text{SAM}}^{u}\right)+\omega(t)\,\mathcal{L}_{\text{seg}}\!\left(P_{\text{SAM}}^{u},\hat{P}_{\text{UNet}}^{u}\right) (5) \omega(t)=\begin{cases}e^{-\left(1-\frac{t}{T_{\max}}\right)^{2}},&0\leq t\leq T_{\max},\\ 1,&t>T_{\max}\end{cases} (6) where t and T_{\max} are the current iteration and the ramp up length, respectively. The final training objective is: \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{sup}}+\mathcal{L}_{\text{unsup}} (7) 3 Experiment results All experiments were performed on a single NVIDIA RTX 3090 Ti GPU (24 GB RAM) using Python 3.10, PyTorch 2.9, and CUDA 12.8. We used Adam optimizer (initial learning rate 10^{-4} ) for

Read the complete text