Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning

Jin Yang, Daniel S. Marcus, Aristeidis Sotiras · 2025

Read Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning by Jin Yang, Daniel S. Marcus, Aristeidis Sotiras (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10784.txt.

Introduction

Jin Yang, Daniel S. Marcus, and Aristeidis Sotiras adapt medical vision foundation models to three-dimensional scans in this 2025 paper. Foundation models promise general skill. Volumetric segmentation remains a weak spot, though. That task outlines organs and lesions through a whole scan. Labeling enough target data by hand is expensive. Randomly chosen fine-tuning samples waste that budget.

Their framework, ASSFT, picks samples actively at test time. Two signals guide the picking. One gauges how much new knowledge a sample carries, and how varied it is. The other gauges how hard the sample's anatomy is to segment. Uncertainty inside the regions of interest supplies that second signal. A selective semi-supervised stage follows. It adds only reliable unlabeled samples. Confidence and closeness to labeled samples do the filtering. Noisy self-training is avoided. Each human annotation is spent where it teaches the most.

Catalog record

CREATORJin Yang, Daniel S. Marcus, Aristeidis Sotiras
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10784

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning Jin Yang, Daniel S. Marcus, Aristeidis Sotiras Abstract Medical Vision Foundation Models (Med-VFMs) have superior capabilities of interpreting medical images due to the knowledge learned from self-supervised pre-training with extensive unannotated images. To improve their performance on adaptive downstream evaluations, especially segmentation, a few samples from target domains are selected randomly for fine-tuning them. However, there lacks works to explore the way of adapting Med-VFMs to achieve the optimal performance on target domains efficiently. Additionally, random selection does not always capture informative samples from target domains, limiting the adaptation performance. Thus, it is highly demanded to design an efficient way of fine-tuning Med-VFMs by selecting informative samples to maximize their adaptation performance on target domains. To achieve this, we propose an Active Source-Free Domain Adaptation (ASFDA) method to efficiently adapt Med-VFMs to target domains for volumetric medical image segmentation. This ASFDA employs a novel Active Learning (AL) method to select the most informative samples from target domains for fine-tuning Med-VFMs without the access to source pre-training samples, thus maximizing their performance with the minimal selection budget. In this AL method, we design an Active Test Time Sample Query strategy to select samples from the target domains via two query metrics, including Diversified Knowledge Divergence (DKD) and Anatomical Segmentation Difficulty (ASD). DKD is designed to measure the source-target knowledge gap and intra-domain diversity. It utilizes the knowledge of pre-training to guide the querying of source-dissimilar and semantic-diverse samples from the target domains. ASD is designed to evaluate the difficulty in segmentation of anatomical structures by measuring predictive entropy from foreground regions adaptively. Thus, instead of considering the informativeness of the whole samples, it allows the querying of samples with high information levels in the region of interest. Additionally, our ASFDA method employs a Selective Semi-supervised Fine-tuning to improve the performance and efficiency of fine-tuning. It identifies samples with high reliability from unqueried ones if Med-VFMs are more likely to generate high-quality pseudo labels for them. These samples with their pseudo labels are combined with queried samples with their annotations to fine-tune Med-VFMs in a semi-supervised manner. To our knowledge, our work is the first one to adapt Med-VFMs to achieve the optimal performance on volumetric medical image segmentation efficiently via active learning and selective semi-supervised fine-tuning. We adapted Med-VFMs to five domains for volumetric medical image segmentation by our method to demonstrate its effectiveness. Our method achieved superior performance on these tasks than other state-of-the-art methods. † † highlights: We are the first one to propose an Active Source-free Domain Adaptation method to adapt Medical Vision Foundation Models to target evaluation domains in volumetric medical image segmentation, achieving high adaptation performance and efficiency. This method employs a novel Active Test Time Sample Query strategy to query informative samples for fine-tuning, thus maximizing their performance. Additionally, it employs a novel Selective Semi-supervised Fine-tuning method to further optimize Medical Vision Foundation Models in a semi-supervised learning. The Active Test Time Sample Query strategy employs two novel query metrics, termed Diversified Knowledge Divergence (DKD) and Anatomical Segmentation Difficulty (ASD). DKD is designed to query samples with large source dissimilarity and intra-domain diversity by measuring their unlearned target knowledge and diversity. Additionally, ASD is proposed to query samples with large difficulty in segmentation of anatomical structure by evaluating the information levels in foreground regions adaptively. We propose a Selective Semi-supervised Fine-tuning method to fine-tune Medical Vision Foundation Models efficiently by leveraging reliable unannotated samples with their high quality pseudo labels and queried samples in a semi-supervised manner. The reliability of these samples are evaluated by the predictive confidence and their sematic distance to annotated samples. We implemented extensive experiments to evaluate the effectiveness of our ASFDA method on five volumetric medical image segmentation tasks. Our method outperformed other state-of-the-art AL and ADA methods. keywords ,Vision Foundation Models ,Domain Adaptation ,Active Learning ,Semi-supervised Learning ,Medical Image Segmentation † † corresponding: Corresponding author 1 Introduction Medical Visual Foundation Models (Med-VFMs) have superior capabilities of interpreting medical imaging and have demonstrated impressive ability in various medical image analysis tasks ( Zhang and Metaxas 2024 ) . These Med-VFMs employ convolutional neural networks or Vision Transformers as basic architectures and are pre-trained on diversified unannotated imaging data via self-supervised learning ( Moor et al. 2023 ; Zhou et al. 2023 ; Pai et al. 2025 ; Wang et al. 2025 ; Zhu et al. 2025 ) . They are applied to non-adaptive downstream tasks for zero-shot predictions, demonstrating superior performance due to the intrinsic knowledge learned from per-training. In contrast, when they are employed for downstream evaluation of adaptive tasks, including classification and segmentation, they demonstrate low performance of zero-shot predictions on new domains. Specifically, when Med-VFMs are employed for segmenting target organs in unseen domains, they are incorporated with a newly initialized decoder with the symmetric architecture to generate an encoder-decoder segmentation network. Thus, this segmentation network has not been fine-tuned to capture target extrinsic features via fully supervised training, and they are not capable to accurately recognize extrinsic patterns of target organs. To boost the performance of Med-VFMs on adaptive downstream evaluation, especially segmentation, some samples from target evaluation domains are selected for fine-tuning, equipping Med-VFMs with the capabilities of recognizing extrinsic target patterns. Random selection is usually implemented to achieve it ( Pai et al. 2025 ) . However, fine-tuning Med-VFMs by randomly selected samples cannot ensure to maximize the performance and efficiency of fine-tuning since informative samples cannot always be selected. To address this limitation, active learning (AL) can be implemented to select the most informative samples from target domains for model optimization iteratively, thus maximizing the model performance with the minimal query budgets ( Settles 2011 ) . These AL methods are employed to query informative samples by designing different query strategies to evaluate the predictive uncertainty and sample diversity ( Siddiqui et al. 2020 ; Wu et al. 2021 ; Xie et al. 2022 ; Li et al. 2023a ; Yang et al. 2024 ) . When these AL methods are employed to adapt models across domains, these Active Domain Adaptation (ADA) methods utilize query strategies to select the most informative samples from target domains for optimizing them on these domains ( Ning et al. 2021 ; Du and Li 2023 ; Zhang et al. 2024 ) . However, applying these AL and ADA methods to actively adapt Med-VFMs to target domains in downstream tasks have three limitations. First, these methods lack mechanisms to query the most informative samples by leveraging the knowledge learned from pre-training. These methods are designed to adapt models without pre-training, mostly U-Net Ronneberger et al. 2015 . However, evaluating the sample informativeness by leveraging the pre-training knowledge from Med-VFMs facilitates the quantification of unlearned knowledge in target domains, thus improving the adaptation performance and efficiency. Second, these methods employ query strategies to evaluate information levels on the whole images or volumes. However, measuring image-level informativeness of target samples may misestimate their information levels due to the imbalance between foreground and background regions. Organs of interest take a small foreground region in the whole image compared with background. Thus, evaluating sample-level information

Read the complete text