MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis

Ning Zhu, Xiaochuan Ma, Shaoting Zhang, Guotai Wang · 2025

Read MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis by Ning Zhu, Xiaochuan Ma, Shaoting Zhang, Guotai Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03441.txt.

Introduction

Ning Zhu and co-authors benchmark a cheap-labelling problem in medical imaging: with no labelled examples at all, which images should experts mark up first? Their benchmark, MedCAL-Bench, is the first to test foundation models in a systematic way for this cold-start task. It pits 14 such models and 7 selection methods against each other on 7 datasets, spanning two task types and several imaging kinds and budgets.

Most of the big pretrained models extracted useful features, with the DINO family strongest for outlining structures. Scores for that outlining task varied widely between models, while scores for sorting images bunched together. The best selection method also depended on the task, one leading for outlining and another for sorting. The authors release their code so others can extend the comparison.

Catalog record

CREATORNing Zhu, Xiaochuan Ma, Shaoting Zhang, Guotai Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis Ning Zhu, Xiaochuan Ma, Shaoting Zhang, Guotai Wang Abstract Cold-Start Active Learning (CSAL) aims to select informative samples for annotation without prior knowledge, which is important for improving annotation efficiency and model performance under a limited annotation budget in medical image analysis. Most existing CSAL methods rely on Self-Supervised Learning (SSL) on the target dataset for feature extraction, which is inefficient and limited by insufficient feature representation. Recently, pre-trained Foundation Models (FMs) have shown powerful feature extraction ability with a potential for better CSAL. However, this paradigm has been rarely investigated, with a lack of benchmarks for comparison of FMs in CSAL tasks. To this end, we propose MedCAL-Bench , the first systematic FM-based CSAL benchmark for medical image analysis. We evaluate 14 FMs and 7 CSAL strategies across 7 datasets under different annotation budgets, covering classification and segmentation tasks from diverse medical modalities. It is also the first CSAL benchmark that evaluates both the feature extraction and sample selection stages. Our experimental results reveal that: 1) Most FMs are effective feature extractors for CSAL, with DINO family performing the best in segmentation; 2) The performance differences of these FMs are large in segmentation tasks, while small for classification; 3) Different sample selection strategies should be considered in CSAL on different datasets, with Active Learning by Processing Surprisal (ALPS) performing the best in segmentation while RepDiv leading for classification. The code is available at https://github.com/HiLab-git/MedCAL-Bench . Introduction Deep learning has shown promising performance in various medical image analysis tasks, such as organ or tumor segmentation ( Rokuss et al. 2025 ) and disease classification ( Phan et al. 2024 ) . However, its success relies highly on numerous annotated training samples, which creates a significant barrier in scenarios where labeled data is scarce and annotating large datasets is infeasible ( Jin et al. 2025 ; Wang et al. 2024 ; Kumari and Singh 2023 ; Zhong et al. 2025 ) . Active Learning (AL) has emerged as an effective solution by selecting the most informative samples for labeling to minimize the annotation cost. Traditional warm-start active learning relies on a labeled subset that is randomly selected to iteratively refine the model by selecting valuable samples for further annotation ( Luo et al. 2024 ; Wei, Lin, and Caesar 2024 ) . However, the randomness in selecting initial labeled data often leads to sub-optimal model performance. To address this issue, Cold-Start Active Learning (CSAL) has been proposed to better select the initial subset of images for annotation, which not only can be used for the initialization of traditional multi-round AL, but is also appealing when a single round of sample selection from an unannotated dataset is required ( Jin et al. 2022b ) . While most traditional AL methods leverage uncertainty and diversity for sample selection ( Doucet et al. 2025 ; Hacohen, Dekel, and Weinshall 2022 ) , uncertainty-based methods are hardly applicable for CSAL, as there are no models that have been adapted to the target unannotated dataset for reliable uncertainty estimation. Instead, most CSAL methods use diversity for sample selection, where a feature extractor is firstly trained by Self-Supervised Learning (SSL) on the unannotated dataset, and then used to project the samples into the feature space for diversity-based selection. However, these methods suffer from limited feature representation ability, due to the small size of target dataset for SSL, and the bias of features learned by the pretext task. Figure 1: The workflow of MedCAL-Bench that provides a comprehensive evaluation of both feature extractors and sample selection methods of CSAL on 7 medical datasets for both segmentation and classification tasks. Specifically, we evaluate the effectiveness of foundation models for feature extraction in CSAL, which has not been investigated in existing benchmarks. In recent years, Foundation Models (FMs) have demonstrated exceptional capabilities in extracting semantic and transferable features from images. Pre-trained on large scale and diverse datasets, FMs like CLIP ( Radford et al. 2021 ) , SAM ( Kirillov et al. 2023 ) and DINO ( Zhang et al. 2022 ) have significantly advanced the state of visual representation learning, with generalizable feature representations for unseen image types in downstream datasets, which provides a new type of feature extractors for CSAL. Despite huge potential, FM-based CSAL has been scarcely investigated in current works, and there is also a lack of benchmarks for evaluation of both FMs and sample selection strategies in CSAL. Current works on AL benchmarks mainly focus on warm-start scenarios ( Zhan et al. 2021 ; Holzmüller et al. 2023 ; Werner et al. 2024 ) with only little attention paid to CSAL ( Liu et al. 2023 ; Chen et al. 2022 ) . Moreover, these works often neglect feature extraction methods and only concentrate on comparison of sample selection strategies. To this end, we propose a novel paradigm for CSAL by leveraging pre-trained FMs for feature extraction. Unlike SSL-based CSAL, this approach avoids training the feature extractor for each downstream dataset, which is data- and computation-efficient for sample selection in CSAL. In addition, to comprehensively understand the effectiveness, generalizability and robustness of FMs in CSAL, we present MedCAL-Bench , the first comprehensive benchmark for evaluating pre-trained FM-based CSAL in medical image analysis. We evaluate 14 variants of FMs and 7 CSAL strategies on 7 medical image datasets for segmentation and classification from 6 modalities. Our main contributions are: • We propose a novel CSAL paradigm that leverages pre-trained FMs instead of dataset-specific SSL with powerful feature representation for better sample selection. • We introduce MedCAL-Bench : the first FM-based CSAL benchmark in medical image applications that evaluates 14 FMs and 7 CSAL strategies across 7 datasets under different annotation budget levels. • We present the first comprehensive analysis on the effect of both feature extractor and sample selection rule on CSAL in medical image analysis. Our major findings include: 1) Most FM-driven CSAL methods outperform random selection on the majority of datasets; 2) DINO models prove to be the most effective feature extractors for CSAL in segmentation; 3) Surprisingly, medical FMs do not demonstrate superiority compared with general FMs; 4) The performance of CSAL methods is highly sensitive to FM feature extractor choice; 5) No single CSAL method can consistently achieve top performance across all datasets, and Active Learning by Processing Surprisal (ALPS) performs the best for segmentation, while RepDiv leads in classification. Related Work Cold-Start Active Learning. CSAL can be divided into two categories: uncertainty-based and diversity-based approaches. Typical uncertainty-based approaches, such as ProxyRank-Ent ( Nath et al. 2021 ) and ProxyRank-Var ( Yang et al. 2017 ) , utilize Hounsfield Unit (HU) intensity window or Otsu thresholding ( Liu et al. 2023 ) to quantify uncertainty estimation and select samples with higher uncertainties. However, these methods require prior knowledge of the specific dataset with limited generalizability, and prove to perform worse than diversity-based counterparts ( Doucet et al. 2025 ; Hacohen, Dekel, and Weinshall 2022 ) . Diversity-based approaches aim to select samples from diverse regions to cover the data distribution. Common methods include Coreset ( Sener and Savarese 2017 ) , ALPS ( Yuan, Lin, and Boyd-Graber

Read the complete text