Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST
Fuping Wu 1,2, Bartłomiej W. Papież 1,2 · 2025
Read Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST by Fuping Wu 1,2, Bartłomiej W. Papież 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14685.txt.
Introduction
Fuping Wu and Bartłomiej Papież benchmark in 2025 how well large pretrained foundation models classify medical images, using the MedMNIST collection. Such models transfer well to new tasks, but the growing crowd of releases makes choosing one hard. They tested models from convolutional to transformer designs, with both full training and simple linear probing on top.
Pretrained models showed strong potential when moved to medical classification. Extra experiments varied image size and training set size to see what drives performance. The authors draw early, practical lessons for picking models in clinical imaging work. Their study is a guide rather than a final verdict.
Catalog record
| CREATOR | Fuping Wu 1,2, Bartłomiej W. Papież 1,2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST Fuping Wu 1,2, Bartłomiej W. Papież 1,2 Abstract Foundation models are widely employed in medical image analysis, due to their high adaptability and generalizability for downstream tasks. With the increasing number of foundation models being released, model selection has become an important issue. In this work, we study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset. Specifically, we adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks. The results demonstrate the significant potential of these pre-trained models when transferred for medical image classification. We further conduct experiments with different image sizes and various sizes of training data. By analyzing all the results, we provide preliminary, yet useful insights and conclusions on this topic. \midlauthor 1 Nuffield Department of Population Health, University of Oxford, UK 2 Big Data Institute, University of Oxford, UK and Corresponding author † † volume: – Under Review † † year: 2025 † † workshop: Full Paper – MIDL 2025 submission † † editors: Under Review for MIDL 2025 keywords Foundation model, pre-trained model, medical image, classification, benchmark study. 1 Introduction Foundation models, which are pre-trained on large-scale diverse datasets, have achieved rapid advancement in various areas, such as computer vision Awais et al. (2023) , natural language processing Zhou et al. (2023) , and audio signal analysis Huang et al. (2024) . These models have been proven effective in adapting to numerous downstream tasks, including semantic segmentation Wang et al. (2024) , recommendation systems Zhao et al. (2024) , showcasing remarkable generalizability and adaptability Touvron et al. (2023) ; Anil et al. (2023) . In recent years, the success of foundation models has triggered significant research into their application in medical image analysis Zhang and Metaxas (2024) , either by transferring models from other fields Mazurowski et al. (2023) or training models with large-scale medical data Butoi et al. (2023) . While studies have shown the great potential of foundation models in organ segmentation Ma et al. (2024) , clinical reports generation Thawakar et al. () , and medical Q \& A systems Chen et al. (2024) , few have conducted large-scale comparisons of the existing models to provide insights into their performance in specific applications Huix et al. (2024) . In this work, we examine the performance of various foundation models in medical image classification tasks. Although some studies have performed similar comparisons, they were often limited in scope, focusing on small model categories or datasets. For example, Huix et al. (2024) compared eight models across only four medical image classification tasks. Wang et al. (2023) presented a novel dataset with five classification tasks but compared only three models for transfer learning. Woerner and Baumgartner (2024) conducted a benchmark study of few-shot and zero-shot medical image classification with linear probing solely focusing on fine-tuned ResNet-based models. Baharoon et al. (2023) compared DINOv2 Oquab et al. (2023) with nine other models on four disease classification tasks. To provide a comprehensive study of foundation models for medical image classification, we selected a diverse set of models, including four convolutional neural networks (CNN), and eight vision transformer (ViT)-based Dosovitskiy (2020) models. The ViT variants were trained for feature extraction or classification, offering deeper insights into their performance. We used MedMNIST v2 Yang et al. (2023) , a widely used benchmark for medical image classification, which comprises 12 distinct biomedical 2D datasets. The most relevant work to ours is by Doerrich et al. (2024) , who also validated foundation models on MedMNIST v2 but used a different learning pipeline. Notably, compared to Doerrich et al. (2024) , we adopted more foundation models and achieved significantly better results for eight common models, leading to different conclusions. This finding motivated us to conduct further analyses including the effect of image resizing techniques. The main contributions of this work are as follows. • We investigate various foundation models, demonstrating their potential in transferring to medical image classification tasks. • We study the impact of image size and resizing strategies on the classification performance when transferring these foundation models, as well as their data efficiency. • We analyze the results and provide several insights for transferring foundation models to medical image classification tasks, such as optimal learning rate settings for different model categories, model selection, and image resizing strategy. 2 Method 2.1 Pre-trained Model Selection With thousands of pre-trained models publicly available, we select representative backbones for validation and comparison, as shown in fig:framework, ranging from CNN-based to ViT-based architectures. For CNN-based models, we include two baseline models: VGG16 Simonyan and Zisserman (2014) and DenseNet-121 Huang et al. (2017) , and two models with skip-connection via addition: EfficientNet-B4 Tan (2019) and ResNet-18 He et al. (2016) . All these CNN-based models were trained with ImageNet-1k Russakovsky et al. (2015) for image classification. For ViT-based models, we categorized them into two classes: (1) Four models trained for image classification: Three models, including ViT-B/16 Steiner et al. (2021) , CLIP ViT-B/16 Ilharco et al. (2021) and SAM-C ViT-B/16 Chen et al. (2021) , are adopted for comparison. These models were trained with different optimization strategies and datasets Wightman (2019) , all including ImageNet-1k. We also included SAM ViT-B/16 Kirillov et al. (2023) to compare with SAM-C ViT-B/16. Although it was trained with SA-1B 1 1 1 https://segment-anything.com for segmentation, its representations were demonstrated to be useful for various tasks. (2) Four models trained for feature extraction: OpenAI ViT-B/16 Ilharco et al. (2021) was trained on publicly available image-caption data to maximize the similarity of (image,text) pairs; EVA-02 ViT-B/14 Fang et al. (2024) can extract visual representation for reconstructing language-aligned vision features via masked image modeling; DINO ViT-B/16 Caron et al. (2021) was trained with the self-supervised DINO method, and DINOv2 ViT-B/14 Oquab et al. (2023) was trained with different losses and a larger dataset, namely LVD-142M. For further details of these models, please refer to tab:model details in Appendix A . 2.2 General Validation Framework As shown in fig:framework, to validate the selected foundation models on different datasets for classification tasks, we append a one-layer linear classifier after the encoder of each model. For a fair comparison, each classification model is trained with 15,000 iterations, and optimized with the AdamW optimizer Loshchilov (2017) using two optimization strategies, linear probing and end-to-end fine-tuning. In linear probing, the encoder is frozen and only the last classifier layer is optimized. The initial learning rate is set to 0.001 , which is reduced by 0.9 every 200 iterations. For end-to-end fine-tuning, we adopt the same updating strategy as in linear probing for the classifier, while keeping the learning rate for the encoder, denoted as lr_{e} , as a constant. Using a low learning rate for the encoder is a common method to adapt the pre-trained model to downstream tasks Baharoon et al. (2023) . The selection of lr_{e} will be discussed in Section 3.2 . 3 Benchmark Results 3.1 Datasets To evaluate the potential of foundation models in medical image classification tasks, we utilize MedMNIST Yang et al. (2023) collection for validation.