Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis

Iván Matas, Carmen Serrano, Miguel Nogales, David Moreno, Lara Ferrándiz, Teresa Ojeda, Begoña Acha · 2025

Read Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis by Iván Matas, Carmen Serrano, Miguel Nogales, David Moreno, Lara Ferrándiz, Teresa Ojeda, Begoña Acha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16773.txt.

Introduction

Iván Matas and co-authors question whether skin-disease AI should borrow its eyes from general photo collections, in a 2025 paper. Models for spotting skin conditions usually start from networks pretrained on ImageNet's everyday images, which may not teach the textures that matter in dermatology. The team instead trained a variational autoencoder from scratch on their own skin image collection, letting it learn a structured map of skin features without labels.

They then raced the self-taught feature extractor against an ImageNet-pretrained one under identical classification conditions. The two learned in visibly different patterns, and the self-supervised model reached a lower final validation loss. The comparison matters because labeled medical images are scarce and precious: spending compute on domain-specific pretraining may beat borrowing generic vision, especially for clinics whose patients differ from stock photo libraries.

Catalog record

CREATORIván Matas, Carmen Serrano, Miguel Nogales, David Moreno, Lara Ferrándiz, Teresa Ojeda, Begoña Acha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis Iván Matas, Carmen Serrano, Miguel Nogales, David Moreno, Lara Ferrándiz, Teresa Ojeda, Begoña Acha Abstract Deep learning has transformed computer vision but relies heavily on large labeled datasets and computational resources. Transfer learning, particularly fine-tuning pretrained models, offers a practical alternative; however, models pretrained on natural image datasets such as ImageNet may fail to capture domain-specific characteristics in medical imaging. This study introduces an unsupervised learning framework that extracts high-value dermatological features instead of relying solely on ImageNet-based pretraining. We employ a Variational Autoencoder (VAE) trained from scratch on a proprietary dermatological dataset, allowing the model to learn a structured and clinically relevant latent space. This self-supervised feature extractor is then compared to an ImageNet-pretrained backbone under identical classification conditions, highlighting the trade-offs between general-purpose and domain-specific pretraining. Our results reveal distinct learning patterns. The self-supervised model achieves a final validation loss of 0.110 (-33.33%), while the ImageNet-pretrained model stagnates at 0.100 (-16.67%), indicating overfitting. Accuracy trends confirm this: the self-supervised model improves from 45% to 65% (+44.44%) with a near-zero overfitting gap, whereas the ImageNet-pretrained model reaches 87% (+50.00%) but plateaus at 75% (+19.05%), with its overfitting gap increasing to +0.060. These findings suggest that while ImageNet pretraining accelerates convergence, it also amplifies overfitting on non-clinically relevant features. In contrast, self-supervised learning achieves steady improvements, stronger generalization, and superior adaptability, underscoring the importance of domain-specific feature extraction in medical imaging. I Introduction Deep learning has emerged as a transformative force in computer vision, facilitating state-of-the-art performance in tasks such as image classification, object detection, and segmentation. However, the inherent computational expense and extensive labeled datasets required to train deep neural networks from scratch render this approach infeasible in many practical applications. To address these challenges, fine-tuning has become a pivotal methodology within transfer learning, enabling the adaptation of pretrained models to specific domains while mitigating the burden of training large-scale models from the ground up. Fine-tuning leverages the knowledge from a large source dataset—typically a corpus of natural images. This approach accelerates convergence, enhances generalization, and improves performance with fewer labeled samples. A widely adopted paradigm in fine-tuning involves initializing models with weights pretrained on extensive datasets such as ImageNet, which encompasses a diverse collection of natural images. These pretrained architectures capture hierarchical feature representations, which can be repurposed for domain-specific applications with limited training data. Nevertheless, the effectiveness of transfer learning is fundamentally contingent upon the degree of similarity between the source and target domains. When the target dataset exhibits significant divergence from ImageNet, conventional fine-tuning strategies may lead to suboptimal generalization and increased susceptibility to overfitting. This limitation underscores the necessity of exploring the comparative advantages of using ImageNet pretraining versus domain-specific pretraining, particularly in specialized fields such as medical imaging and fine-grained visual categorization. Recent studies have examined the limitations and benefits of ImageNet pretraining in medical imaging. Juodelyte et al. and Raptis et al. [ 1 , 2 ] found that while ImageNet-pretrained models achieved competitive performance, they were prone to overfitting to spurious correlations, reducing their robustness to out-of-distribution samples. Their analyses, spanning X-ray, CT, cytological, and MRI scans, suggest that models pretrained on domain-specific datasets often outperform ImageNet-based counterparts, particularly when feature distributions diverge significantly from natural images. Similarly, Zhang et al. [ 3 ] demonstrated that ImageNet pretraining accelerates convergence and improves early-stage accuracy, yielding a 3–4% classification accuracy gain over training from scratch. These findings highlight both the efficiency advantages of transfer learning and its fundamental limitations when applied to specialized medical domains. Kornblith et al. [ 4 ] evaluated ImageNet-pretrained models across twelve datasets, finding a strong correlation ( r=0.96 ) between ImageNet accuracy and transfer learning performance. While superior ImageNet models generally improved fine-tuning outcomes, their study also revealed that for small, fine-grained classification tasks, ImageNet pretraining offered minimal benefits, indicating limited generalizability to highly specialized domains. These findings highlight the critical role of pretraining strategies in model robustness and generalization. While ImageNet pretraining remains a widely used baseline, models trained on domain-specific datasets demonstrate greater resilience to dataset biases and domain shifts. This underscores the trade-off between broad cross-domain generalization and specialization for specific tasks, an ongoing challenge in transfer learning research. A key limitation of ImageNet-pretrained models is their tendency to overfit more quickly than models pretrained on datasets closely aligned with the target domain. Although ImageNet pretraining provides a strong initialization, its feature representations may be suboptimal for highly specialized tasks with distinct data distributions. Addressing this issue requires tailored fine-tuning strategies that enhance feature adaptation while retaining the advantages of pretraining. In this study, we systematically analyze the overfitting behavior of ImageNet-pretrained models compared to those pretrained on domain-specific datasets. By evaluating performance across architectures, datasets, and training conditions, we identify the inherent limitations of conventional transfer learning and propose strategies to improve generalization in specialized applications. Our findings contribute to the ongoing discourse on transfer learning, offering insights into optimizing pretraining approaches for enhanced model robustness and performance. I-A Rationale and Advantages The decision to train a model from scratch using unsupervised learning offers several critical advantages in medical imaging applications. Domain-specific feature extraction enables the identification of intrinsic patterns within dermatological data, capturing nuanced features that might elude human annotation. This is particularly valuable for distinguishing between complex conditions such as Basal Cell Carcinoma (BCC), melanoma, actinic keratosis, squamous cell carcinoma (SCC),and so on. By initializing with random weights rather than pretrained parameters from ImageNet (which predominantly contains non-medical images), the model constructs hierarchical feature representations specific to dermatological imaging, avoiding the introduction of biases from unrelated visual domains. The model develops representations uniquely tailored to the hospital’s imaging techniques and patient demographics, enhancing real-world clinical applicability. Unlike supervised approaches that may overfit to predefined labels, our unsupervised methodology focuses on learning the inherent structure of skin lesion data, reducing the risk of capturing superficial or clinically irrelevant patterns. I-B Clinical Implications This approach addresses a fundamental challenge in medical image classification: the extraction of weak or non-informative features that can compromise diagnostic accuracy. By focusing on the intrinsic characteristics of dermatological lesions, the model distinguishes critical visual markers essential for accurate diagnosis, improving both classification performance and clinical interpretability. The enhanced feature extraction capability facilitates better generalization across diverse dermatological conditions encountered in clinical practice, potentially improving diagnostic support in ambiguous cases where visual distinction between pathologies requires sophisticated pattern recognition. II Material II-A Database The dataset used in this study originates from “Hospital Universitario Virgen Macarena”, a public hospital from the Andalusian Healthcare System, located in Seville, Spain. It is an extensive collection of dermatological cases compiled from 60 primary healthcare centers, where clinical data and images have been gathered over multiple years from numerous patients presenting various skin lesions. These images are sent to the main hospital via teledermatology. The dataset contained a total of 200,000 images, with approximately 40% being clinical images (capturing a broader anatomical context) and 60% being dermatoscopic images (focused on skin lesions). Given the study’s objective, only the dermatoscopic images were retained for further processing. The dataset comprises 15 distinct skin lesion classes, which include the most common dermatological conditions related

Read the complete text