From pre-training to downstream performance: Does domain-specific pre-training make sense?
Felix H. Krones · 2026
Read From pre-training to downstream performance: Does domain-specific pre-training make sense? by Felix H. Krones (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08819.txt.
Introduction
Felix Krones asks whether pretraining medical image models on medical data actually pays off downstream. Deep learning has transformed medical imaging, yet the link between pretraining choices and final performance stays murky. Krones systematically compares convolutional networks and transformers across supervised and self supervised pretraining, varied starting points, and data types, then tests them on natural photos, chest X-rays, chest scans, and retinal images.
Only pretraining closely matching the target modality reliably improves results. Self supervised learning can beat supervised approaches, but its edge shifts with context. The study gives developers concrete guidance: match the pretraining diet to the clinical target instead of assuming any large scale start transfers equally well to patient care.
Catalog record
| CREATOR | Felix H. Krones |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From pre-training to downstream performance: Does domain-specific pre-training make sense? Felix H. Krones Abstract Deep learning techniques have revolutionised medical imaging, improving diagnostic accuracy and enabling both more accurate and earlier disease detection. However, the relationship between pre-training strategies and downstream performance in medical imaging models requires further exploration. Here, we systematically compare convolutional neural networks and transformers, examining various pre-training approaches, including supervised and self-supervised learning, as well as different initialisations and data modalities. Models are evaluated on natural images, chest X-rays, chest CT and retina OCT images, considering the effects of matching pre-training data with target modalities. Our findings indicate that only pre-training on data closely matching the target modality significantly improves downstream performance. While self-supervised learning can outperform supervised methods, its effectiveness varies with context. The study underscores the importance of pre-training strategies to enhance the reliability and effectiveness of deep learning models in medical imaging. By addressing these key factors, our research aims to contribute to the development of more accurate and dependable diagnostic tools, ultimately improving patient outcomes in clinical settings. 1 Introduction Background. Deep learning techniques have revolutionised various domains, with significant impact on medical imaging [ 30 , 42 ] . These advanced machine learning methods, particularly convolutional neural networks (CNNs) and transformers, have enhanced diagnostic accuracy, enabling earlier detection and treatment of diseases [ 42 ] . The detailed understanding of the relationship between pre-training on large datasets such as ImageNet and specific domain adaptation is a critical step in achieving good performance of diagnostic models. As it was shown that incorporating more domain-specific data during pre-training can significantly improve model performance [ 3 , 4 ] and to address the need for labelled data, self-supervised learning is increasingly explored for pre-training [ 3 , 11 ] . Research gap. Previous studies in medical imaging have primarily focused on comparing model designs [ 4 , 48 ] , fine-tuning [ 43 ] and various supervised and self-supervised learning strategies [ 3 , 20 , 35 ] . Three primary findings emerged from there: i) Self-supervised pre-training has the potential to outperform supervised pre-trained models on the same dataset. ii) Pre-training on extensive datasets such as ImageNet, followed by further pre-training on domain-specific data, tends to be more successful than starting with domain-specific data alone. iii) A good initialisation is more crucial for transformer-based models than for CNN based models. [ 39 ] showed that while CNNs show better results on clean images and are more invariant towards translation, ViTs show better robustness towards obstructions. However, the relationship between pre-training and downstream performance, particularly across different data collection sites and performance metrics, remains under-explored. An important question here is how closely pre-training data need to be aligned with downstream data, such that a model pre-trained on large, domain-agnostic data would be outperformed. Contributions. We investigated the opportunities of domain specific pre-training on downstream performance, demonstrating that only pre-training data closely aligned with downstream data can improve model performance compared to pre-training on large, domain-agnostic data. We evaluated the challenges of achieving robust performance across different data collection sites, showing that performance varies significantly across datasets, partially due to differing disease distributions. Additionally, we examined the risks associated with using a limited set of evaluation metrics, revealing that models with high AUC values can still exhibit weak subset accuracies and varying performance across diseases. Finally, we explored the potential of human-in-the-loop systems by assessing the so-called Oracle AUC. Figure 1: Examples of image modalities used in this study. Examples of images, one from each of eight datasets used in this study, are shown. First row (from left to right): ImageNet, MIMIC-IV-CXR, OCT, CovidxCT. Second row (from left to right): NIH ChestX-ray, CheXpert, PadChest, VinDr-CX. Table 1: Experiments comparing model backbones and training strategies across pre-training datasets. Pre-training happened for 1k epochs on a random subset of 100k images. All models were fine-tuned five-times using different seeds for 30 epochs, using the NIH ChestX-ray train (75,312) and validation (11,212) data with 14 classes. As baseline (b) we consider models initialised with ImageNet1k weights without further pre-training. Exp. Back-bone Initiali-sation Pre-training strategy Pre-training domain Pre-training dataset Fine-tuning 1.1 ResNet50 Random None None None NIH ChestX-ray 1.2 (b) ResNet50 ImageNet1k None None None NIH ChestX-ray 2.1 ViT-s Random None None None NIH ChestX-ray 2.2 (b) ViT-s ImageNet1k None None None NIH ChestX-ray 2.3 ViT-s Random Supervised X-ray, chest MIMIC-IV-CXR NIH ChestX-ray 2.4 ViT-s ImageNet1k Supervised X-ray, chest MIMIC-IV-CXR NIH ChestX-ray 3.1 ViT-s Random Self-supervised X-ray, chest NIH ChestX-ray NIH ChestX-ray 3.2 ViT-s ImageNet1k Self-supervised X-ray, chest NIH ChestX-ray NIH ChestX-ray 3.3 ViT-s Random Self-supervised X-ray, chest MIMIC-IV-CXR NIH ChestX-ray 3.4 ViT-s ImageNet1k Self-supervised X-ray, chest MIMIC-IV-CXR NIH ChestX-ray 3.5 ViT-s Random Self-supervised OCT, retina OCT NIH ChestX-ray 3.6 ViT-s ImageNet1k Self-supervised OCT, retina OCT NIH ChestX-ray 3.7 ViT-s Random Self-supervised CT, chest CovidxCT NIH ChestX-ray 3.8 ViT-s ImageNet1k Self-supervised CT, chest CovidxCT NIH ChestX-ray 2 Methodology 2.1 Datasets We used eight different datasets in this study. For pre-training, we randomly sampled 100,000 images from each dataset (ImageNet [ 9 ] , ChestX-ray [ 49 ] , MIMIC-IV-CXR [ 24 , 14 , 23 ] , CovidxCT [ 17 ] , OCT [ 25 ] ). Every available slice from 3D images was treated as an individual image. Subsequently, we fine-tuned and evaluated all models on a single image modality, specifically chest X-rays (sourced from NIH ChestX-ray [ 49 ] ) due to their widespread availability. For our evaluation, we used datasets which were not part of any training step (CheXpert [ 21 ] , PadChest [ 6 ] , VinDr-CXR [ 37 ] and NIH ChestX-ray again for comparative purposes). Table 1 shows all experiments conducted. Representative examples of images are shown in Figure 1 . 2.2 Pre-training Due to the good performance of GMML in combination with the smaller ViT-small backbone and less fine-tuning epochs, we opted to use GMML [ 2 ] as the primary pre-training strategy for our main experiments. We pre-trained ViT-small for 1,000 epochs. 2.3 Fine-tuning All models were fine-tuned five-times using different seeds for 30 epochs, using the NIH ChestX-ray train (75,312) and validation (11,212) data with 14 binary, multi-label classes. The decision to stop fine-tuning after 30 epochs is grounded in our belief that fine-tuning should serve as a brief adjustment to the target tasks rather than prolonged retraining. This contrasts with [ 35 ] and [ 41 ] , who extended their training to 200 epochs. 2.4 Evaluation We used the five diseases (Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion) as reported in CheXpert for evaluation, owing to their presence across several X-ray datasets (MIMIC-IV-CXR, NIH ChestX-ray, CheXpert, PadChest). We first compared the multi-site performance using the AUC metric as it is unaffected by threshold choice. We then performed a more extensive reliability assessment by evaluating the models on a range of metrics. For this we used the hand-labelled CheXpert validation data. We emphasised on accuracy and the underdiagnosis rate (false negative rate) across diverse decision thresholds. Moreover, we evaluated the subset accuracy measuring if a model predicted all labels correctly. We also included Oracle AUC, which is the AUC if the x\% most uncertain cases could be referred back