Novel large empirical study of deep transfer learning for COVID-19 classification based on CT and X-ray images Authors: Mansour Almutaani, Turki Turki, Y.-H. Taguchi Published: 2024 ABSTRACT The early and highly accurate prediction of COVID-19 based on medical images can speed up the diagnostic process and thereby mitigate disease spread; therefore, developing AI-based models is an inevitable endeavor. The presented work, to our knowledge, is the first to expand the model space and identify a better performing model among 10,000 constructed deep transfer learning (DTL) models as follows. First, we downloaded and processed 4481 CT and X-ray images pertaining to COVID-19 and non-COVID-19 patients, obtained from the Kaggle repository. Second, we provide processed images as inputs to four pre-trained deep learning models (ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34) on more than a million images from the ImageNet database, in which we froze the convolutional and pooling layers pertaining to the feature extraction part while unfreezing and training the densely connected classifier with the Adam optimizer. Third, we generate and take a majority vote of two, three, and four combinations from the four DTL models, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{4} {\left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right)} = 11$$\end{document} DTL models. Then, we combine the 11 DTL models, followed by consecutively generating and taking the majority vote of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{11} {\left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right)} = 2036$$\end{document} DTL models. Finally, we select \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7953$$\end{document} DTL models from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right).$$\end{document} Experimental results from the whole datasets using five-fold cross-validation demonstrate that the best generated DTL model, named HC, achieving the best AUC of 0.909 when applied to the CT dataset, while ConvNeXt yielded a higher marginal AUC of 0.933 compared to 0.93 for HX when considering the X-ray dataset. These promising results set the foundation for promoting the large generation of models (LGM) in AI.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-024-76498-4. Introduction Coronavirus disease 2019 (COVID-19) patients experience different symptoms, such as a loss of smell and/or taste, fever, headache, and shortness of breath, to name a few1. Human-to-human transmission can happen via (1) sneeze droplets carrying SAR-CoV-2; (2) direct contact with an infected individual; and (3) touching a virus-containing surface followed by touching the eyes or mouth2. To lower the risk of COVID-19 transmission, the Centers for Disease Control and Prevention suggested prevention strategies including wearing masks, social distancing, and washing hands with soap for at least 20 s (see Fig. 1)3,4. The early detection of COVID-19 and thereby isolating the infected individuals is considered the most effective prevention strategy5,6. Hence, researchers have proposed various computational methods based on deep learning to detect COVID-19 using various image sources. Ozturk et al.7 proposed a deep learning approach (DarkCovidNet) composed of interleaved convolutional and max pooling layers for feature extraction and a densely connected classifier for binary and multiclass COVID-19 classification tasks. DarkCovidNet and nine other deep learning (DL) models were applied to 635 (and 1125) X-ray test images from GitHub pertaining to binary (and multiclass) COVID-19 classification. DarkCovidNet had the highest accuracy of 0.9808 (and 0.8702) for binary (and multiclass) COVID-19 classification. Fig. 1COVID-19 disease symptoms as well as transmission and prevention methods for the SARS-CoV-2 virus. Figure created with BioRender.com. COVID-19 disease symptoms as well as transmission and prevention methods for the SARS-CoV-2 virus. Figure created with BioRender.com. To address the COVID-19 classification task using X-ray images, Apostolopoulos et al.8 employed five pre-trained DL models, including MobileNetV2, VGG19, Inception, Xception, and InceptionResNetV2, applied to two testing datasets obtained from GitHub and Kaggle. The first testing dataset had 1428 X-ray images for the binary class task pertaining to COVID-19 and non-COVID-19 images, while the multiclass task was to discriminate between COVID-19, pneumonia, and normal images. VGG19 achieved the highest accuracy of 0.9875 and 0.9348 for the binary and multiclass classification tasks, respectively. For the second testing dataset of 1442 test images, MobileNetV2 generated the highest accuracy of 0.9678 and 0.9472 for the binary and multiclass classification tasks, respectively. Ismael et al.9 aimed to discriminate between normal and COVID-19 X-ray images as follows. They employed five pre-trained DL models, namely VGG16, VGG19, ResNet18, ResNet50, and ResNet101, for feature extraction, in which each was coupled with SVM to train and perform prediction. The data were downloaded COVID-19 X-ray images from GitHub, Kaggle, and Radiology Assistant. The results demonstrated that applying ResNet50 to 95 test images for feature extraction followed by SVM for classification resulted in the highest accuracy of 0.947. Ravi et al.10 proposed a deep and machine learning approach to discriminate between COVID-19 and non-COVID-19 X-ray and CT images. The proposed approach consisted of pre-trained EfficientNet for feature extraction followed by logistic regression as a machine learning (ML) model for classification. Compared to 26 DL models, EfficientNet + ML when applied to 2417 CT images yielded the highest accuracy of 0.9946. Moreover, EfficientNet + ML generated the highest accuracy of 0.9948 when applied to 2864 COVID-19 X-ray images. Asif et al.11 proposed a shallow CNN trained on COVID-19 X-ray images from Kaggle. The proposed shallow CNN was compared against five pre-trained DL models (i.e., NasNet, MobileNet, Xception, InceptionV3, and DenseNet201), in which all were applied to 636 COVID-19 X-ray test images. The results demonstrated that the proposed shallow CNN achieved the highest accuracy of 0.9968. Gupta et al.12 proposed a deep learning model (DLM) for the task of COVID-19 classification based on COVID-19 CT images from Kaggle. The proposed DLM is composed of a series of convolutional and max pooling layers for feature extraction, followed by a densely connected classifier to discriminate between COVID-19 and non-COVID-19 CT images. Compared to two pre-trained models (DarkNet19 and MobileNetV2), the results demonstrated that the proposed DLM when applied to 2481 images yielded the highest accuracy of 0.9891. Ayalew et al.13 proposed an approach based on a CNN for feature extraction coupled with SVM for COVID-19 classification. They downloaded COVID-19 X-ray images from GitHub and then applied the CNN + SVM to 1200 X-ray images pertaining to COVID-19 and normal cases. The results demonstrated that the proposed approach achieved the highest accuracy of 0.991. Constantinou et al.14 employed five pre-trained DL models, including InceptionV3, DenseNet169, ResNet101, DenseNet121, and ResNet50, for the multiclass COVID-19 classification task. They downloaded COVID-19 X-ray images from Kaggle. Results on 6788 test images demonstrated that ResNet101 achieved the highest accuracy of 0.96 when compared to the other DL models. In Table 1, we provide a comparison of our proposed work against existing works for COVID-19 classification. Table 1Comparison of our proposed work against other studies.YearStudyData sourceScanningNo. of modelsBest modelClassTest imagesResults2020Ozturk et al.7GitHub19X-ray10DarkCovidNet2-class (COVID-19, no findings)635ACC (0.9808)3-class (COVID-19, P, no findings)1125ACC (0.8702)2020Apostolopoulos et al.8GitHub19,X-ray5VGG192-class (COVID-19, non-COVID-19)1428ACC (0.9875)Kaggle20,213-class (COVID-19, P, normal)ACC (0.9348)MobileNetv22-class (COVID-19, non-COVID-19)1442ACC (0.9678)3-class (COVID-19, P, normal)ACC (0.9472)2020Ouchicha et al.22GitHub,X-ray1CVDNet3-class (COVID-19, P, normal)2905ACC (0.9669)Kaggle2021Ismael et al.9GitHub19,X-ray5ResNet50 features + SVM2-class (COVID-19, normal)95ACC (0.947)Kaggle20,21, andRadiologyAssistant232021Ahamed et al.24Kaggle25–27CT1ResNet50V22-class (COVID-19, normal)3000ACC (0.99990)3-class (COVID-19, normal, CAP)ACC (0.99012)X-ray2-class (COVID-19, VP)4593ACC (0.98954)3-class (COVID-19, BP, normal)ACC (0.97242)4-class (COVID-19, BP, VP, normal)ACC (0.96452)2021Jain et al.28Kaggle29X-ray3Xception3-class (COVID-19, P, normal)965ACC (0.9797)2022Ravi et al.10Mendeley30CT27EfficientNet + ML2-class (COVID-19, non-COVID-19)2417ACC: (0.9946)X-ray2864ACC (0.9948)2022Furtado et al.31Multiple32–38CT1Cimatec-CovNet-192-class (COVID-19, non-COVID-19)414AUC (0.95) + local2022Asif et al.11Kaggle25,26X-ray6Shallow CNN2-class (COVID-19, normal)636ACC (0.9968)2023Gupta et al.12Kaggle39,40CT3DLM2-class (COVID-19, non-COVID-19)2481ACC (0.9891)2023Ayalew et al.13GitHub19,41X-ray1CNN + SVM2-class (COVID-19, normal)1200ACC (0.991)2023Constantinou et al.14Kaggle42X-ray5ResNet1013-class (COVID-19, non-COVID-19, normal)6788ACC (0.96)2023Tuncer et al.43Kaggle39,40CT23Swin-textural3-class (COVID-19, others, healthy)4171ACC (0.9871)2023Kathamuthu et al.44Kaggle39,40CT6VGG162-class (COVID-19, normal)497ACC (0.98)2024Liu et al.45Kaggle46,47X-ray1CECT2-class (positive, negative)1381ACC (0.981)2-class (positive, negative)400ACC (0.972)2024Chen et al.48Kaggle25X-ray8ViT-based model2-class (COVID-19, non-COVID-19)2112ACC (0.9957)3-class (COVID-19, (normal + LO), P)ACC (0.9957)4-class (COVID-19, normal, LO, P)ACC (0.9579)2024Echtioui et al.49Shared50X-ray4DenseNet-169 + 2Dense3-class (COVID-19, normal, P)231ACC (0.9004)2024Abdullah et al.51Kaggle25X-ray12Hybrid DL-NN2-class (COVID-19, normal)2766ACC (0.9200)2024Akyol et al.52Kaggle39,40CT19ETSVF-COVID192-class (COVID-19, non-COVID-19)497ACC (0.9920)Multiple53X-ray3-class (COVID-19, P, normal)1388ACC (0.9856)2024Talukder et al.54Mendeley55,56X-ray6EfficientNetB44-class (COVID-19, LO, VP, normal)480ACC (0.9917)2-class (COVID-19, normal)224ACC (1.00)2024ProposedKaggle39,40CT10,008HC2-class (COVID-19, non-COVID-19)2481AUC (0.909)Kaggle25X-ray10,008ConvNeXt2-class (COVID-19, non-COVID-19)2000AUC (0.933)P stands for pneumonia. VP stands for viral pneumonia. BP stands for bacterial pneumonia. CAP stands for community-acquired pneumonia. LO stands for lung opacity. CT is computed tomography. ACC is accuracy. AUC is area under the ROC curve. Table 1Comparison of our proposed work against other studies.YearStudyData sourceScanningNo. of modelsBest modelClassTest imagesResults2020Ozturk et al.7GitHub19X-ray10DarkCovidNet2-class (COVID-19, no findings)635ACC (0.9808)3-class (COVID-19, P, no findings)1125ACC (0.8702)2020Apostolopoulos et al.8GitHub19,X-ray5VGG192-class (COVID-19, non-COVID-19)1428ACC (0.9875)Kaggle20,213-class (COVID-19, P, normal)ACC (0.9348)MobileNetv22-class (COVID-19, non-COVID-19)1442ACC (0.9678)3-class (COVID-19, P, normal)ACC (0.9472)2020Ouchicha et al.22GitHub,X-ray1CVDNet3-class (COVID-19, P, normal)2905ACC (0.9669)Kaggle2021Ismael et al.9GitHub19,X-ray5ResNet50 features + SVM2-class (COVID-19, normal)95ACC (0.947)Kaggle20,21, andRadiologyAssistant232021Ahamed et al.24Kaggle25–27CT1ResNet50V22-class (COVID-19, normal)3000ACC (0.99990)3-class (COVID-19, normal, CAP)ACC (0.99012)X-ray2-class (COVID-19, VP)4593ACC (0.98954)3-class (COVID-19, BP, normal)ACC (0.97242)4-class (COVID-19, BP, VP, normal)ACC (0.96452)2021Jain et al.28Kaggle29X-ray3Xception3-class (COVID-19, P, normal)965ACC (0.9797)2022Ravi et al.10Mendeley30CT27EfficientNet + ML2-class (COVID-19, non-COVID-19)2417ACC: (0.9946)X-ray2864ACC (0.9948)2022Furtado et al.31Multiple32–38CT1Cimatec-CovNet-192-class (COVID-19, non-COVID-19)414AUC (0.95) + local2022Asif et al.11Kaggle25,26X-ray6Shallow CNN2-class (COVID-19, normal)636ACC (0.9968)2023Gupta et al.12Kaggle39,40CT3DLM2-class (COVID-19, non-COVID-19)2481ACC (0.9891)2023Ayalew et al.13GitHub19,41X-ray1CNN + SVM2-class (COVID-19, normal)1200ACC (0.991)2023Constantinou et al.14Kaggle42X-ray5ResNet1013-class (COVID-19, non-COVID-19, normal)6788ACC (0.96)2023Tuncer et al.43Kaggle39,40CT23Swin-textural3-class (COVID-19, others, healthy)4171ACC (0.9871)2023Kathamuthu et al.44Kaggle39,40CT6VGG162-class (COVID-19, normal)497ACC (0.98)2024Liu et al.45Kaggle46,47X-ray1CECT2-class (positive, negative)1381ACC (0.981)2-class (positive, negative)400ACC (0.972)2024Chen et al.48Kaggle25X-ray8ViT-based model2-class (COVID-19, non-COVID-19)2112ACC (0.9957)3-class (COVID-19, (normal + LO), P)ACC (0.9957)4-class (COVID-19, normal, LO, P)ACC (0.9579)2024Echtioui et al.49Shared50X-ray4DenseNet-169 + 2Dense3-class (COVID-19, normal, P)231ACC (0.9004)2024Abdullah et al.51Kaggle25X-ray12Hybrid DL-NN2-class (COVID-19, normal)2766ACC (0.9200)2024Akyol et al.52Kaggle39,40CT19ETSVF-COVID192-class (COVID-19, non-COVID-19)497ACC (0.9920)Multiple53X-ray3-class (COVID-19, P, normal)1388ACC (0.9856)2024Talukder et al.54Mendeley55,56X-ray6EfficientNetB44-class (COVID-19, LO, VP, normal)480ACC (0.9917)2-class (COVID-19, normal)224ACC (1.00)2024ProposedKaggle39,40CT10,008HC2-class (COVID-19, non-COVID-19)2481AUC (0.909)Kaggle25X-ray10,008ConvNeXt2-class (COVID-19, non-COVID-19)2000AUC (0.933)P stands for pneumonia. VP stands for viral pneumonia. BP stands for bacterial pneumonia. CAP stands for community-acquired pneumonia. LO stands for lung opacity. CT is computed tomography. ACC is accuracy. AUC is area under the ROC curve. Comparison of our proposed work against other studies. P stands for pneumonia. VP stands for viral pneumonia. BP stands for bacterial pneumonia. CAP stands for community-acquired pneumonia. LO stands for lung opacity. CT is computed tomography. ACC is accuracy. AUC is area under the ROC curve. These existing AI-driven computational methods with which to detect COVID-19 based on medical images are far from perfect. Therefore, automatically expanding the model space is needed to identify a better performing model improving over the performance of existing methods. Hence, we propose a computational framework for the large generation of models. The novelty in our study can be mainly ascribed to the following summarized contributions:Unlike existing AI-based studies for detecting COVID-19, to our knowledge, this is the first study to investigate the large generation of AI models in which we expand the model space, constructing thousands of prediction models for the task of COVID-19 classification based on CT and X-ray images.Our framework works as follows: We employed four randomly selected pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. Then, we downloaded 2481 and 2000 COVID-19 images from Kaggle pertaining to CT and X-ray scanning methods, respectively. For each pre-trained model, we froze the feature extraction part composed of convolutional and pooling layers while training from scratch the densely connected layer with the Adam optimizer for the binary classification, resulting in four deep transfer learning (DTL) models. We took a majority vote of two, three, and four combinations from the four DTL models, resulting in a total of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{4} {\left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right)}$$\end{document} = 11 DTL models. Next, we took 2, 3, …, and 11 combinations from the 11 DTL models, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{11} {\left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right)}$$\end{document} = 2036 DTL models. Finally, we chose 7953 DTL models from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document}.According to our large empirical study of 10,008 models attributed to 11 + 2036 + 7953 generated DTL models plus 8 DTL models, the generated model, named HC, which corresponds to the composition of DTL models obtained the highest AUC of 0.909 when COVID-19 CT images were used. For the COVID-19 X-ray images, ConvNeXt and the generated model, HX, obtained an AUC of 0.933 and 0.930, respectively.Additionally, HC achieved 3.8%, 7.0%, 2.1%, and 1.4% AUC performance improvements over the FasterViT15, SHViT16, Swin17, and ViT18 models, respectively. HX gained 4.2%, 7.2%, 1.6%, and 1.1% AUC performance improvements over the FasterVit, SHViT, Swin and ViT models, respectively.These results demonstrate that our computational framework (1) is computationally efficient when working under the transfer learning setting; (2) leads to promoting never-ending AI model generation; (3) expands the model space from 4 to 10,000; and (4) aids in the search of high-performance models with which to tackle the studied classification task. Unlike existing AI-based studies for detecting COVID-19, to our knowledge, this is the first study to investigate the large generation of AI models in which we expand the model space, constructing thousands of prediction models for the task of COVID-19 classification based on CT and X-ray images. Unlike existing AI-based studies for detecting COVID-19, to our knowledge, this is the first study to investigate the large generation of AI models in which we expand the model space, constructing thousands of prediction models for the task of COVID-19 classification based on CT and X-ray images. Our framework works as follows: We employed four randomly selected pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. Then, we downloaded 2481 and 2000 COVID-19 images from Kaggle pertaining to CT and X-ray scanning methods, respectively. For each pre-trained model, we froze the feature extraction part composed of convolutional and pooling layers while training from scratch the densely connected layer with the Adam optimizer for the binary classification, resulting in four deep transfer learning (DTL) models. We took a majority vote of two, three, and four combinations from the four DTL models, resulting in a total of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{4} {\left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right)}$$\end{document} = 11 DTL models. Next, we took 2, 3, …, and 11 combinations from the 11 DTL models, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{11} {\left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right)}$$\end{document} = 2036 DTL models. Finally, we chose 7953 DTL models from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document}. Our framework works as follows: We employed four randomly selected pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. Then, we downloaded 2481 and 2000 COVID-19 images from Kaggle pertaining to CT and X-ray scanning methods, respectively. For each pre-trained model, we froze the feature extraction part composed of convolutional and pooling layers while training from scratch the densely connected layer with the Adam optimizer for the binary classification, resulting in four deep transfer learning (DTL) models. We took a majority vote of two, three, and four combinations from the four DTL models, resulting in a total of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{4} {\left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right)}$$\end{document} = 11 DTL models. Next, we took 2, 3, …, and 11 combinations from the 11 DTL models, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{11} {\left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right)}$$\end{document} = 2036 DTL models. Finally, we chose 7953 DTL models from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document}. According to our large empirical study of 10,008 models attributed to 11 + 2036 + 7953 generated DTL models plus 8 DTL models, the generated model, named HC, which corresponds to the composition of DTL models obtained the highest AUC of 0.909 when COVID-19 CT images were used. For the COVID-19 X-ray images, ConvNeXt and the generated model, HX, obtained an AUC of 0.933 and 0.930, respectively. According to our large empirical study of 10,008 models attributed to 11 + 2036 + 7953 generated DTL models plus 8 DTL models, the generated model, named HC, which corresponds to the composition of DTL models obtained the highest AUC of 0.909 when COVID-19 CT images were used. For the COVID-19 X-ray images, ConvNeXt and the generated model, HX, obtained an AUC of 0.933 and 0.930, respectively. Additionally, HC achieved 3.8%, 7.0%, 2.1%, and 1.4% AUC performance improvements over the FasterViT15, SHViT16, Swin17, and ViT18 models, respectively. HX gained 4.2%, 7.2%, 1.6%, and 1.1% AUC performance improvements over the FasterVit, SHViT, Swin and ViT models, respectively. Additionally, HC achieved 3.8%, 7.0%, 2.1%, and 1.4% AUC performance improvements over the FasterViT15, SHViT16, Swin17, and ViT18 models, respectively. HX gained 4.2%, 7.2%, 1.6%, and 1.1% AUC performance improvements over the FasterVit, SHViT, Swin and ViT models, respectively. These results demonstrate that our computational framework (1) is computationally efficient when working under the transfer learning setting; (2) leads to promoting never-ending AI model generation; (3) expands the model space from 4 to 10,000; and (4) aids in the search of high-performance models with which to tackle the studied classification task. These results demonstrate that our computational framework (1) is computationally efficient when working under the transfer learning setting; (2) leads to promoting never-ending AI model generation; (3) expands the model space from 4 to 10,000; and (4) aids in the search of high-performance models with which to tackle the studied classification task. Materials and methods Data preprocessing The CT dataset (named the SARS-COV-2 CT-Scan Dataset) was downloaded from Kaggle: https://www.kaggle.com/datasets/plameneduardo/sarscov2-ctscan-dataset (accessed on 17 March 2024). We randomly selected and processed 2481 images, in which the class label distribution was 1252 images for COVID-19 and 1229 images for non-COVID-19. The X-ray dataset (named the COVID-19 Radiography Database) was downloaded from Kaggle at https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database (accessed on 17 March 2024). We randomly selected 2000 images, in which 1000 images were categorized as COVID-19 and the other remaining 1000 images were categorized as non-COVID-19. All images were resized to 256 × 256 when addressing the COVID-19 classification task in this study. Figure 2 illustrates sample images from the CT and X-ray datasets prior to and after processing. Fig. 2From left to right: CT and X-ray images pertaining to two COVID-19 and non-COVID-19 samples, respectively. (a) Before and (b) after processing. From left to right: CT and X-ray images pertaining to two COVID-19 and non-COVID-19 samples, respectively. (a) Before and (b) after processing. Pre-trained CNNs ConvNeXt This pre-trained model on the ImageNet database of 1000 class labels initially starts with 1 convolutional layer, followed by 18 ConvNeXt blocks, which resemble residual blocks. However, each ConvNeXt block employs (1) layer normalization instead of batch normalization; (2) Gaussian error linear unit (GELU) activation instead of rectified linear unit (ReLU); and (3) 18 ConvNeXt blocks spanned over 4 stages as (3, 3, 9, 3) rather than 16 residual blocks spanned over 4 stages as (3, 4, 6, 3) in ResNet50. Each ConvNeXt block is composed of three convolutional layers coupled with a residual connection. Flattening is achieved before classification, which is a 1000-way fully connected dense layer coupled with a softmax activation function. This pre-trained model is referred to in57 as ConvNeXt-T. EfficientNetV2 Unlike EfficientNet, which only uses MBConv blocks for feature extraction, EfficientNetV258 includes both Fused-MBConv and MBConv blocks for feature extraction. The Fused-MBConv block improves training speed via replacing the pointwise and depthwise convolutional layers in MBConv with one convolutional layer. In our study, the pre-trained model initially starts with convolutional layer, followed by three Fused-MBConv blocks, three MBConv blocks, and a convolutional layer for feature extraction. A global average pooling is applied before the classification, consisting of a 1000-way fully connected dense layer coupled with a softmax activation function. DenseNet121 The feature extraction part in this pre-trained model59 initially starts with one standard convolutional and pooling layer, followed by four dense blocks with three interleaved transition layers. The four dense blocks consist of 6, 12, 24, and 16 convolutional blocks. Each convolutional block is composed of two convolutional layers. The transition layers include average pooling layers for dimensionality reduction in feature maps. The classification is a 1000-way fully connected dense layer coupled with a softmax activation function. ResNet34 For feature extraction, the ResNet34 pre-trained model60 on the ImageNet database of 1000 class labels initially starts with one convolutional and max pooling layer, followed by 16 residual blocks, spanned over four stages as (3, 4, 6, 3). Each residual block is composed of two convolutional layers coupled with skip connection, where batch normalization is placed after each convolution. Global average pooling is then performed before the classification, which is a 1000-way fully connected dense layer coupled with a softmax activation function. DTL In Fig. 3, we illustrate the DTL part pertaining to our approach applied to CT and X-ray images. First, we utilize four pre-trained deep learning (DL) models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. Each pre-trained DL model was induced after training on a million images from the ImageNet database to tackle the multiclass classification task of categorizing images into 1 of the 1000 predefined classes. We replace the 1000-way fully connected dense layer with softmax activation by training a 1-way fully connected layer with sigmoid activation. In this study, we leave the weights of the feature extraction part unchanged, and it is applied to COVID-19 images followed by a flattening step, and then training is performed to update the weights of subsequent layers in the classification part. The training phase incorporates the Adam optimizer and binary cross-entropy with a logit loss function. Each DTL model yields probabilities when performing predictions, in which, if the probability is greater than 0.5, then the image is classified as non-COVID-19. If the prediction value is less than or equal to 0.5, then the image is classified as COVID-19. In our DTL model, the fully connected dense layer is a weight layer that is not frozen and thereby has a few numbers of parameters during the training process. The only exception is ConvNeXt, in which we also train the layer normalization right before the fully connected dense layer (see Table 2). Fig. 3Flowchart showing our computational approach for COVID-19 classification. Data preprocessing: acquiring COVID-19 CT and X-ray images pertaining to COVID-19 and non-COVID-19 patients. DTL: CT images are provided to a pre-trained CNN for feature extraction followed by training a new classifier to discriminate between COVID-19 and non-COVID-19 images. Model generation: four models are provided as inputs to sequentially generate DTL models. Figure created with BioRender.com. Flowchart showing our computational approach for COVID-19 classification. Data preprocessing: acquiring COVID-19 CT and X-ray images pertaining to COVID-19 and non-COVID-19 patients. DTL: CT images are provided to a pre-trained CNN for feature extraction followed by training a new classifier to discriminate between COVID-19 and non-COVID-19 images. Model generation: four models are provided as inputs to sequentially generate DTL models. Figure created with BioRender.com. Table 2Summary for layers and parameters pertaining to the DTL models in this study.ModelNo. of ParamsUnfrozen layersTransferredTrainableConvNeXt27,820,897227,818,5922,305EfficientNetV220,178,769120,177,4881,281DenseNet1216,954,88116,953,8561,025ResNet3421,285,185121,284,672513 Table 2Summary for layers and parameters pertaining to the DTL models in this study.ModelNo. of ParamsUnfrozen layersTransferredTrainableConvNeXt27,820,897227,818,5922,305EfficientNetV220,178,769120,177,4881,281DenseNet1216,954,88116,953,8561,025ResNet3421,285,185121,284,672513 Summary for layers and parameters pertaining to the DTL models in this study. Model generation To generate a large number of DTL models performing prediction for a test example, z, in our study, we start with 4 DTL models, namely h1(z), h2(z), h3(z), and h4(z), which correspond to models obtained from training ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34, respectively. Let D1 = {h1(z), h2(z), h3(z), h4(z)}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$hi\left( \text{z} \right):\text{z} \to$$\end{document} {COVID-19, non-COVID-19}, and we generate all 2, 3 and 4 combinations from D1, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{j}^{1}$$\end{document} for j = 1…11 attributed to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{4} {\left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right)}$$\end{document}. It can be seen that the size of a given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{j}^{1}$$\end{document} is upper bounded by 4 and lower bounded by 2 (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2 \le \left| {S_{j}^{1} } \right| \le 4$$\end{document}). Assuming that hi(z) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\in S_{j}^{1}$$\end{document}, we then perform majority prediction as shown in Eq. (1), in which I is an indicator function returning 1 if true and 0 otherwise. Let D2 = {h11(z), h12(z),…, h111(z)}. Now, we generate all 2, 3, 4, 5, 6, 7, 8, 9, 10, and 11 combinations from D2, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{j}^{2}$$\end{document} for j = 1…2036 ascribed to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sum\nolimits_{r = 2}^{11} {\left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right)}$$\end{document}. It can be seen that the size of a given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{j}^{2}$$\end{document} is lower bounded by 2 and upper bounded by 11 (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2 \le \left| {S_{j}^{2} } \right| \le 11)$$\end{document}. Let h1i(z) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\in S_{j}^{2} ;$$\end{document} prediction is performed using Eq. (2). We set D3 = {h21(z), h22(z),…, h22036(z)}. We only select 7953 from the generated 2 combinations of D3. Therefore, we have \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{j}^{3}$$\end{document} for j = 1…7953 from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document} models. For h2i(z) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\in S_{j}^{3}$$\end{document}, prediction is performed using Eq. (3). It can be shown that we had \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left| {\left\{ {S_{j}^{1} } \right\}_{j = 1}^{11} } \right| + \left| {\left\{ {S_{j}^{2} } \right\}_{j = 1}^{2036} } \right| + 7953 = 10000$$\end{document} models.1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{j} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{i} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h2_{k} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{i} {I\left( {h1_{i} \left( \text{z} \right) = y} \right)}$$\end{document}3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h3_{l} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{i} {I\left( {h2_{i} \left( \text{z} \right) = y} \right)}$$\end{document} Classification methodology In our study, we randomly adapted four pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. All of these four pre-trained DL models were trained on more than a million images from the ImageNet database, in which the number of different class labels is equal to 1000. We froze all layers in feature extraction related to convolutional and pooling layers to extract features from the COVID-19 images generated via imaging techniques while updating weights in the subsequent layers pertaining to the classification part during the training phase. For all of the DTL models, we utilized the Adam optimizer with binary cross-entropy with a logit loss function. In terms of the optimization parameters when building models during the training phase, we used 32 for the batch size, 0.001 for the learning rate, and 10 for the number of epochs. When performing prediction for an unseen image, if the predicted value is greater than 0.5, then we designate the image as non-COVID-19. Otherwise, we designate the image as COVID-19. For all of the DTL models including the 10,000 constructed models in each imaging technique, we employed the following five performance measures to evaluate the studied models61,62:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{Accuracy}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{TP}} + {\text{TN}} + {\text{FP}} + {\text{FN}}}}$$\end{document}5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{F}}1 = \frac{{2 \times {\text{TP}}}}{{2 \times {\text{TP}} + {\text{FP}} + {\text{FN}}}}$$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{Precision}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FP}}}}$$\end{document}7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{Sensitivty}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}$$\end{document}8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{Sepcificity}} = \frac{{{\text{TN}}}}{{{\text{TN}} + {\text{FP}}}}$$\end{document}where TP stands for true positive, referring to the number of non-COVID-19 images that were correctly classified as non-COVID-19. FN stands for false negative, referring to the number of non-COVID-19 images that were incorrectly classified as COVID-19. TN stands for true negative, referring to the number of COVID-19 images that were correctly classified as COVID-19. FP stands for false positive, referring to the number of COVID-19 images that were incorrectly classified as non-COVID-19. To assess the performance on whole datasets, we employed five-fold cross-validation. Implementation details When running the experiments in this study using an NVIDIA Tesla P100 GPU, the deep learning models are implemented in Python (version 3.10.13)63 based on the PyTorch library (version 2.1.2)64,65 and the torchvision library (version 0.16.2), in which the installed CUDA version is 12.166. Data processing is achieved through various libraries, including NumPy (version 1.26.4) and pandas (version 2.2.2)67,68. The performance evaluation is performed through the sklearn library (version 1.2.2)69. Results Classification results Training results Figure 4a,b display the average training accuracy and loss over 10 epochs for the eight DTL models during the running of five-fold cross-validation, obtained when applied to COVID-19 CT and X-ray datasets, respectively. The accuracy and loss moving in the upward and downward directions, respectively, show the capability of models to learn and thereby will be able to classify unseen COVID-19 images during the testing phase. When considering the CT dataset (Fig. 4a), ViT achieved the highest and lowest average accuracy and loss of 0.871 and 0.257, respectively, followed by ConvNeXt, achieving an average accuracy (and loss) of 0.860 (and 0.260). DenseNet121, yielding an average accuracy (and loss) of 0.846 (and 0.303). SHViT had the lowest average accuracy (and a loss) of 0.806 (and 0.356). For the X-ray dataset as shown in Fig. 4b, ConvNeXt achieved the highest average accuracy of 0.899 while attaining the lowest average loss of 0.204. ViT had an average accuracy and loss of 0.893 and 0.215, respectively. Swin obtained an average accuracy (and loss) of 0.877 (and 0.230). DenseNet121 achieved an average accuracy and loss of 0.870 and 0.260, respectively. SHViT yielded an average accuracy and loss of 0.831 and 0.318, respectively. Fig. 4Average accuracy and loss for ten epochs during the running of five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. Average accuracy and loss for ten epochs during the running of five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. In Fig. 5a,b pertaining to CT and X-ray datasets, respectively, we illustrate the total running time for the model induction process of four DTL models during the running of five-fold cross-validation. In terms of the CT dataset as shown in Fig. 5a, ResNet34 is the fastest. Specifically, ResNet34 is 1.14 × faster than ConvNeXt. Moreover, the running time difference between DenseNet121 and EfficientNetV2 is marginal, in which the former and latter are 1.067 and 1.062 × faster than ConvNeXt. For the X-ray dataset as illustrated in Fig. 5b, ResNet34 is still the fastest method, which is 1.37 × faster than ConvNeXt. The second and third fastest methods are DenseNet121 and EfficientNetV2, respectively. Both DenseNet121 and EfficientNetV2 resulted in marginal running time differences in which the former and latter are 1.242 and 1.210 × faster than ConvNeXt. These results demonstrate that, although ConvNeXt yielded the highest performance results, it was the slowest among all of the models. Because we did not incorporate transformer-based models in the model generation process, we moved the total running time of each transformer-based model in Figure S3 within the Supplementary Additional File. Fig. 5Total running time for the four studied DTL models during the training phase in five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. Total running time for the four studied DTL models during the training phase in five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. Testing results The boxplots in Fig. 6a,b illustrate the generalization (testing) results of the four DTL models utilizing five-fold cross-validation pertaining to CT and X-ray datasets, respectively. Regarding Fig. 6a,b, the median results are indicated via the horizontal lines that cross each box. In terms of the CT dataset in Fig. 6a, ConvNeXt yielded the best results, achieving the highest medians of 0.901, 0.903, 0.874, 0.935, and 0.872 in terms of accuracy, F1, precision, sensitivity, and specificity, respectively. The second best performing model was DenseNet121, achieving a median accuracy of 0.885, a median F1 of 0.889, a median precision of 0.857, a median sensitivity of 0.930, and a median specificity of 0.844. ResNet34 is the third best performing model, generating a median accuracy of 0.853, a median F1 of 0.863, a median precision of 0.832, a median sensitivity of 0.894, and a median specificity of 0.828. EfficientNetV2 is the worst performing method, yielding the lowest median accuracy of 0.842, the lowest median F1 of 0.851, the lowest median precision of 0.802, a median sensitivity of 0.910, and the lowest median specificity of 0.780. Regarding the X-ray dataset in Fig. 6b, ConvNeXt obtained the highest median accuracy of 0.930, the highest median F1 of 0.928, the highest median precision of 0.931, the highest median sensitivity of 0.930, and the highest median specificity of 0.930. The second best performing method is DenseNet121, which had a median accuracy of 0.917, a median F1 of 0.912, a median precision of 0.910, a median sensitivity of 0.925, and a median specificity of 0.915. The third best method is ResNet34, yielding a median accuracy, F1, precision, sensitivity, and specificity of 0.885, 0.887, 0.896, 0.900, and 0.895, respectively. EfficientNetV2 was the worst performing model, obtained a median accuracy, F1, precision, sensitivity, and specificity of 0.865, 0.862, 0.848, 0.915, and 0.835, respectively. Fig. 6Boxplots illustrate several generalization (testing) performance results (based on accuracy, F1, precision, sensitivity, and specificity) of the four DTL models when utilizing five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. Boxplots illustrate several generalization (testing) performance results (based on accuracy, F1, precision, sensitivity, and specificity) of the four DTL models when utilizing five-fold cross-validation pertaining to (a) CT and (b) X-ray datasets. Table 3a,b report the testing performance results on the whole datasets pertaining to (a) CT and (b) X-ray datasets, including four transformer-based models. In terms of the CT dataset (see Table 3a), ConvNeXt is the best performing model, yielding the highest ACC of 0.894 (tie with ViT) and the highest F1 of 0.900. The second best model is ViT, which yielded an ACC of 0.894, an F1 of 0.896, the highest PRE of 0.877, an SEN of 0.915, and the highest SPE of 0.874. SHViT is the worst performing model, yielding the lowest ACC of 0.839, the lowest F1 of 0.843, a PRE of 0.813, the lowest SEN of 0.876, and an SPE of 0.802. When considering the X-ray dataset as shown in Table 3b, ConvNeXt is still the best performing model, achieving the highest ACC, F1, PRE, SEN, and SPE of 0.933, 0.933, 0.933, 0.934, and 0.932, respectively. ViT, the second best performing model, obtained an ACC, F1, PRE, SEN, and SPE of 0.919, 0.918, 0.929, 0.907, and 0.931, respectively. The worst performing model, SHViT, obtained the lowest ACC, F1, and SEN of 0.858, 0.855, and 0.839, respectively. These reported performance results illustrate the high-performance results of ConvNeXt. Table 3Generalization (testing) results of the eight DTL models pertaining to (a) CT and (b) X-ray datasets.(a)(b)ModelACCF1PRESENSPEModelACCF1PRESENSPEConvNeXt 0.894 0.900 0.8730.9290.868ConvNeXt 0.933 0.933 0.933 0.934 0.932 EfficientNetV20.8400.8490.7960.9100.771EfficientNetV20.8600.8630.8420.8860.834DenseNet1210.8870.8900.8560.9270.847DenseNet1210.9090.9090.9090.9100.909ResNet340.8580.8610.8330.8920.824ResNet340.8870.8860.8870.8860.888FasterViT0.8700.8770.8260.9340.808FasterViT0.8880.8900.8770.9040.873SHViT0.8390.8430.8130.8760.802SHViT0.8580.8550.8710.8390.876Swin0.8870.8920.849 0.939 0.837Swin0.9140.9130.9210.9050.923ViT 0.894 0.896 0.877 0.915 0.874 ViT0.9190.9180.9290.9070.931ACC is accuracy. PRE is precision. SEN is sensitivity. SPE is specificity. Bold indicates a model obtaining the highest performance results. Table 3Generalization (testing) results of the eight DTL models pertaining to (a) CT and (b) X-ray datasets.(a)(b)ModelACCF1PRESENSPEModelACCF1PRESENSPEConvNeXt 0.894 0.900 0.8730.9290.868ConvNeXt 0.933 0.933 0.933 0.934 0.932 EfficientNetV20.8400.8490.7960.9100.771EfficientNetV20.8600.8630.8420.8860.834DenseNet1210.8870.8900.8560.9270.847DenseNet1210.9090.9090.9090.9100.909ResNet340.8580.8610.8330.8920.824ResNet340.8870.8860.8870.8860.888FasterViT0.8700.8770.8260.9340.808FasterViT0.8880.8900.8770.9040.873SHViT0.8390.8430.8130.8760.802SHViT0.8580.8550.8710.8390.876Swin0.8870.8920.849 0.939 0.837Swin0.9140.9130.9210.9050.923ViT 0.894 0.896 0.877 0.915 0.874 ViT0.9190.9180.9290.9070.931ACC is accuracy. PRE is precision. SEN is sensitivity. SPE is specificity. Bold indicates a model obtaining the highest performance results. Generalization (testing) results of the eight DTL models pertaining to (a) CT and (b) X-ray datasets. ACC is accuracy. PRE is precision. SEN is sensitivity. SPE is specificity. Bold indicates a model obtaining the highest performance results. Figure 7a,b unveil computational insights for the generalization capability of the four studied DTL models, applied to testing images pertaining to CT and X-ray datasets, respectively. Prediction differences for ConvNeXt (and all other models) between COVID-19 and non-COVID-19 images were statistically significant (P < \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.2 \times 10^{ - 16}$$\end{document} from a t-test), indicating that all the models are general predictors for COVID-19 or non-COVID-19 image prediction. Similarly, for the X-ray dataset in Fig. 7b, prediction differences in all models were statistically significant (P < \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.2 \times 10^{ - 16}$$\end{document} from a t-test), demonstrating that these models are general, rather than specific predictors for COVID-19 prediction. Fig. 7Boxplots and strip charts of predicted COVID-19 and non-COVID-19 images on all testing folds of the four DTL models pertaining to (a) CT and (b) X-ray datasets. Boxplots and strip charts of predicted COVID-19 and non-COVID-19 images on all testing folds of the four DTL models pertaining to (a) CT and (b) X-ray datasets. After carrying out the model generation process explained in “Model generation” section., we obtained 10,000 models and plotted their AUC results when applied to CT and X-ray datasets, as displayed in Fig. 8a,b, respectively. For the CT dataset in Fig. 8a, we selected the best model, which we named HC, constructed as follows: Let h1(z), h2(z), h3(z), and h4(z) correspond to the DTL models obtained from training ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34, respectively, and applied to an unseen image, z. In Eq. (9), h11(z) corresponds to a majority vote pertaining to two classifiers, namely ConvNeXt and EfficientNetV2, applied to a testing example. I is an indicator function resulting in 1 if the arguments are true and 0 otherwise. h12(z) in Eq. (10) corresponds to the majority vote of two classifiers (i.e., ConvNeXt and DenseNet121). In Eq. (11), h19(z) corresponds to a majority vote of three classifiers (i.e., ConvNeXt, DenseNet121, and ResNet34). HC in Eq. (12) is constructed as a majority vote mechanism of h11(z), h12(z), and h19(z). Figure S1 in the Supplementary Additional File provides an additional illustration for HC.9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{1} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,2\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{2} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,3\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{9} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,3,4\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{HC}}\left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,2,9\} }} {I\left( {h1_{i} \left( \text{z} \right) = y} \right)}$$\end{document} Fig. 8AUC performance results of 10,000 generated models pertaining to (a) CT and (b) X-ray datasets. AUC performance results of 10,000 generated models pertaining to (a) CT and (b) X-ray datasets. The AUC for HC is 0.909 as illustrated in Fig. 9a. For the X-ray dataset as illustrated in Fig. 8b, we selected the best model attaining the highest AUC performance and designated it HX, working as shown in Eqs. (13–16) as follows: h12(z) in Eq. (13) corresponds to a majority vote pertaining to two classifiers (i.e., ConvNeXt and DenseNet121). h13(z) in Eq. (14) is the majority vote of ConvNeXt and ResNet34. h19(z) in Eq. (15) is just the majority vote of ConvNeXt, DenseNet121, and ResNet34. In Eq. (16), HX is the majority vote of h12(z), h13(z), and h19(z). We report the obtained AUC results via HX in Fig. 9b. Figure S2 in the Supplementary Additional File provides an additional illustration for HX. Fig. 9ROC curves and recorded AUC performance results of eight DTL models against the generated models HC and HX to (a) CT and (b) X-ray datasets. ROC curves and recorded AUC performance results of eight DTL models against the generated models HC and HX to (a) CT and (b) X-ray datasets. Compared to transformer-based models including FasterViT, SHViT, Swin, and ViT using the CT dataset (see Fig. 9a), HC achieves 3.8%, 7.0%, 2.1%, and 1.4% AUC performance improvements over the FasterViT, SHViT, Swin and ViT models, respectively. ConvNeXt yields 2.8%, 6.0%, 1.1%, 0.4% AUC performance improvements over FasterViT, SHViT, Swin, and ViT, respectively. When considering the X-ray dataset as shown in Fig. 9b, HX yields 4.2%, 7.2%, 1.6%, and 1.1% AUC performance improvements over the FasterVit, SHViT, Swin and ViT models, repressively. ConvNeXt generates 4.5%, 7.5%, 1.9%, and 1.4% AUC performance improvements over FasterViT, SHViT, Swin, and ViT, respectively.13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{2} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,3\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{3} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,4\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h1_{9} \left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 1,3,4\} }} {I\left( {h_{i} \left( \text{z} \right) = y} \right)}$$\end{document}16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{HX}}\left( \text{z} \right) = \mathop {\arg \max }\limits_{y} \sum\limits_{{i \in \{ 2,3,9\} }} {I\left( {h1_{i} \left( \text{z} \right) = y} \right)}$$\end{document} Figure 10a,b report the confusion matrices calculated based on testing for CT and X-ray datasets, respectively. When considering the CT dataset, it can be shown that HC is the best model, accurately predicting 2254 out of 2481 CT images. One hundred and twenty-one images were predicted as non-COVID-19 while their actual label was COVID-19, and thereby counted as one hundred and twenty-one FP. One hundred and six images were predicted as COVID-19 while their actual label was non-COVID-19, therefore counted as one hundred and six FN. Fig. 10The combined confusion matrices of five testing folds during the running of five-fold cross-validation for the studied models pertaining to (a) CT and (b) X-ray datasets. The combined confusion matrices of five testing folds during the running of five-fold cross-validation for the studied models pertaining to (a) CT and (b) X-ray datasets. The second best model is ConvNeXt, accurately predicting 2229 out of 2481 CT images. One hundred and sixty-five images were predicted by ConvNeXt as non-COVID-19 while their actual label was COVID-19, and thereby counted as one hundred and sixty-five FP. Eighty-seven images were incorrectly predicted as COVID-19 while their actual label was non-COVID-19, therefore counted as eighty-seven FN. For the X-ray images, ConvNeXt accurately predicts 1866 (934 TP + 932 TN) images, which is marginally higher than HX (1861 calculated from 907 TP + 954 TN). Both ConvNeXt and HX inaccurately predict 134 (68 FP + 66 FN) and 139 (46 FP + 93 FN) images, respectively. In terms of transformer-based models (i.e., FasterViT, SHViT, Swin, and ViT) when using the CT dataset, the combined confusion matrices of Fig. 10a demonstrate that FasterViT, SHViT, Swin, and ViT accurately predict 2159, 2081, 2203, and 2220 images, respectively, out of 2481 CT images. HC, which accurately predicted 2254 CT images (see Fig. 10a), had 95, 173, 51, and 34 more accurately predicted images than FasterViT, SHViT, Swin, and ViT, respectively. FasterViT, SHViT, Swin, and ViT had 241, 248, 204, and 157 images, respectively, predicted as non-COVID-19 while their actual label was COVID-19, and thereby counted as 241 FP, 248 FP, 204 FP, and 157 FP (compared to 121 FP for HC), respectively. Moreover, FasterViT, SHViT, Swin, and ViT had 81, 152, 74, and 104 images, respectively, predicted as COVID-19 while their actual label was non-COVID-19, and thereby counted as 81 FN, 152 FN, 74 FN and 104 FN (compared to 106 FN for HC), respectively. For the X-ray dataset as shown in Fig. 10b, FasterViT, SHViT, Swin, and ViT had a total of 1777, 1715, 1828 and 1838 accurately predicted images, respectively, out of 2000 (compared to 1861 for HX as shown in Fig. 10b). Hence, HX had 84, 146, 33, and 23 more accurately predicted images than FasterViT, SHViT, Swin, and ViT, respectively. FasterViT, SHViT, Swin, and ViT had 127, 124, 77, and 69 images, respectively, that were predicted as non-COVID-19 while their actual label was COVID-19; therefore, they had 127 FP, 124 FP, 77 FP, and 69 FP (compared to 46 FP for HX), respectively. FasterViT, SHViT, Swin, and ViT had 96 FN, 161 FN, 95 FN, and 93 FN, respectively, compared to 93 FN for HX. Figure 11 provides a summary for the number of updated parameters during the training process. FasterViT (tie with ResNet34) has the least number of parameters when compared to all other models including transformer-based models (SHViT, Swin, and ViT). Additional details about layers and number of parameters pertaining to transformer-based models are provided in Supplementary Additional File (see Table S1). Fig. 11An overview for the number of updated parameters pertaining to each studied model. An overview for the number of updated parameters pertaining to each studied model. Discussion To conduct a large empirical study on COVID-19 classification using CT and X-ray images, our computational approach comprises three parts: data preprocessing and DTL followed by model generation. In terms of the data preprocessing part, CT images composed of 2481 images pertaining to COVID-19 and non-COVID-19 patients were downloaded from Kaggle at https://www.kaggle.com/datasets/plameneduardo/sarscov2-ctscan-dataset. The 2000 X-ray images regarding COVID-19 and non-COVID-19 were downloaded from Kaggle at https://www.kaggle.com/tawsifurrahman/covid19-radiography-database. We prepared all PNG images to be of size 256 × 256. For the DTL computation, we employed four pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. Each of these DTL models was already trained on more than a million images from the ImageNet database. We extracted features from COVID-19 images using weights obtained from pre-trained models on the ImageNet database, followed by a flattening step before training from scratch the densely connected classifier with the Adam optimizer and binary cross-entropy with a logit loss function. The model generation part receives four DTL models. Then, taking the majority of two, three, and four combinations from the four DTL models resulted in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathop \sum \nolimits_{r = 2}^{4} \left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right) =$$\end{document} 11 constructed models. After that, taking the majority of 2, 3, …, 11-combinations from the 11 models resulted in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathop \sum \nolimits_{r = 2}^{11} \left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right) =$$\end{document} 2036 constructed models. Finally, we selected 7953 models from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document}. In our computational approach, it can be seen from Eqs. (1–3) that h3l depends on h2i, which also depends on h1i. Therefore, this demonstrates the sequential nature for constructing these generated models. For the HC model as shown in Eqs. (9–12), Eqs. (9, 10, and 11) are exactly like calculations in Eq. (1), while HC in Eq. (12) is like h2k in Eq. (2). Hence, it can be viewed as HC(h11,2,9(z)). The same computational nature holds true for HX corresponding to HX(h12,3,9(z)). Figure 8 reports the AUC for 10,000 models applied to CT and X-ray datasets, respectively. In Supplementary ctLGM and xrayLGM, we report the 10,000 constructed models obtained from the CT and X-ray datasets, respectively. To ensure the fairness of the experimental comparisons, we utilized five-fold cross-validation in which in all of the five runs the training is used to induce models while testing is used to perform prediction and evaluation. The sum of entries of each confusion matrix in Fig. 10 is equal to the number of images in the studied datasets. The sum of each confusion matrix in Fig. 10a is equal to 2481. Similarly, the sum of entries in each confusion matrix in Fig. 10b is equal to 2000. This demonstrates that we evaluated the performance on the studied datasets. The generalization performance is based on evaluating the prediction performance of unseen images against their ground truth, and thereby, it is achieved in a try-and-see manner. Additionally, DTL models in our study are computationally efficient, attributed to few updated weight layers in the classification layer during the training process. In terms of reducing the overfitting effect for induced DTL models, we included a dropout layer in the classification part pertaining to the densely connected classifier, setting the ratio to 0.2. Furthermore, we transferred weights in the feature extraction part of pre-trained models already trained on more than one million images while training from scratch the classification part in which weights in those layers are changed. In other words, incorporating a dropout layer as well as transfer learning via freezing all layers in the feature extraction part contributed to the mitigation of overfitting in this study70 . Table 1 provides an overview of 20 existing studies, showing that our proposed study is unique when considering the number of models (see the no. of models column). There is no large empirical DL study to date, to the best of our knowledge, reporting and investigating the feasibility of DL when tackling the COVID-19 classification task using CT and X-ray images like ours. Specifically, for each imaging technique (i.e., CT or X-ray), we reported the results of eight DTL models plus 10,000 generated DTL models (see Fig. 8). These results show the power of our computational approach, and we have 10,008 reported results for CT images + 10,008 reported results for X-ray images, resulting in a total of 20,016 reported results. For the conducted experiments, we did not perform data augmentation while we normalized the images with a specific mean and standard deviation using the normalize function in the torchvision library. Moreover, we found that a batch size of 32 with 10 epochs works better in our experiments with the small datasets at hand than a larger number of epochs. Hence, we halted the training after 10 epochs. Our computational framework has both advantages and disadvantages. Some advantages include (1) the ability to expand the model space in our study to thousands of models in less than two hours, attributed to fast training time under the transfer learning setting, and (2) the availability of a large number of models that can be exploited via clustering models of similar performance differences for use in learning scenarios such as active learning. However, our framework requires additional training time in cases demanding large DL models obtained via training from scratch in which scaling to thousands of models becomes more expensive. We employed the majority vote mechanism within the performed combinations to create new models and thereby expanding the model space, potentially identifying a better performing model. These constructed models can aid in the search for better performance models in which more automatically constructed models increase the odds for identifying a better performing model, as reported in our experimental results. Moreover, constructed models can be useful in active learning strategies such as the committee construction in the query by committee71,72. Conclusions and future work In this work, we presented a novel computational approach with which to augment and expand empirical studies for the task of COVID-19 classification using CT and X-ray images. First, we downloaded and processed 2481 CT and 2000 X-ray images from Kaggle for the COVID-19 binary classification task. Then, we provided processed images of each imaging technique to four pre-trained DL models, including ConvNeXt, EfficientNetV2, DenseNet121, and ResNet34. To employ transfer learning, we froze all layers, including convolutional and pooling layers, in the feature extraction part applied to COVID-19 images while unfreezing and updating weights during the training phase in the classification part of the densely connected classifier using the Adam optimizer and binary cross-entropy with a logit loss function. The generation of models was achieved in a sequential manner, generating \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathop \sum \nolimits_{r = 2}^{4} \left( {\begin{array}{*{20}c} 4 \\ r \\ \end{array} } \right) = 11$$\end{document} DTL models through taking the majority vote of two, three, and four combinations from the four DTL models, and then generated \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathop \sum \nolimits_{r = 2}^{11} \left( {\begin{array}{*{20}c} {11} \\ r \\ \end{array} } \right) = 2036 \ {\text{DTL}}$$\end{document} models through taking majority vote of 2, 3, …, 11 combinations from 11 DTL models. Finally, 7953 DTL models were generated from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ }\left( {\begin{array}{*{20}c} {2036} \\ 2 \\ \end{array} } \right)$$\end{document}. For each imaging technique, we report results for 10,004 (4 + 11 + 2036 + 7953) DTL models plus 4 transformer-based models. Generalization (testing) results demonstrate that a generated DTL model, named HC, generated the highest AUC of 0.909 when applied to COVID-19 CT images. For the COVID-19 X-ray images, ConvNeXt generated the highest AUC of 0.933, which is marginally higher than the generated DTL, named HX, yielding an AUC of 0.930. The results attributed to the computational approach successfully demonstrate how to scale empirical studies as well as set a unique record for COVID-19 classification. Future work can include the following: (1) adopting the presented computational approach to generate and establish a committee of models to be utilized in learning algorithms under an active learning scenario; (2) adjusting the computational approach to construct models via interleaved weighted majority vote; (3) expanding the model generation to reach one million constructed models; and (4) employing the presented approach to augment and enlarge empirical studies for other problems in biology and medicine. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1