A few-shot diabetes foot ulcer image classification method based on deep ResNet and transfer learning Authors: Cheng Wang, Zhen Yu, Zhou Long, Hui Zhao, Zhenwei Wang Published: 2024 ABSTRACT AbstractDiabetes foot ulcer (DFU) is one of the common complications of diabetes patients, which may lead to infection, necrosis and even amputation. Therefore, early diagnosis, classification of severity and related treatment are crucial for the patients. Current DFU classification methods often require experienced doctors to manually classify the severity, which is time-consuming and low accuracy. The objective of the study is to propose a few-shot DFU image classification method based on deep residual neural network and transfer learning. Considering the difficulty in obtaining clinical DFU images, it is a few-shot problem. Therefore, the methods include: (1) Data augmentation of the original DFU images by using geometric transformations and random noise; (2) Deep ResNet models selection based on different convolutional layers comparative experiments; (3) DFU classification model training with transfer learning by using the selected pre-trained ResNet model and fine tuning. To verify the proposed classification method, the experiments were performed with the original and augmented datasets by separating three classifications: zero grade, mild grade, severe grade. (1) The datasets were augmented from the original 146 to 3000 image datasets and the corresponding DFU classification’s average accuracy from 0.9167 to 0.9867; (2) Comparative experiments were conducted with ResNet18, ResNet34, ResNet50, ResNet101, ResNet152 by using 3000 image datasets, and the average accuracy/loss is 0.9325/0.2927, 0.9276/0.3234, 0.9901/0.1356, 0.9865/0.1427, 0.9790/0.1583 respectively; (3) Based on the augmented 3000 image datasets, it was achieved 0.9867 average accuracy with the DFU classification model, which was trained by the pre-trained ResNet50 and hyper-parameters. The experimental results indicated that the proposed few-shot DFU image classification method based on deep ResNet and transfer learning got very high accuracy, and it is expected to be suitable for low-cost and low-computational terminal equipment, aiming at helping clinical DFU classification efficiently and auxiliary diagnosis.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-024-80691-w.Keywords: Diabetes foot ulcer, Classification method, Deep learningSubject terms: Biomedical engineering, Computer science Introduction Diabetes foot ulcer (DFU) is one of the common complications of diabetes patients, which may lead to infection, necrosis and even amputation1–3. Therefore, early diagnosis, classification of severity and related treatment are crucial for the patients4. The classification of severity is a part of diagnosis and the basic step of treatments for DFU. In clinics, current classification method of DFU severity is mainly based on Wagner or Texas5 scale measurement by experienced doctors, and it is subjective and requires high professionalism. Recently, many artificial intelligence and machine learning related algorithms had been applied to the classification of DFU. Traditional machine learning based DFU classification methods (SVM, KMA) mostly performed binary classification of ischemia or non-ischemia, infection or non-infection, and achieved high accuracy6–8. However, it has the following disadvantages: manual feature extraction, time-consuming for processing a large number of DFU images, and limited application with coarse (binary) classification9,10. Different from traditional machine learning, deep learning can automatically extract features11, and it played an important role in medical related measurements, evaluations and diagnosis12–16. At present, deep learning based DFU classification methods (CNN, GoogleNet, VGG16etc) mostly also performed binary classification17–19, and the studies of DFU multi-classifications is rare, and the accuracy is not high20. Considering the difficulty in obtaining clinical DFU images21, it is a few-shot problem. Thus, based on the above study backgrounds of DFU classification, the study aims to propose a three-classification method for few-shot DFU images based on deep ResNet22 and transfer learning23, with the ultimate goal of yielding high multi-classification accuracy for DFU images24. The main contributions of this paper are as follows: (1) A DFU images datasets photographed and marked by professional clinicians are provided, including 146 original images and 3000 augmented images. All images are made public on GitHub (https://github.com/git-yuzhen/DFU-classification). (2) A classification method combining deep learning and transfer learning is proposed to achieve a new three-classification. The remainder of the article is organized as follows. “Materials and methods” provides a brief overview of a three-classification method for few-shot DFU images based on deep ResNet and transfer learning, and related experimental results and analyses are given in “Experiments and results” and “Discussion”, “Conclusion” summarizes the article. Materials and methods The research framework of the article is show in Fig. 1. A few-shot DFU image classification method based on deep ResNet and transfer learning. Data obtaining Data obtaining environment: including images of different cases and different stages. The picture was taken by a clinician using Apple 13pro max, and the lens was parallel to the ulcer surface, with a distance of 30–40 cm. The shooting environment is indoors under natural light or in an operating room with sufficient light. A total of 146 original images were taken, and the taken photos were marked and divided into three classifications by doctors with professional clinical qualifications: 27 images for zero grade, 45 images for mild grade and 74 images for severe grade. The principle of the three-classification is based on Wagner classification and DFU severity (illustrated in Fig. 2) as it is showed in Table 1. The reason why three classifications were categorized are that: (1) From the perspective of patients and doctors, they are more concerned about the presence (zero grade) and severity (mild grade, severe grade) of DFU diseases. (2) The merging of Wagner’s fine-grained five classification into coarse-grained three classification is also due to the heavy workload of doctors in labeling DFU images. New class (a) zero grade, (b) mild grade, (c) severe grade. Table 1Wagner classification merging from five to three (new class).WagnerGrade 0Grade 1Grade 2Grade 3Grade 4Grade 5New classZero gradeMild gradeSevere grade*Zero grade: No obvious infection, skin surface intact but with erythema.Mild grade: Presence of infection, but the ulcer is shallow and does not affect deep tissues.Severe grade: Infection penetrates deep tissues, with necrosis, possibly involving bone. Wagner classification merging from five to three (new class). *Zero grade: No obvious infection, skin surface intact but with erythema. Mild grade: Presence of infection, but the ulcer is shallow and does not affect deep tissues. Severe grade: Infection penetrates deep tissues, with necrosis, possibly involving bone. This study is a retrospective study, following the ethical guidelines of Helsinki Declaration and approved by the Ethics Committee of Beijing Chao-Yang Hospital, Capital Medical University (ethical approval number: 2022-ke-629). All patients provided written informed consent before operation. Data augmentation In order to improve the generalization ability of the DFU classification model, the original datasets were augmented in the diversity and quantity by techniques: scaling25, rotating (90°), flipping (horizontal flipping, vertical flipping), brightness changing (brightening, darkening), moving (pixel shifting26,27), adding noise (salt-and-pepper noise, gaussian noise28,29), and the detailed parameters of these transformations are showed in Table 2. The datasets were divided into training set, test set and verification set for training and evaluation of the model. Images of the datasets after data augmentation are illustrated in Fig. 3. Table 2Techniques and parameters of data augmentation.Transformation nameExplanation and parametersScalingImages are randomly scaled between 50–200% to simulate different distances between patients’ foot and cameraRotatingImages are rotated at different angles (90°) to simulate various shooting locationFlipping (horizontal flipping, vertical flipping)Images are rotated at different angles (180°,270°) to simulate various shooting locationBrightness changing (brightening, darkening)The brightness of the images is randomly varied within a ± 10% range to simulate different lighting conditionsMoving (pixel shifting)The pixels of the image are randomly shifted 10 units to the left and rightAdding noise (salt-and-pepper noise, gaussian noise)Salt-and-pepper noise (noise density 0.05) and gaussian noise (mean 0, variance 0.01) and are added to simulate interference during. image capture Techniques and parameters of data augmentation. DFU images augmentation mode (a) original image, (b) rotation, (c) flipped horizontally, (d) adding noise, (e) brightening, (f) darkening, (g) flipped vertically, (h) moving. Deep ResNet models selection Resnet series models were selected as the baseline models: ResNet18, ResNet34, ResNet50, ResNet101 and ResNet152 were tested respectively, and the model with the best experimental results was selected as the pre-trained model. The model was trained on the training set, and the cross-entropy error was defined as the loss function. L represents the cross-entropy error, defining m samples, n categories, represents the true labeling of the k-th category for the i-th sample, and represents the predicted probability of the model for the k-th category for the i-th sample. The stochastic gradient descent method was defined as the optimizer. g represents the gradient, ∇θL(xi,θ) represents the gradient of the loss function for all samples, m samples are randomly selected from within n samples at a time, ε is the learning rate. During the training process, we monitor the performance of the model on the verification set to avoid over-fitting. Evaluation indicators include average accuracy and average loss. In this paper, the average accuracy is defined as this formula: represents the average accuracy, n represents the total number of rounds of training, and represents the accuracy of the i-th round of training. The average loss is defined as this formula: represents the average loss, n represents the total number of rounds of training, and represents the loss of the i-th round of training. According to the above method, the detailed model selection comparison experiments and hyper-parameters setting were conducted in Experiment 2. According to Experiment 2, the average accuracy/average loss of ResNet18, ResNet34, ResNet50, ResNet101 and ResNet152 are 0.9325/0.2927, 0.9276/0.3234, 0.9901/0.1356, 0.9865/0.1427 and 0.9790 respectively, and ResNet50 was finally selected. DFU classification model training DFU classification model training method is using the pre-trained model and hyper-parameters fine-tuning, and ResNet50 is the pre-trained model. In deep learning, the selection of hyper-parameters usually includes: image_size, learning rate, learning rate decay, batch_size, number of rounds (epoch)30. Detailed value of these parameters is showed in Experiment 3. In order to avoid overfitting and ensure the generalization of the model, data augmentation was performed based on the original datasets as shown in Data augmentation, and L2 regularization technique31 was applied to the model training process. Experiments and results Windows 10 was used as the operating system for the experiment, PyTorch version 2.2.0 was used as the deep learning development framework, and Python version 3.8 was used as the development language. The CPU used in the experiment was Intel Core i5-8250U, and the GPU was NVIDIA GeForce RTX 3060. The detailed experimental environment is showed in Table 3. Table 3Experimental environment.Operating systemWindows 10Python3.8PyTorch2.2.0Cuda10.1CPUIntel Core i5-8250UGPUNVIDIA GeForce RTX 3060 Experimental environment. In this section, three experiments were conducted. Experiment 1 was to verify the effect of DFU data augmentation. Experiment 2 is to select pre-trained model. Experiment 3 is to verify the effectiveness of DFU classification model. And the experimental related data are showed in Table 4. Table 4Experimental data of DFU.DFU datasetsA*TrainTestValidZero grade271935Mild grade453159Severe grade7452715Total1461021529DFU datasetsB*TrainTestValidZero grade1000700100200Mild grade1000700100200Severe grade1000700100200Total30002100300600*DFU datasets A is the original datasets, and DFU datasets B is the augmented datasets. Experimental data of DFU. *DFU datasets A is the original datasets, and DFU datasets B is the augmented datasets. Experiment 1: After data augmentation, the datasets A had been increased from 146 (including 27 for zero grade, 45 for mild grade and 74 for severe grade) to 3,000 (including 1000 for zero grade, 1000 for mild grade and 1000 for severe grade). And the average accuracy of DFU classification reached 0.9867 from 0.9167, as it is showed in Table 5. Table 5Comparison experiment of using pre-trained ResNet50 model on datasets a and datasets B.DFU datasetsAverage accuracyA0.9167B0.9867 Comparison experiment of using pre-trained ResNet50 model on datasets a and datasets B. Experiment 2: Cuda is used to train the datasets B with different ResNet models. The cross-entropy error was defined as the loss function. The stochastic gradient descent method was defined as the optimizer. The learning rate was set to 0.001. The random seed was 42. The last fully connected layer of the models was modified to match the number of categories of the datasets and make three classifications. Compared with ResNet18, ResNet34, ResNet50, ResNet101 and ResNet152, datasets B were used with a total of 5 rounds training, and the average accuracy/loss were 0.9325/0.2927, 0.9276/0.3234, 0.9901/0.1356, 0.9865/0.1427, and 0.9790/0.1583 respectively. The experimental results are showed in Table 6; Fig. 4. Table 6Comparative experiment of different residual models in the datasets B.ModelResNet18ResNet34ResNet50ResNet101ResNet152Accuracy0.93250.92760.99010.98650.9790Avg loss0.29270.32340.13560.14270.1583 Comparative experiment of different residual models in the datasets B. Comparative experiment of different residual models in the datasets B. Experiment 3: The datasets A and the datasets B were trained respectively using the selected ResNet50 as a pre-trained model. Record the experimental results, including average loss, average accuracy. The training hyper-parameters of ResNet50 for the datasets A are: img_size = 224, learning rate = 0.001, learning rate decay = 0.00001, batch_size = 6, epochs = 20. The average accuracy of the experiment was 0.9167 and the average loss was 0.2546. The training results of the datasets A are showed in Fig. 5. The training hyper-parameters of ResNet50 with the datasets B are: img_size = 224, learning rate = 0.001, learning rate decay = 0.00001, batch_size = 6, epochs = 5. The classification accuracy for three severity categories (zero, mild and severe) is showed in Table 7 and the average accuracy was 0.9867. The training results of the datasets B are showed in Fig. 6. Pre-trained, datasets A. Table 7The accuracy of classification under three severity categories (zero, mild and severe).ClassAccuracyZero grade0.9894Mild grade0.9820Severe grade0.9887Average0.9867 The accuracy of classification under three severity categories (zero, mild and severe). Pre-trained, datasets B. Discussion The images were shot by professional clinicians in a standardized way, and 146 original data sets (datasets A) were obtained, of which 27 were zero grade, 45 were mild grade and 74 were severe grade. The shooting location is usually collected in an outpatient clinic with sufficient natural light or an operating room with sufficient light. The datasets A are augmented by using geometric transformation and random noise, and the amplified datasets reach 3000, including 1000 for zero grade, 1000 for mild grade and 1000 for severe grade. After data augmentation, the uneven image classification is balanced. The paper proposed a few-shot DFU image classification method based on deep residual neural network and transfer learning, and related experiments are given. From the experimental results of Experiment 2, it can be found that the comparative experiments of ResNet18, ResNet34, ResNet50, ResNet101 and ResNet152 were trained on dataset B respectively. ResNet50 has the best performance, so ResNet50 was finally selected as the trained model. Experimental results of Experiment 1&3 show that the proposed method has achieved excellent performance in the image classification task of DFU, and the augmented data obviously helped to improve the experimental accuracy. After data augmentation, the accuracy of the model on the verification set reached 0.9867 from 0.9167. Meanwhile, it also indicates the generalizability of the model has good result. The accuracy of zero grade (0.9894) and severe grade (0.9887) is relatively higher than the accuracy of mild grade (0.9820) as it is showed in Table 7, it is because the characteristics of zero grade and severe grade are more obvious than mild grade (Fig. 2) and the model can extract features more accurately. It showed that the proposed method can accurately classify DFU images. Although, the study achieved good results on the augmented dataset from the experimental results, there is some limitations. For example, images are collected in a standardized and experimental environment, and the influence of other complex environments on the accuracy of image classification is not considered. Therefore, considering the less controlled environments of real-world, in order to improve applicability, image data from actual clinical environments will be used to further optimize the performance of the model in future work. In addition, multimodal data (such as patient medical history information) fusion and multiple environmental factors (such as lighting conditions, image resolution, etc.) fusion will be utilized to further improve the accuracy of the model. The interpretability of medical diagnosis model has been paid more and more attention by researchers32. For example, Grad-CAM33 will be considered to enhance trust and understanding of the model’s decisions in clinical settings in our future work. Conclusion In the study, a few-shot DFU image classification method based on deep residual neural network and transfer learning is proposed to solve the low accuracy and manual classification of DFU by clinical doctors. The experimental results show that the proposed method has achieved good performance in the image classification task of DFU, and it is expected to be suitable for low-cost and low-computational terminal equipment, aiming at helping clinical DFU classification efficiently and auxiliary diagnosis. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1 Author contributions Zhen Yu did the experiments. Zhen Yu, Zhou Long, Zhenwei Wang and Cheng Wang analyzed experimental data. Cheng Wang and Hui Zhao designed the research. Zhenwei Wang directed the writing of the manuscript. Zhen Yu and Cheng Wang together as co-author wrote the manuscript, and both are the first authors of the manuscript. All authors contributed to the article and approved the submitted version. Funding This work is partly supported by Shandong Province Science and Technology Small and Medium sized Enterprise Innovation Capability Enhancement Project (2022TSGC2532), 2025 Key Tech-nology Innovation Program of Ningbo(2020Z014) and Capital Health Development Research Special Project (2024-2-7042). The authors are grateful for these supports. Data availability The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation. Any data or questions regarding the study, please kindly contact Zhen Yu(yuzhenqd@qq.com) and Cheng Wang (corresponding author, wangcheng01@ict.ac.cn). Declarations Competing interests The authors declare no competing interests. Ethics statement The study was reviewed and approved by the institutional review board of Beijing Chao-Yang Hospital, Capital Medical University (ethical approval number is 2022-ke-629). All participants have signed informed consent forms. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Cheng Wang, Email: wangcheng01@ict.ac.cn. Zhenwei Wang, Email: fmmuwzw@163.com. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation. Any data or questions regarding the study, please kindly contact Zhen Yu(yuzhenqd@qq.com) and Cheng Wang (corresponding author, wangcheng01@ict.ac.cn).