An Improved Microaneurysm Detection Model Based on SwinIR and YOLOv8 Authors: Bowei Zhang, Jing Li, Yun Bai, Qing Jiang, Biao Yan, Zhenhua Wang Published: 2023 ABSTRACT Diabetic retinopathy (DR) is a microvascular complication of diabetes. Microaneurysms (MAs) are often observed in the retinal vessels of diabetic patients and represent one of the earliest signs of DR. Accurate and efficient detection of MAs is crucial for the diagnosis of DR. In this study, an automatic model (MA-YOLO) is proposed for MA detection in fluorescein angiography (FFA) images. To obtain detailed features and improve the discriminability of MAs in FFA images, SwinIR was utilized to reconstruct super-resolution images. To solve the problems of missed detection of small features and feature information loss, an MA detection layer was added between the neck and the head sections of YOLOv8. To enhance the generalization ability of the MA-YOLO model, transfer learning was conducted between high-resolution images and low-resolution images. To avoid excessive penalization due to geometric factors and address sample distribution imbalance, the loss function was optimized by taking the Wise-IoU loss as a bounding box regression loss. The performance of the MA-YOLO model in MA detection was compared with that of other state-of-the-art models, including SSD, RetinaNet, YOLOv5, YOLOX, and YOLOv7. The results showed that the MA-YOLO model had the best performance in MA detection, as shown by its optimal metrics, including recall, precision, F1 score, and AP, which were 88.23%, 97.98%, 92.85%, and 94.62%, respectively. Collectively, the proposed MA-YOLO model is suitable for the automatic detection of MAs in FFA images, which can assist ophthalmologists in the diagnosis of the progression of DR. 1. Introduction Diabetic retinopathy (DR) is one of microvascular complications affecting the retina, caused by diabetes. It is also known as a major cause of blindness worldwide [1,2]. The pathogenesis of DR is tightly associated with an altered vessel structure due to increased blood glucose levels. Initially, DR presents as tiny dilations of capillaries, known as microaneurysms (MAs) [3,4]. MAs are primarily distributed in the inner nuclear layer and deep capillary plexus and are often an early clinical manifestation of various retinal and systemic diseases, including DR, retinal vein occlusion, and infections. In fundus images, MAs appear as small dots and a visible pathology at the early stages of DR. Therefore, the accurate detection of MA is crucial for the prevention, diagnosis, and treatment of DR [5]. The advancement of modern retinal imaging techniques, such as fundus fluorescein angiography (FFA) and non-mydriatic fundus photography (NMFCS), has improved the identification of MAs. FFA is a technique that uses the injection of a contrast agent to observe the retinal vessels. NMFCS refers to a non-invasive imaging technique of capturing retinal images through fundus photography. Figure 1 shows images obtained by FFA and NMFCS. The fundus images obtained by FFA exhibit higher contrast and clearer presentation of the features of the retinal structure compared to the images obtained by NMFCS. In the clinical practice, FFA is widely recognized as an important standard for visualizing the retinal vasculature and describing subtle vascular changes. Figure 2 shows a normal FFA image and an FFA image with microaneurysms. In the FFA image, MAs typically appear as round and bright spot-like structures with diameters ranging from 10 μm to 100 μm. MAs hold an important value in disease diagnosis and screening. Their objective quantitative evaluation is still limited as it requires manual detection by experienced technicians. Over the last two decades, automatic detection models for MAs have rapidly developed based on deep learning. The convolutional neural network (CNN) is a deep learning algorithm that extracts features from images through multiple layers of convolution and pooling operations and utilizes the fully connected layers for classification or regression tasks. The CNN has achieved great success in the field of image processing and is widely used for object recognition and semantic segmentation. Object recognition using CNNs has great advantages such as high accuracy, application flexibility, automation, and real-time performance, providing support for practical applications such as SSD [6], RetinaNet [7], YOLOv5, YOLOv7 [8], and YOLOX [9]. Meanwhile, previous studies reported several segmentation models for the automatic detection of MAs. Liao et al. proposed a deep convolutional encoder–decoder network with a weighted dice loss for MA localization [10]. Xia et al. introduced a multi-scale model for detecting and classifying MAs using residual and efficient networks [11]. Chudzik et al. proposed a three-stage detection method as an alternative to the traditional five-stage MA detection. This study demonstrated successful transfer learning between small MA datasets [12]. Zhou et al. proposed a collaborative learning model based on a fine-tuning detection module in a semi-supervised manner to improve the performance of MA detection [13]. Xie et al. proposed a segmentation–emendation–resegmentation–verification framework to predict and correct detection errors in models, enhancing the detection of MAs [14]. Wang et al. utilized a region-based fully convolutional network (R-FCN) incorporating a feature pyramid network and an improved region proposal network for MA detection [15]. Guo et al. proposed a novel end-to-end unified framework for MA detection that utilizes multi-scale feature fusion and multi-channel bin loss [16]. Mateen et al. proposed a hybrid feature embedding approach using pre-trained VGG-19 and Inception-v3 for MA detection [17]. Kumar et al. trained a model of radial basis function neural network for MA detection [18]. Table 1 shows the strengths and weaknesses of the reported models for MA detection. The above-mentioned MA detection models based on deep learning have enhanced the efficiency of MA detection in FFA images. However, the tiny size of MAs, their low contrast with the background, and the lack of an annotated MA database still pose a great challenge for MA detection. Thus, further study is still required to design a novel detection method to enhance MA detection efficiency. MAs are relatively small in size and often appear as tiny and blurry lesions in retinal images, which are particularly pronounced in low-resolution images. They are often similar to the pixels of blood vessels. Super-resolution reconstruction is an image processing technique that can enhance the spatial resolution and detail clarity of an image by recovering high-resolution details from a low-resolution image. The Swin Transformer [19] has shown great promise as it integrates the advantages of both CNN and Transformer. The Swin Transformer processes large images using a self-attention mechanism and models long-range dependencies with a shifted window scheme. An image restoration model, SwinIR [20], was designed based on the Swin Transformer. SwinIR could not only enhance the detail features of MAs, but also improve the visibility and discriminability of MAs in FFA images. Except for tiny MAs in FFA images, sample imbalance and loss of information are two problems to be solved that affect the accuracy and efficiency of MA detection. YOLOv8 is an object recognition algorithm that is characterized by its ability to perform object localization and classification in a single forward pass. YOLOv8 contains a backbone, a neck, and a head. The neck section utilizes the path aggregation network (PAN)–feature pyramid network (FPN) structure for feature fusion [21,22]. FPN constructs a multi-scale feature pyramid by adding lateral connections and up-sampling layers to capture rich semantic information and better detect objects of different sizes. PAN addresses the issue of feature propagation in FPN by aggregating and propagating features through horizontal and vertical paths. PAN–FPN combines the strengths of FPN and PAN to provide powerful feature representation capabilities. The backbone and neck sections of YOLOv8 draw inspiration from the design principles of YOLOv7 ELAN. The C3 structure of YOLOv5 is replaced with the C2f structure in YOLOv8, which has a richer gradient flow, allowing for a better capturing of image details and contextual information. Given its powerful identification efficiency, multi-scale feature fusion, and contextual information capturing, the proposed MA detection model was designed based on YOLOv8. Therefore, an improved MA detection model for FFA images based on SwinIR and YOLOv8 is proposed, called MA-YOLO. The major contributions of this study proposing the new MA-YOLO model are as follows:SwinIR was used to reconstruct high-resolution FFA images, which could enhance the visibility and discriminability of MAs in FFA images.A detection layer was added to the YOLOv8 model, which could avoid feature information loss in shallow layers and improve the performance of MA detection.Transfer learning was utilized between high- and low-resolution images to expand the data samples and improve the generalization ability.Taking Wise-IoU as the bounding box regression loss, the loss function of MA-YOLO was improved, which could relieve the sample distribution imbalance problem and enhance the generalization performance. SwinIR was used to reconstruct high-resolution FFA images, which could enhance the visibility and discriminability of MAs in FFA images. SwinIR was used to reconstruct high-resolution FFA images, which could enhance the visibility and discriminability of MAs in FFA images. A detection layer was added to the YOLOv8 model, which could avoid feature information loss in shallow layers and improve the performance of MA detection. A detection layer was added to the YOLOv8 model, which could avoid feature information loss in shallow layers and improve the performance of MA detection. Transfer learning was utilized between high- and low-resolution images to expand the data samples and improve the generalization ability. Transfer learning was utilized between high- and low-resolution images to expand the data samples and improve the generalization ability. Taking Wise-IoU as the bounding box regression loss, the loss function of MA-YOLO was improved, which could relieve the sample distribution imbalance problem and enhance the generalization performance. Taking Wise-IoU as the bounding box regression loss, the loss function of MA-YOLO was improved, which could relieve the sample distribution imbalance problem and enhance the generalization performance. In addition, the proposed MA-YOLO model could calculate the MA area in FFA images, which would assist ophthalmologists in assessing the progression of DR. 2. Materials and Methods 2.1. Materials 2.1.1. Datasets The experimental dataset used in this study contained two datasets. The first dataset was collaboratively constructed by the Nanjing Medical University-Affiliated Eye Hospital and includes 1200 FFA images (768 × 868 pixels) from 1200 eyes of DR patients (age range, 31–81 years). Image acquisition was performed using the Heidelberg retina angiograph (Heidelberg Engineering, Germany) device. To ensure data quality, the dataset excluded images that were blurry or overexposed because of environmental factors or equipment materials. The second dataset originated from a study conducted at the Persian Eye Clinic (Feiz Hospital) at the Isfahan University of Medical Sciences. The dataset includes 70 retinal images (576 × 720 pixels) from a total of 70 patients, with 30 images classified as normal, and 40 images representing different stages of abnormality. Prior to image collection, each patient underwent a comprehensive ophthalmic evaluation, which involved medical history assessment, applanation tonometry, slit-lamp examination, dilated fundus biomicroscopy, and ophthalmoscopy [23]. Based on the above datasets, a total of 1240 FFA images were selected as the experimental dataset. All images were resized to 768 × 768 pixels and annotated by clinical doctors with more than 10 years of clinical experiences. The 1240 FFA images were independently divided into 992 training FFA images, 124 validation FFA images, and 124 test FFA images. 2.1.2. Implementation The hardware configuration used for the experiment was Ubuntu 20.04.5, 2GPUs, GPU NVIDIA RTX 2080ti and 1 GPU memory (11 GB). As software, we used the deep-learning framework Pytorch 2.0.0 and the programming language python 3.8. 2.1.3. Evaluation Metrics Five metrics were calculated to estimate the performance of MA detection [24], i.e., recall (Re), precision (Pre), F1 score (F1), average precision (AP), and frames per second (FPS). (1)Re=TP/(TP+FN) (2)Pre=TP/(TP+FP) (3)F1=2×Pre×Re/(Pre+Re) (4)AP=∫01 Pre(Re)dRe (5)FPS = frameNum elapsedTime TP, FP, and FN denote true positive regions, false positive regions, and false negative regions, respectively. frameNum is the number of FFA images inputted into the detection model; elapsedTime is the time consumed by the detection model. Re and Pre are the proportion of correct predictions in all MAs and the proportion of real MAs in the samples predicted as MAs. F1 is a balanced metric determined by precision and recall. AP is the area under the precision–recall (PR) curve, obtained by plotting recall on the x-axis and precision on the y-axis, based on a set of precision and recall values calculated at different thresholds. FPS is the number of FFA images inferred per second. 2.2. Methods Figure 3 illustrates the flowchart of the proposed MA-YOLO model. In this model, SwinIR was used for the super-resolution reconstruction of FFA images (Figure 3A). The MA detection layer was added to modify YOLOv8 (Figure 3B), and transfer learning was applied to increase the amount of data. The Wise-IoU loss was utilized to enhance the detection capability (Figure 3C). 2.2.1. Super-Resolution FFA Image Reconstruction Based on SwinIR MAs are small in size and usually appear as tiny and blurry structures in FFA images. The subtle features of MAs are easily lost in low-resolution images. Therefore, reconstructing high-resolution images is helpful for MA detection in FFA images. Here, SwinIR was employed to perform the super-resolution reconstruction of FFA images. SwinIR, an image restoration technique, contains three modules, i.e., shallow feature extraction, deep feature extraction, and HQ image reconstruction (high-quality image reconstruction modules). The shallow feature extraction module uses a convolution layer to extract shallow features, which are directly transmitted to the reconstruction module and preserve low-frequency information. The deep feature extraction module is mainly composed of residual Swin Transformer blocks (RSTB), each of which utilizes several Swin Transformer layers for self-attention and cross-window interaction. Additionally, a convolution layer was incorporated at the end of the block to enhance features. A residual connection was utilized to establish a shortcut for feature aggregation. Finally, both shallow and deep features were transmitted to the HQ image reconstruction module, which used the sub-pixel convolution layer [25] to up-sample the features for high-quality image reconstruction. Figure 4 illustrates the structure of the residual Swin Transformer block and the Swin Transformer layer. Based on SwinIR, the original FFA images with a size of 768 × 768 pixels were reconstructed into super-resolution FFA images with a size of 1536 × 1536 pixels and 2304 × 2304 pixels, respectively, which effectively enhanced the detail features of MAs. 2.2.2. YOLOv8 Modified by MA Detection Layer and Transfer Learning During the down-sampling of the convolution layers of YOLOv8, the regions containing MAs become blurred, and it was difficult to accurately localize the MAs. Thus, down-sampling convolution caused the loss of small features and missed and false MA detection. Here, an MA detection layer was introduced into the neck and head of YOLOv8 to handle shallow feature maps from the P2 layer of the backbone network and integrate them into the PAN–FPN structure. The architecture of the MA detection layer is shown in Figure 5. The MA detection layer up-sampled deep-level feature maps with stronger semantic features from the FPN structure and then concatenated them with shallow-level feature maps outputted by the P2 layer of the backbone network, enhancing the semantic expression of the shallow-level features. After feature extraction by the c2f module, the resulting features were passed into the added detection head. Simultaneously, the MA detection layer down-sampled the obtained feature maps using convolution, concatenated them with the deep-level feature maps, and then underwent another feature extraction by the c2f module. This process integrated the feature information extracted from the shallow levels into the PAN structure, enhancing the model’s localization capability at various scales. Based on the modified YOLOv8, small MA features could be obtained, and the accuracy of MA detection could be enhanced. Due to the limited amount of MA data, few annotated samples are available for training and evaluation, and it is a challenge to construct accurate and reliable models. Here, transfer learning was used to modify YOLOv8. Transfer learning [26] is a machine learning technique that leverages the knowledge gained from one task to improve its performance on different but related tasks. Transfer learning is usually used to transfer pre-trained models or features. Using transfer learning, three different datasets were leveraged while training the model, including the original MA images with a size of 768 × 768 pixels and two super-resolution reconstructed images with a size of 1536 × 1536 and 2304 × 2304 pixels. Figure 6 shows the flowchart of transfer learning applied to these three different datasets. Based on the original MA images of 768 × 768 pixels, the detection model was pre-trained, and the learned knowledge was retained. Based on the super-resolution reconstructed images of 1536 × 1536 pixels, the detection model was transferred and fine-tuned. The learned knowledge was updated. Based on the super-resolution reconstructed images of 2304 × 2304 pixels, the detection model was transferred and fine-tuned, and the learned knowledge was updated again. 2.2.3. Loss Function Optimization Based on Wise-IoU The loss function of the official YOLOv8 consists of two components: classification and regression. For classification, binary cross-entropy loss (BCEL) is used as the loss function, while for regression, distribution focal loss (DFL) [27] and CIoU [28] bounding box regression loss (CIoUL) are incorporated. The loss function of YOLOv8 is represented as (6)floss =λ1fBCEL+λ2fDFL+λ3fCIoUL, On the basis of the official YOLOv8 weight parameter settings, the weight parameters λ1, λ2, and λ3 were always set to 0.05, 0.15, and 0.75, respectively. BCEL is defined as (7)fBCEL= weight [ class ](−x[ class ]+log⁡(∑j exp⁡(x[j]))), where class is the number of categories, weight [ class ] denotes the weight for each class, and x is the probability value after sigmoid activation. DFL is an optimization of the focal loss function, which generalizes the discrete results of classification into continuous results through integration, denoted as (8)fDFLSi,Si+1=−yi+1−ylog⁡Si+y−yilog⁡Si+1, where yi,yi+1 represent the values from the left and right sides near the consecutive labels y, satisfying yi