SegImgNet: Segmentation-Guided Dual-Branch Network for Retinal Disease Diagnoses
Xinwei Luo, Songlin Zhao, Yun Zong, Yong Chen, Gui-shuang Ying, Lifang He · 2025
Read SegImgNet: Segmentation-Guided Dual-Branch Network for Retinal Disease Diagnoses by Xinwei Luo, Songlin Zhao, Yun Zong, Yong Chen, Gui-shuang Ying, Lifang He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00267.txt.
Introduction
Xinwei Luo, Songlin Zhao, Yun Zong, Yong Chen, Gui-shuang Ying, and Lifang He guide an eye-disease classifier with anatomy maps, in a 2025 paper. Retinal photos reveal many diseases, but a plain image network may miss the thin vessels and layers where signs live. Outlining those structures first gives the classifier a map to reason with.
SegImgNet builds segmentation maps at several scales, then reads the photo and its maps through two encoders. A special attention block lets the maps steer the reading of the raw image toward telling regions. The two streams merge for the final diagnosis. Anatomy-guided reading mimics how human graders work and can sharpen accuracy on hard cases.
Catalog record
| CREATOR | Xinwei Luo, Songlin Zhao, Yun Zong, Yong Chen, Gui-shuang Ying, Lifang He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SegImgNet: Segmentation-Guided Dual-Branch Network for Retinal Disease Diagnoses Xinwei Luo, Songlin Zhao, Yun Zong, Yong Chen, Gui-shuang Ying, Lifang He Abstract Retinal image plays a crucial role in diagnosing various diseases, as retinal structures provide essential diagnostic information. However, effectively capturing structural features while integrating them with contextual information from retinal images remains a challenge. In this work, we propose segmentation-guided dual-branch network for retinal disease diagnosis using retinal images and their segmentation maps, named SegImgNet. SegImgNet incorporates a segmentation module to generate multi-scale retinal structural feature maps from retinal images. The classification module employs two encoders to independently extract features from segmented images and retinal images for disease classification. To further enhance feature extraction, we introduce the Segmentation-Guided Attention (SGA) block, which leverages feature maps from the segmentation module to refine the classification process. We evaluate SegImgNet on the public AIROGS dataset and the private e-ROP dataset. Experimental results demonstrate that SegImgNet consistently outperforms existing methods, underscoring its effectiveness in retinal disease diagnosis. The code is publicly available at https://github.com/hawk-sudo/SegImgNet . Introduction Retinal imaging, particularly fundus photography, is a non-invasive technique widely used in ophthalmology to capture detailed visualizations of retinal structures. By analyzing these images, clinicians can diagnose not only ocular diseases but also systemic conditions such as hypertension and diabetes ( Li et al. 2023 ; Tan et al. 2024 ) . However, manual interpretation by ophthalmologists is costly, time-consuming, and subject to variability, potentially leading to delays in patient care and inconsistent diagnoses. Therefore, there is an urgent need for automated tools to improve disease detection efficiency through retinal image analysis. Figure 1: Comparison of different approaches: a) Direct classification of raw images using a standard deep learning model. b) Classification based on the segmented image extracted from a segmentation model. c) Combined classification using both raw and segmented images in a shared encoder. d) Classification using both raw images and enhanced segmentation maps in dual encoders ( ours ). Figure 2: Overview of SegImgNet. The input retinal image is first processed by the segmentation module to generate segmentation maps (top). These segmentation maps, along with the raw retinal image, are then fed into separate encoders to extract disease-related features (bottom). The SGA block leverages intermediate feature maps from the segmentation module to generate attention maps, guiding the segmented image encoder’s focus on retinal structural features. Finally, the classifier integrates outputs from both encoders for disease classification. Deep learning has emerged as a promising tool for automating disease detection using retinal images ( Zhou et al. 2023 ; Huang et al. 2023 ; Zhao et al. 2023 ) . These methods typically leverage established computer vision architectures and employ transfer learning to adapt them for various medical applications, as illustrated in Figure 1 (a). For example, RETFound ( Zhou et al. 2023 ) , built on the Vision Transformer (ViT) architecture, is pretrained on large-scale datasets and later fine-tuned on retinal image datasets for disease detection. However, despite their effectiveness, these approaches focus primarily on modeling the overall data distribution of retinal images rather than on highlighting structural features of the retina. Critical diagnostic features are often embedded in the fine-grained structural details of the retina elements that may not significantly impact the overall data distribution but are essential for accurate disease diagnosis. Consequently, compared to natural image classification tasks, retinal disease diagnosis requires models with a stronger ability to capture and interpret key structural features. To address this challenge, a common strategy is to segment key retinal structures from retinal images ( Li and Liu 2022 ; Almeida et al. 2024 ; Wang et al. 2021a ; Sivapriya et al. 2024 ) . By isolating diagnostically significant structures, the model can focus on extracting relevant features, as shown in Figure 1 (b). For example, ( Almeida et al. 2024 ) utilizes a customized image processing technique to segment retinal blood vessels and feed them into DenseNet121 for disease classification, while ( Sivapriya et al. 2024 ) employs ResEAD2Net for blood vessel segmentation and subsequently applies multiple machine learning algorithms to the segmented data for disease prediction. Although these methods improve attention to segmented regions, they ignore valuable information from complementary image areas, potentially limiting overall diagnostic performance. To extract more comprehensive features, recent studies have integrated both segmentation results and retinal images for disease diagnoses ( Alam et al. 2023 ; Joshi, Sharma, and Dutta 2024 ; Xiong et al. 2025 ) . Specifically, some approaches fuse segmented and raw images into a single input and then feed it into an encoder for classification, as shown in Figure 1 (c), while others process segmented and raw images through separate encoders to extract features for disease classification, as shown in Figure 1 (d). For example, ( Alam et al. 2023 ) stacks segmentation maps and retinal images into a single input for GoogleNet, whereas VisionDeep-AI ( Joshi, Sharma, and Dutta 2024 ; Xiong et al. 2025 ) concatenates features extracted from segmented images and retinal images using separate EfficientNet or ResNet50 models. However, these methods lack explicit interactions between segmentation and classification feature spaces. As a result, retinal anatomical features are not fully leveraged to enhance the learned representations in the classification model, limiting the model’s ability to incorporate prior structural information for improved disease diagnosis. In this paper, we propose SegImgNet, a deep learning framework for retinal disease classification that integrates both retinal images and segmentation maps. By leveraging multi-scale structural feature maps obtained from segmentation along with original retinal images, SegImgNet enhances classification performance. The framework consists of two main components: a segmentation module and a classification module. The segmentation module, based on the U-Net ( Ronneberger, Fischer, and Brox 2015 ) architecture, generates retinal structure feature maps. The classification module includes a segmented image encoder, a raw image encoder, a classifier, and Segmentation-Guided Attention (SGA) blocks. The segmented image encoder extracts disease-related local features, while the raw image encoder captures broader global contextual information. Both encoders are built on the ConvNeXt architecture, and the classifier combines their outputs into a unified representation for disease classification. Additionally, the SGA block enhances feature extraction by generating attention maps from structural segmentation, allowing the model to focus on critical retinal details. Extensive experiments on public AIROGS and private e-ROP datasets demonstrate that SegImgNet consistently outperforms existing state-of-the-art methods for retinal disease diagnosis. Our Approach Figure 2 illustrates the architecture of SegImgNet, which consists of two main components: a segmentation module and a classification module. The details of these two modules are introduced below. Segmentation Module The segmentation module f_{seg}(\cdot) employs a U-Net architecture to generate retinal structure feature maps. U-Net utilizes a symmetric encoder-decoder architecture with skip connections, enabling it to capture both low-level spatial details and high-level abstract features. This structure ensures the precise localization of the retinal structures while preserving fine-grained anatomical details. The U-Net’s encoder consists of multiple convolutional layers followed by downsampling operations, progressively reducing spatial resolution while enhancing feature abstraction. This hierarchical representation enables the model to capture retinal structures across multiple scales, which is essential for detecting both fine-grained details and broader pathological patterns. The decoder, on the other hand, reconstructs the segmented image by gradually upsampling the encoded features,