Image super-resolution method using a generative adversarial network incorporating attention and residual density Authors: Qiong Zhang, Yiliu Hang, Hui Zhang, Xi Lu, Jianlin Qiu Published: 2025 ABSTRACT AbstractImage super-resolution reconstruction is currently gaining huge attention because of its very promising applications. However, there are still problems such as loss of detailed features and low signal-to-noise ratio. To address this problem, we propose a generative adversarial network that incorporates attention and residual density. Firstly, the network makes full use of the different layers of features for fusion and prevents gradient decay through the residual dense module. Then through the attention gate structure, the noisy information is suppressed and the signal-to-noise ratio value is improved. The experimental results show that the method proposed is improved in peak signal-to-noise ratio and structural similarity index measure compared to other comparative methods, which proves the advancement of the method proposed.Keywords: Attention gate, Residual density module, Generative adversarial network, Image super-resolutionSubject terms: Computational science, Computer science Introduction Image super-resolution reconstruction technology, which has important application scenarios in the fields of security and surveillance images1, medical images2, remote sensing images3, etc., has gained tremendous attention. As low-resolution images or small-size images are enlarged in the process, it causes the image to become blurred, which affects people’s recognition of image features. To address this problem, Dong C et al.4 applies deep learning to image super-resolution applications for the first time, using a three-layer shallow convolutional neural network to achieve image super-resolution. The low-resolution images are up-scaled to high-resolution image size, and the high-resolution images are obtained through feature extraction, non-linear feature mapping and reconstruction by neural networks. Compared with the traditional methods5, the SRCNN has a great improvement in the reconstructed super-resolution image, which opens the precedent of deep learning method in image super-resolution method. Since the SRCNN method requires the low-resolution image to be up-sampled and interpolated to get the same size as the high-resolution image before feeding it into the network. This means that the convolution operation has to be performed at a higher resolution, which increases the computational complexity. Shi W et al.6 proposed a more efficient network, ESPCN, which extracts features directly on a small low-resolution image and then obtains a high-resolution image by rearranging each pixel of the feature image, which greatly reduces the training time. Kim J et al.7 proposed the VDSR network. This network deepens the overall network with full convolutional layers, and uses residual networks8 to drive the entire network to learn the differences between low-resolution and high-resolution images. Park S W et al.9 proposed the NeXtSRGAN, which integrates a ConvNeXt-based discriminator to overcome these limitations and achieve more realistic and high-quality super-resolution images. But with CNN-based super-resolution methods10,11, the recovered image usually loses high-frequency information, which reduces the visual perception. Ledig C et al.12 proposed SRGAN method, which for the first time used generative adversarial network (GAN13) to solve the super-resolution problem. The SRGAN method use perceptual loss and adversarial loss to enhance the realism of the recovered images and achieve better visual results. Wang X et al.14 proposed an enhanced adversarial generative super-resolution reconstruction method, which improves the multi-layer feature fusion capability by incorporating residual dense network (RDN15) into the generator structure of GAN, and improves the feature generation effect of the generator from low-resolution to super-resolution. Meng F et al.16 proposed a layered dense sampling and chain training to generate adversarial network, which enhances the fusion of internal features and improves the clarity of super-resolution images through dense sampling of features in each layer, but also brings noise interference. Based on the shortcomings of the above methods, we propose image super-resolution method using a generative adversarial network incorporating attention and residual density. Our main contribution points: Propose a novel and effective image super-resolution method that overcomes the shortcomings of existing methods and improves image super-resolution quality. Propose a novel and effective image super-resolution method that overcomes the shortcomings of existing methods and improves image super-resolution quality. Combine residual dense networks with generative adversarial network to fully use all the hierarchical features of the original image, while prevent the gradient decay that occurs when the number of network layers increases. Combine residual dense networks with generative adversarial network to fully use all the hierarchical features of the original image, while prevent the gradient decay that occurs when the number of network layers increases. Incorporate the attention gate network structure into the generative adversarial network to suppress the noise information of the input image and improve the signal-to-noise ratio value of the image. Incorporate the attention gate network structure into the generative adversarial network to suppress the noise information of the input image and improve the signal-to-noise ratio value of the image. Related work In the subsection, we give a brief introduction to generative adversarial network and attention gate. Image super-resolution method generates images that are structurally similar to real images, while achieving high fidelity and low loss of detail features. For high fidelity, we use a generative adversarial network structure to generate high-fidelity images. For low feature loss, we use residual dense connections and attention gates to enhance the fusion of features and prevent feature loss. Generative adversarial network Generative Adversarial Network (GAN12) is a deep generative model based on adversarial learning17 which consists of a generative network and a discriminative network. The generative network is responsible for generating the fake super resolution image based on the input noisy data; the discriminative network discriminates the authenticity based on the real high-resolution and generated fake super-resolution image. The generative network evades the judgement of the discriminator by continuously optimizing the generated image, and the discriminator makes its judgement more accurate by optimizing the network structure, thus forming an adversarial relationship between the two. The GAN structure is shown in the Fig. 1. The structure diagram of GAN, including a generator and a discriminator. The generator generates super-resolution images, and the discriminator compares the generated images with the real images to identify differences. Attention gate Attention mechanism18 as a feature weight allocation mechanism can effectively highlight important feature information and suppress background feature information, and its core idea is to assign different weights according to the importance of features. For the attention gate(AG19), shown in Fig. 2, based on input features G and P, after convolution by 1 × 1 × 1, the corresponding position features are added, then negative features are suppressed by ReLU function, and the features are normalized by Sigmoid function. The processed results are multiplied with the initial feature P by channel to obtain the new feature Pl after the assigned weight. The structure of this attention gate is shown in Fig. 2 and the flow is shown in Method 1. The structure diagram of AG, including ReLU, Sigmoid, and resampling operations. Method 1: Attention Gate. Methodologies Multi-level feature fusion adopts the method of residual in residual density network. Because residual connections can prevent gradient disappearance when the number of network layers increases, and dense connections can better concatenate features of different scales from different layers. Therefore, the way of residual in residual density network contains both the shallow features of the low layer and the high-level features of the high layer, so the features are more abundant. At the same time, an attention gate structure is used to highlight important features and suppress noise. Residual dense network Residual Density Network (RDN14) is a deep learning network based on residual connectivity for image super-resolution reconstruction. The core of RDN is dense residual, which uses the structure of dense block to transfer information between each block and improve the utilization of information. To avoid gradient vanishing, the RDN uses the strategy of residual connectivity, which allows the inputs and outputs of each dense block to be added to each other to ensure the flow of information. The structure of the Residual Network (Fig. 3a), Residual Density Network (Fig. 3b) and Residual in Residual Density Network is shown in Fig. 3c. (a) Residual Network in MDSR20. (b) Residual Density Network in SRDenseNet21. (c) Residual in Residual Density block, β is the residual scaling parameter and β = 0.214 in the paper. According to the principle of residual networks, when the input variable xi−1 is in the mapping process, the gradient decay problem occurs. Through the internal skip connection function, the input variable xi−1 and the identity mapped result F(xi−1) are added to enhance the output result. With the residual network, as the number of network layers increases, overfitting will not occur. Residual network is shown in Eq. 1. Density block is the fusion of local feature information through dense connection. The dense connection module strengthens the fusion between internal features. And, it uses the residual network to strengthen the feature fusion between layers. Therefore, the residual density network not only retains rich internal features, but also strengthens the feature fusion between layers. The residual dense connection is shown in Eq. 2. In the network structure, only focus on the internal feature fusion of modules, lack of connections between modules, and leading to the problem of gradient decay between modules. Therefore, we build the residual-dense network in the residual, which is a kind of global residual connection. Through the residual in residual-dense network, the feature isolation between layers can be solved. Residual in Residual-dense network is shown in Eq. 3. β is the residual scaling parameter and β = 0.214 in the paper. When , the variable xi−1 is connected to the mapping result of the variable xi−1, and when i = n, the initial variable x0 is skip-connected to the output data of the residual dense module. Attention gate in generator In order to suppress the background noise and improve the peak signal-to-noise ratio of the generated image, we propose to process the input information by means of attention gate. After the input feature P and process feature G are convolved by 1 × 1 × 1, the corresponding position features are added, then the negative features are suppressed by ReLU function which the expression of the ReLU function is f(x) = max(0, x), and then the features are normalized by Sigmoid function. The processed output results are multiplied with the input feature P by channel to obtain the new features after the assigned weights . By adding attention gates to the generator, different weights can be assigned to the input feature maps, thereby increasing the contribution of important features in the generator according to subsection "Attention gate", the principle of attention gates. The process of operating on the input features P and G can be represented by Eqs.4 and 5, respectively: where is the rectified linear unit (ReLU) function and is the sigmoid activation function. Parameter:,,,Offset:,。 Generator network architecture Based on GAN, the proposed method generates new super-resolution output images through the adversarial training of generator and discriminator through low-resolution and high-resolution images. Its calculation is represented by Eq. 6: where D is the discriminator, G is the generator, IHR is the high-resolution image, ILR is the low-resolution image, Ptrain(IHR) is the probability distribution of training set IHR, PG(ILR) is the probability distribution of ILR after generator processing. The generator network structure is shown in the Fig. 4. Method 2:Generator Network. Generator network architecture, where RRDN is the residual-dense in the residual, and AG is the attention gate. According to the input low-resolution image with noisy information, it is processed by the convolution module, and then processed by the N-layer RRDN module, and the residual intensive feature matrix is output. Then the feature size is adjusted through the convolution layer, and batch normalization. The process matrix G and the initial matrix P are input into the Attention Gate. The method flow is shown in Method 2. Experiment Experimental environment The experiment was run on the Linux server and the configuration environment is Ubuntu20.04 Python 3.10, Stable Diffusion v1-5, CUDA 11.7, cuDNN 8, Pytorch 2, JupyterLab. Computing power is NVIDIA Tesla T4 GPU—16 GB +| 8 + TFlops SP CPU—8 cores| Memory—32 GB. The experimental details include batch size = 16, learning rate = 1 × e−4, and the optimization function is Adam. The loss function adopts a hybrid loss function of perceptual loss, content loss and adversarial loss in SRGAN12. Datasets The training set adopted in this experiment is DIV2K_train_HR in DIV2K22, the verification set is DIV2K_valid_HR, and the test set is Set523, Set1424, T9125, Urban10026. The training set DIV2K_train_HR contains 800 high-resolution images with 2 K resolution and the verification set DIV2K_valid_HR contains 100 high-resolution images, each with rich texture information. At the same time, to evaluate the generalization ability of the model, we used the images from the test sets Set5, Set14, T91 and Urban100 data sets for a single image super-resolution. Evaluation indicators There are two commonly used evaluation indicators for super resolution, one is peak signal-to-noise ratio (PSNR), and the other is structural similarity index measure (SSIM27). These two evaluation indicators are the most basic ones used to measure the quality of compressed reconstructed images in super-resolution. The formula derivation process is represented by Eqs.7,8,9. MSE is the mean square error, where I is the real high-resolution image, K is the generated high-resolution image, and the image size is H × W. PSNR is defined by the formula 8: where, MAXI is the largest pixel value in image I. Images into YCbCr format, and then calculate the PSNR of Y component. The larger the value of PSNR, the better the effect. Mean structural similarity index measure (SSIM) assumes that the human visual system is highly adapt..ed to extract image structures, and measures the structural similarity between images based on the comparisons of luminance, contrast, and structures. As shown in formula 9 where x and y denote two images, and are the mean and variance, is the covariance between x and y, and c1, c2 are constant relaxation terms. c1 = (k1 × L)2, k1 is a constant and default value is 0.01. c2 = (k2 × L)2, k2 is a constant and default value is 0.03. L is the range of pixel values and value is . Qualitative results Experiment is trained according to the DIV2K_train_HR dataset. The model is evaluated using any single image in Set5, T91 and Urban100. Through evaluation indicators of PSNR and SSIM, it can be seen that our proposed method is superior to other comparison methods on the whole. In terms of evaluation indicators, compared with the suboptimal value, our method improved PSNR by 0.92%, 1.48%, 2.18% and 0.44%, and SSIM by 0.86%, −8.69%, 2.19% and 2.6%. Therefore, the superiority of our proposed method can be proved. The results of this experiment are shown by Fig. 5. [4 × upscaling]. Super-resolution effects in different methods. Through analysis of Fig. 5, we can see that the SRARDGAN method is superior to the comparison method in visual. The images generated by SRARDGAN method are more high-fidelity in terms of detail features and do not show the phenomenon of pixel blurring. At the same time, the feature contours are also clearer, and the contrast of boundary features is more obvious. Ablation study To validate the performance of the SRARDGAN method, we validate it with the dataset DIV2K_valid_HR. We compared the SRARDGAN method with the SRGAN method, and the curve value changes of average PSNR and SSIM in the iterative process can be seen from Fig. 6a, b. SRARDGAN method is superior to SRGAN in average PSNR and SSIM. At the same time, the jitter amplitude of SRARDGAN method in average PSNR is smaller, and the model is more stable. Therefore, it can be proved that the performance of SRARDGAN is better. Comparison of SRARDGAN and SRGAN in an iterative process[4 × upscaling]. To further verify the effectiveness of the SRARDGAN method, we use the experiment of ablation research to compare the improvement effect of different aspects of the proposed method. As shown in Fig. 7, 1st is the original image, 2nd is the super-resolution image generated by GAN, 3rd is the generated super-resolution image of residual-dense block and GAN structure, 4th is the generated super-resolution image of attention gate and GAN structure, 5th is the effect of the SRARDGAN method. Through several sets of experimental results, in both PSNR and SSIM evaluation metrics. The effect in ablation study[4 × upscaling]. We can analyze that 3rd has a higher value on SSIM than 2nd, dense networks are those that densely connect features from different layers and integrate low-level and high-level features. Residual networks concatenate the features between different modules through skip connections. So the RDN module enhances feature fusion across different layers, the generated image detail features are closer to those of real images, thus proving the effectiveness of RDN. The 4th shows a significant improvement in PSNR values over the 2nd, When the image is input into the attention gate structure, since the noise features belong to the background, they are suppressed features when fused with higher-level features. So the AG suppresses image noise, the generated images have a higher peak signal-to-noise ratio, thus proving the validity of the AG. 5th improves more significantly on PSNR and SSIM than 2nd,3rd,4th and thus it can be proved that the effectiveness of the method proposed. Discussion Through the research in this paper, it can be fully proved that the method proposed can effectively improve the resolution of the image in the reconstruction of the super-resolution. However, this method also has some defects. When it is integrated into the residual-dense network and the network parameters are increased, the computing resource consumption is also increased. Such as, the FLOP is 734,219 for SRGAN and 867,149 for SRARDGAN. Meanwhile, we adopt the loss function in SRGAN, which uses perceptual loss and adversarial loss to enhance the realism of the recovered images, but this loss function has the potential to bring about gradient explosion and make the training unstable. Therefore, in the future, we will focus on how to reduce the network parameters while guaranteeing the image effect, and at the same time, we will study the new loss function and train the model to be more stable. Conclusions We propose image super-resolution method using a generative adversarial network incorporating attention and residual density. Increase feature fusion in multi-layer networks by incorporating residual-dense modules to prevent gradient decay. Attention gates are also incorporated into the GAN structure to suppress low-resolution image background noise. This method can effectively improve the image super-resolution and provide the basis for the later research of image segmentation28, target detection29, attitude tracking30 and so on. Acknowledgements This research was funded by the program of the Natural Science Research of Jiangsu Province Higher Education Institutions in 2023 under Grant No.23KJD520011, the General Program of Nantong Science and Technology Bureau in 2024 under Grant No. MSZ2024110, the Program of the Nantong Institute of Technology Doctoral Research Start-up Fund No.2025XKB30. We would like to thank everyone who has contributed to this article. We also would like to thank the anonymous reviewers and editors for their helpful suggestions and comments. Abbreviations Convolutional neural network Generative adversarial network Peak signal-to-noise ratio Structural similarity index measure Super resolution Low resolution High resolution Residual dense network Attention gate Residual in residual dense network Author contributions Conceptualization: Qiong Zhang, Jianlin Qiu Data curation: Qiong Zhang, Hui Zhang Software: Qiong Zhang Supervision: Xi Lu Validation: Yiliu Hang, Jianlin Qiu Visualization: Qiong Zhang Writing-original: Qiong Zhang Writing-review: Qiong Zhang, Xi Lu Methodology: Qiong Zhang, Xi Lu Project administration: Qiong Zhang Investigation: Yiliu Hang Formal analysis: Xi Lu Funding acquisition: Qiong Zhang, Xi Lu Resources: Qiong Zhang, Xi Lu. Funding Major Basic Research Project of the Natural Science Foundation of the Jiangsu Higher Education Institutions,23KJD520011,the Program of the Nantong Institute of Technology Doctoral Research Start-up Fund,2025XKB30,Nantong Municipal Science and Technology Bureau,MSZ2024110 Data availability The data that support the findings of this study are openly available at the following URL: DIV2K: https://data.vision.ee.ethz.ch/cvl/DIV2K/; Common Benchmark datasets for testing image super resolution performance include Set5, Set14, Urban100, T91: https://github.com/ChaofWang/Awesome-Super-Resolution/blob/master/dataset.md. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The data that support the findings of this study are openly available at the following URL: DIV2K: https://data.vision.ee.ethz.ch/cvl/DIV2K/; Common Benchmark datasets for testing image super resolution performance include Set5, Set14, Urban100, T91: https://github.com/ChaofWang/Awesome-Super-Resolution/blob/master/dataset.md.