A novel deep unsupervised approach for super-resolution of remote sensing hyperspectral image using gompertz-function convergence war accelerometric-optimization generative adversarial network (GF-CWAO-GAN) Authors: K. Deepthi, Aditya K. Shastry, E. Naresh Published: 2024 ABSTRACT Hyperspectral remote sensing images obtained from cameras are characterized by high-dimensions and low quality, which makes them unfavorable for various analytics purposes. This is due to the presence of visible and invisible frequencies of the reflected light making it poorly reveal the spectral signatures of the image. Visual communication advancement has paved the need for Image Super-Resolution (SR) which recovers high-resolution images from low-resolution images. Several works were carried out earlier on image SR using variants of supervised and unsupervised models that still lack accuracy. In this paper, we propose an unsupervised learning model titled Gompertz Function–based Convergence War Accelerometric Optimization–GAN framework for generating of High-Resolution (HR) images. The framework comprises a pre-processing stage, where the incoming Low-Resolution (LR) image is preprocessed for noise removal by applying Shannon-Gaussian Filter (S-GF). Following is the Gradient Domain Approach based Tone-Mapping (TM). Skew correction is done to remove distortion and maintain original resolution that may change during TM stage. The next stage comprises the boundary and edge enhancement of the resulting preprocessed image generated by the method of Inverse Gradient Mapping (IGM) followed by patch extraction to extract minute low-frequency information from the resulting boundary and edge-enhanced image. The contrast of the enhanced patches is improved by removing blurriness effect. The preprocessed image patches are then fed into the Gompertz Function-based Convergence War Accelerometric Optimization – GAN for feature mapping on the trained SR Image features that are clustered using Krzanowski and Li- Kantorovich Metric-K-Means clustering Algorithm (KL-KM-KMA) for effective generation of SR image. The developed model is validated for both qualitative and quantitative measurements. Comparisons are made with several other state-of -the-art methods for accuracy of 98.05%, precision of 97.98%, inception score of 8.71, Fréchet Inception Distance of 36.4 with reduced clustering and training time proving the efficiency of the proposed model. Introduction Remote sensing images attract the attention of various fields like environment monitoring, remote sensing and computer vision. Hence, in the process of exploring the Earth, obtaining HR image of Earth is of great significance for scientific research. However, due to the variations in temperature, atmospheric pressure and other environmental conditions on Earth, the images obtained from hyperspectral cameras are unstable and most of the times blurry, obscure thereby reducing the usability of data. Image SR aims to facilitate better visualization of remote sensing images thereby addressing the challenges associated with a wide range of real-world applications. Deep learning-based image SR of remote sensing images has gained importance in recent years. There are several conventional SR methods that are based on Interpolation often require extensive computational resources and enormous training time, hindering their practical implementation under resource-constrained environment. From the literature, it is prevalent that most of the earlier work proposed on image SR was supervised method based that demands huge paired LR-HR images for training and posed a huge challenge in accumulating huge data set. Several unsupervised approaches were also proposed that provide relaxation with respect to data set size and unavailability of paired LR-HR images. Recent advances in Convolutional Neural Network (CNN) addressing SR have progressed in image SR, on the other hand due to ill-posed nature of SR images generated, the existing Convolutional Neural Network oriented techniques often loose texture details thereby causing excessive smoothing of image1,2. To generate more visually realistic SR images, Unsupervised Generative Adversarial Networks (GANs), their variants and hybrid GANs can be used and have affirmed to accomplish superior super-resolved satellite images3. Although GAN-based approaches achieve SR of satellite photos, there are several challenges that are still prevalent. Firstly, they often struggle with generalization across diverse datasets leading to performance inconsistencies. Secondly, training instability leads to generators producing limited outputs affecting the quality of SR images. Existing works also failed to preserve the low frequency information during image SR generation step. Third, there is a gap in exploring hybrid GAN architecture specifically designed to improve SR performance. Lastly, Unsupervised GANs are computationally intensive, leading to scalability issues in real-time applications suggesting a need for more efficient hybrid architectures, advanced algorithms and techniques. To solve the above-mentioned problems, an efficient image super resolution model using GF-CWAO-GAN based image SR technique is proposed to perform effective matching of feature points resulting in better SR image generation. Patch extraction and Inverse Gradient Matching are introduced for better preservation of low frequency information and image characteristics respectively. Specifically, the model also stands better in effective preprocessing of input image by the introduction of Tone Mapping and Skew Correction task for preserving both high and low frequency information. Extensive experimental results justify that the proposed GF-CWAO-GAN obtains acceptable performance than existing traditional, CNN and GAN-based methods qualitatively and quantitatively. The contribution of the current article can be made as follows: A framework named GF-CWAO-GAN is proposed that significantly improves the grade of SR reformation of Hyperspectral Remote-sensing image.An improved strategy for Pre-processing Step for extracting determinant features/pixels inclusive of Boundary and Edge Enhancement, Patch Extraction and Patch Enhancement is proposed to effectively enhance the features extraction process for SR reconstruction. Krzanowski and Li-Kantorovich Metric-K-Means clustering Algorithm is proposed during the training phase followed by further effective generation of SR image with GF function in the Generator Model and initial matching and propagation matching in the Discriminator Model.The Fréchet Inception Distance was introduced to measure the loss between real and generated images to improve the spatial-spectral characteristics in reconstructed SR hyperspectral remote-sensing image. The rest of the work is organized as follows. In Sect. 2, the earlier work and issues related to proposed work is presented. Section 3 provides the framework/network architecture of the proposed model. Section 4 provides algorithms, experiments and theoretical analysis of model parameters and loss. In Sect. 5, experimental results along with their quantitative and qualitative analysis are presented. Finally, the article is concluded with a conclusion. A framework named GF-CWAO-GAN is proposed that significantly improves the grade of SR reformation of Hyperspectral Remote-sensing image. A framework named GF-CWAO-GAN is proposed that significantly improves the grade of SR reformation of Hyperspectral Remote-sensing image. An improved strategy for Pre-processing Step for extracting determinant features/pixels inclusive of Boundary and Edge Enhancement, Patch Extraction and Patch Enhancement is proposed to effectively enhance the features extraction process for SR reconstruction. Krzanowski and Li-Kantorovich Metric-K-Means clustering Algorithm is proposed during the training phase followed by further effective generation of SR image with GF function in the Generator Model and initial matching and propagation matching in the Discriminator Model. An improved strategy for Pre-processing Step for extracting determinant features/pixels inclusive of Boundary and Edge Enhancement, Patch Extraction and Patch Enhancement is proposed to effectively enhance the features extraction process for SR reconstruction. Krzanowski and Li-Kantorovich Metric-K-Means clustering Algorithm is proposed during the training phase followed by further effective generation of SR image with GF function in the Generator Model and initial matching and propagation matching in the Discriminator Model. The Fréchet Inception Distance was introduced to measure the loss between real and generated images to improve the spatial-spectral characteristics in reconstructed SR hyperspectral remote-sensing image. The Fréchet Inception Distance was introduced to measure the loss between real and generated images to improve the spatial-spectral characteristics in reconstructed SR hyperspectral remote-sensing image. The rest of the work is organized as follows. In Sect. 2, the earlier work and issues related to proposed work is presented. Section 3 provides the framework/network architecture of the proposed model. Section 4 provides algorithms, experiments and theoretical analysis of model parameters and loss. In Sect. 5, experimental results along with their quantitative and qualitative analysis are presented. Finally, the article is concluded with a conclusion. Related work Image Super-Resolution (Image SR) refers to the process of obtaining HR image from LR image retaining high quality, sharp edges and wanting for more pixels with very few artifacts. according to previous literature, SR methods are classified as Single-Image SR(SISR) and Multi-Image SR(MISR)4. SR can do either using supervised or unsupervised methods using CNNs5,6and GANs. However, GANs provide powerful framework and outperform CNNs in unsupervised SR tasks by leveraging adversarial learning and focusing on perceptual image quality in generating realistic images. The SR methods are applied to the various image types like medical images7, remote sensing images6,8–10, hyperspectral images11, realistic images4and other types of images. The literature overview on SR is included in12,13. This section conducts brief literature review on GAN-based SISR models for satellite image SR and their related issues recapitulated as below. Ever since Ian Good Fellow et al., proposed GAN in14, they are rigorously studied by several researchers as detailed below. Several attempts on employing GANs for unsupervised learning in recent years. Adoption of GAN-based Deep Learning SR models for remote sensing image SR has gained increasing popularity in recent years, each one offering some improvement in terms of accuracy and resolution produced. Recent advancements in utilizing GANs to upgrade the SR of Mars image, include novel two step framework incorporating new degradation framework. It is designed to estimate blur kernels and GAN trained to generate noise distribution over Mars2K dataset15. According to8, Multiattention GAN(MA-GAN) comprises of three blocks to automatically learn and perform scale adjustments for better image representation. The channel attention (CA) and pixel attention (PA) upscale LR images to HR images taking adversarial loss and pixel loss as guiding parameters. Cascade GAN(CGAN) addressed the issue of uncontrollable performance of discriminator and unstable training by introduction of content fidelity to avoid vanishing gradient and scene constraint to achieve high quality SR image3. A GAN-based approach has demonstrated superior performance for hyperspectral image SR is proposed with residual learning incorporated to achieve spectral fidelity utilizing gradient features as auxiliary information to carry out counter training with discriminator11. Shaolei Zhang et al., 2021 had put forward an unsupervised GAN approach to hyperspectral image SR by group convolution and attention mechanism via degradation learning on the public dataset CAVE and have justified to outperform the SR ability over other models16. According to9, Yingfei Xiong et al., proposed improved SRGAN (ISRGAN) with stable training and enhanced generalization abilities across locations and two sensors Landsat 8 OLI and Chinese GF1 with improved PSNR (35.816) and SSIM (0.988) parameters. Also, Radford et al., proposed Deep Convolutional GAN where GANs performed feature extraction to capture semantic content from the images enabled with vector arithmetic17. The unsupervised objective of image SR is realized focusing on working principle of GAN. In principle, the GAN architecture design comprises of generator plus discriminator are trained together/simultaneous with competing aims. While the generator network attempts to generate images that are indistinguishable from natural images, the discriminator network is trained to discriminate between original image and synthesized image. Wang et al., in18 proposed ESRGAN, a groundbreaking approach in SISR prioritizing perceptual quality over PSNR (traditional metric). ESRGAN is an enhanced version of SRGAN with Residual-in-Residual Dense Blocks (RRDBs) supporting improved feature extraction, removes Batch Normalization layers to avoid artifacts. More realistic image generation was observed using relativistic discriminator inception. Several clustering algorithms were seen in earlier literature. James c et al., proposed Fuzzy C-means Clustering (FCM) that generates fuzzy partitions and prototypes useful for suggesting substructures in unexplored data using least-square objective function19. In20, researchers achieved contrast improvements using Mean Shift (MS) clustering resulting in greater accuracy. Tian Zhang et al., proposed data clustering technique titled BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) to generate best quality clusters via incremental and dynamic cluster formation of multidimensional data points21. Several researchers have used K-Means Clustering algorithm and their variants for identifying clusters. From the thorough study of the earlier works, it is prevalent that they faced the issues and challenges in terms of loss of naturalness in generated images, convergence, extensive need for computational resources while training, lack of generalization and so on. In summary, all the unsupervised GANs discussed above are computationally intensive and resource constrained. The current work proposed focusses on resource efficient GF-CWAO-GAN architecture and leverages Google Colab’s TPUs for accessibility. The proposed GF-CWAO-GAN framework/proposed network architecture The aim of remote sensing hyperspectral image super-resolution is to obtain high resolution image from low resolution image to improve the visual communication in various fields like satellite and astronomical imaging, military surveillance, defense and remote sensing. Hence, this paper presents an efficient novel deep learning-based GF-CWAO-Generative Adversarial Network (GF-CWAO-GAN) based framework to generate super-resolution images. Basic network architecture The original ESRGAN Architecture is as shown in the Fig. 1 below which forms the basis for the proposed hybrid network architecture model. Fig. 1Basic ESRGAN Architecture. Basic ESRGAN Architecture. The overall configuration is retained consisting of Generator and Discriminator networks as proposed in18. Proposed architecture of GF-CWAO-GAN Super-Resolution images are always of interest as they conserve additional and crucial details playing a key role in various fields like medical imaging, astrophotography, surveillance and others. Various deep learning-based frameworks are proposed by several researchers and Generative Adversarial Networks (GAN) has played a prominent role in Image Super-Resolution. Inspirited by the remarkable performance of GANs under unsupervised learning, herewith we propose Gompertz Function – Convergence War Accelerometric Optimization GAN (GF-CWAO-GAN) with densely-connected CNN generator network and modified Discriminator framework. Earlier GAN as proposed by Z Wang et al., 2020 used sigmoid activation function at discriminator resulted in gradient diffusion problem and incorporation of Wasserstein distance metric for measuring loss between generated and original image increased the distribution loss. To overcome this drawback, in the proposed model Gompertz Function and Fréchet Inception distance are considered to reduce the gradient diffusion problem and distribution loss respectively. Hence the name Gompertz Function GAN (GF GAN). The framework for the proposed GAN is depicted in Fig. 2 below: Fig. 2Proposed GF-CWAO-GAN Model. Proposed GF-CWAO-GAN Model. Proposed architecture modifications for discriminator in GF-CWAO-GAN The proposed model implements several modifications in Discriminator to bring novelty as shown in the Fig. 3 below: Fig. 3Proposed Architecture modifications for Discriminator in GF-CWAO GAN. Proposed Architecture modifications for Discriminator in GF-CWAO GAN. The model description of Discriminator is detailed below: Introduction of PatchGAN as a discriminator network consisting of comprising of 8 blocks of Conv2D layer, followed by LeakyReLU layer with each block having Batch Norm layer except for the initial layer. This is done to concentrate on local image characteristics rather than overall image details, by reading images in patches. Pre-residual block: This block is introduced before the main RDB network in the generator network comprising of convolutional layer followed by activation function, LeakyReLu for preliminary feature extraction, processing the input LR image and generating initial feature maps. Enhanced Residual Dense Blocks: Within each RDB, Leaky ReLu activation layer is added after each convolutional layer as shown in the figure above. This activation function permits for small positive gradient even for non-positive inputs, avoiding the “dying ReLu” problem22. Following this is the concatenation layer for feature fusion at each step23,24. Lambda and Addition layer: A mechanism is introduced to allow the network to learn the difference between the input and the processed layer. The Lambda layer followed by addition layers does this mechanism by performing identity operations and creating skip connections respectively. Post-residual Block: This block consisting of Conv2D layer followed by LeakyReLu layer is introduced following the final RDB block in the generator network that mirrors the pre-residual block to help improve final feature representation before feeding it to up-sampling block. Sub-pixel Convolution for Up-Sampling: Instead of nearest neighbour interpolation for up-sampling, the proposed model implements sub-pixel convolution that allows generating HR image outputs while reducing artifacts, leading to sharper and more visually appealing images. Reduced RDBs and RDB Blocks: This is one of the major modifications proposed in the current work in which the number of RDBs and RRDB blocks are reduced compared to original ESRGAN model. The number of Residual Dense Blocks (RDBs) and Residual in Residual Dense Blocks (RRDBs) are reduced from 16 RDBs and 23 RRDBs to 1 RRDB and 4 RDBs respectively. Despite of its smaller size, it showed superior performance over the existing GANs. Super-resolution image generation flow diagram Figure 4 below shows the sequence of phases involved in SR image generation. In the first phase, the data is pre-processed and prepared by removing noise while preserving the original pixel intensity and reshaped to serve the purpose of research work. Following it by Boundary and Edge Enhancement, Patch Extraction, Patch Enhancement to make the image apt for Feature Extraction. The dimensions of the images are resized to [128, 128] for the testing purpose. In the second phase, the super-resolution image generation takes place with the GF-CWAO-GAN that suits the image requirements for various application fields like land cover land use monitoring, structural mapping, environmental geology, weather forecasting, mineral exploration, land cover classification etc., In the final phase, qualitative and quantitative measurements are performed to prove the effectiveness, feasibility and standard of the proposed framework. Fig. 4Flow Diagram of Super-Resolution Image generation process. Flow Diagram of Super-Resolution Image generation process. Pre-processing phase To get started with, the input LR image is pre-processed for the removal of unwanted data infused in it. The purpose of preprocessing is to generally preserve the determining/important details in each image pixel while successfully eliminating the noise component present in the image. The mathematical model of input low-resolution image is given in Eq. (1).1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{i}\left(x,y\right)={\sum\:}_{i=1c}^{n}{P}_{i}\left(x,y\right)+{{\aleph\:}}_{0}$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{i}\left(x,y\right)$$\end{document} is the input low-resolution image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(x,y\right)\:$$\end{document}being the pixel value in the image and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\aleph\:}_{0}$$\end{document} is the noise component present in the image. Several pre-processing stages are involved are outlined as given below: Shannon – Gaussian Filter (S-GF) based noise removal Initially, the low-resolution image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{i}\left(x,y\right)$$\end{document} goes through the noise removal stage to eliminate the noise found in the LR image. Noise present in the image impairs the image’s quality by manipulating individual pixel value. As a result, S-GF is applied for noise removal to enhance the quality of the image. Gaussian filter is a linear filter used in image processing either to bring blurriness in image or to remove noise from the image. In our proposed work, such a filter is used as it has effective noise filtering capability and is faster due to multiplying and addition over the median filters which are based on sorting. The degree of randomness in the image without information loss is measured by the introduction of Shannon Entropy. The integration of Shannon Entropy into Gaussian Filter is the emergence of the term Shannon – Gaussian Filter(S-GF) and given in the mathematical Eq. (2) below:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{S-GF}\left(x,y\right)=\frac{\pi\:}{\sqrt{2\mu\:}}{log}\left({\left({P}_{i}\left(x,y\right)\right)}^{2}/2{\mu\:}^{2}\right)$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{S-GF}\left(x,y\right)$$\end{document} denotes Shannon – Gaussian function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mu\:$$\end{document} for Gaussian Filter variance The outcome of this step is the noise removed image denoted as PNR(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:x,y).$$\end{document} Tone mapping: gradient domain based approach Noise removed image is at high dynamic range and to maintain and preserve original pixel intensity, gradient domain – based tone mapping is performed to output low dynamic range satellite image. The gradient domain tone mapping method operates on the assumption that the human visual system is more receptive to local intensity variations rather than absolute brightness values. These variations are accurately represented using gradients. Firstly, the choice of image gradient (∂ldr) is done. The gradient value is computed by taking the finite difference between the pixels in the image and is represented as shown below:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\partial\:}_{ldr}=\text{Difference}(P_{NR}(x,y))$$\end{document} Where Difference() is the finite difference function. Then obtained gradient outcome value is used to generate low dynamic range image (LDR), PLDR(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:x,y)$$\end{document} by resolving the following Poisson’s model below:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{LDR}\left(x,y\right)={\partial\:}_{ldr}\left({P}_{NR}\left(x,y\right)\right)$$\end{document} Skew correction Skew Correction is the next part after the tone mapped image is obtained. Skewing in image is something that is not exact. It indicates the amount of distortion in the image. Hence the skew correction is carried out on tone mapped image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{LDR}\left(x,y\right)$$\end{document} to retain distortion while removing distortion in the image. The skew correction is mathematically modelled as Eq. (5) given below:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{sc}\left(x,y\right)=\frac{3\left(\sigma\:-\gamma\:\right)}{y}\left({P}_{LDR}\left(x,y\right)\right)$$\end{document} Where, σ represents the mean value of tone mapped image, γ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:y$$\end{document} projects to the median and standard deviation of tone mapped image. Edge and boundary detection – inverse gradient mapping (IGM) based An edge in an image is the connection of a set of pixels that forms a boundary between any two disjoint boundaries. After pre-processing stage, the preprocessed image is enhanced to help better generate super-resolution image. Gradient Mapping (GM) is more suitable for boundary and edge enhancement with a better enhancement rate due to implementation of convolution operation. IGM is developed using four components like Superficial Feature Extraction (SFE), Local Feature Extraction (LFE), Global Feature Extraction (GFE) and Upsampling module. The following subsections highlight the above components: Superficial feature extraction (SFE) component This component aims at the extraction of superficial features (SFf) from the pre-processed image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{sc}\left(x,y\right)$$\end{document}. It is given by the equation:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{SF}_{f}=R\left(invconv\left(invconv\left({P}_{sc}\left(x,y\right)\right)\right)\right)$$\end{document} Where R refers to parametric function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:invconv\left(\right)$$\end{document} is the inverse convolution operation performed for edge and boundary enhancement. Local feature extraction (LFE) module The superficial features extracted from the SFE component is input to LFE module where local/neighboring features (Nf) are extracted through multiple Residual Dense Blocks (RDB) plugged in it and is represented as,7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{N}_{f}=rdb\left({SF}_{f}\right)$$\end{document} Where rdb() denotes Residual Dense Block. The superficial and shallow features extracted are then super-imposed to produce resultant merged parameters/features as indicated by8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{simp}_{f}=R\left(conv\left(concat\left({SF}_{f},{N}_{f}\right)\right)\right)$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{simp}_{f}$$\end{document} is a super-imposed feature, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:concat\left(\right)$$\end{document} is concatenation operation. Global feature extraction (GFE) module Global features denote the features extracted from the overall image whereas local features are the features extracted from neighbouring pixels of the given sub image. The resultant coalesced features are then merged with the features extracted from pre-processed image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{sc}\left(x,y\right)\:$$\end{document}to obtain global feature, Gf, represented as,9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{G}_{f}={f}^{-1}+{simp}_{f}$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}^{-1}$$\end{document} is input image feature. Hence, based on the extracted features, the size and scale of the unit capable of distinguishing objects in the image is maintained by the process of up sampling. Based on the resultant slope/gradient characteristics, the boundary and edge of image is characterized, and Boundary and Edge enhanced image is denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{BE}\left(x,y\right)$$\end{document}. Patch extraction Followed by the boundary and edge enhancement step, the next step is the patch extraction, a vital pre-processing step in image processing is done from the boundary and edge enhanced image. Patch in image is a group of pixels of interest in an image. Patch extraction portrays an image as a set of several sub-images (Patches) of interest. Patch extraction is done to recover the low-frequency information as defined in mathematical equation below:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:PE\left({P}_{BE}\left(x,y\right)\right)=PE{\left({P}_{BE}^{q}\left(x,y\right)|q\right)}^{\raisebox{1ex}{$1$}\!\left/\:\!\raisebox{-1ex}{$s$}\right.}$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{BE}^{q}$$\end{document} is a patch of size, s, extracted from edge and boundary enhanced image. Following it is similarity measurement task is carried out by the method of probability multiplication made on local pixels, written as,11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\wp\:}=\alpha\:\left(PE\left({P}_{BE}\left(x,y\right)\right)\right)$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document} is the probability of multiplication function. It gives the probability of occurrence of pixel value in the resulting patch using the probability of occurrence of individual pixels. The resulting patch extracted image is denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{PE}\left(x,y\right)$$\end{document}. Patch enhancement During the patch extraction step, there is a possibility of contrast reduction of image. Hence to overcome this problem, further, the extracted patches are enhanced for the betterment of the image during the image generation process. In the proposed method, the Distributed Signal (DS) processing is used for efficient patch enhancement without the knowledge of the entire image. Generally, the colors in the extracted patches are distributed and in Distributed Signal, Karhunen-Loeve Transform (KLT) is applied to these distributed colors that diagonalizes the correlation between the several distributed colors in the extracted patches after the patch extraction process. The KLT function is modeled as below:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{KLT}\left(x,y\right)={\sum\:}_{n=1}^{{\infty\:}}{V}_{n}{PE}_{P}\left(x,y\right)$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{V}_{n}$$\end{document} is termed as random variable. Further, post correlation determination, the contrast of the distributed colors are expanded to optimize the variance in the colors. Hence, the resultant patch enhanced image is obtained using this method is denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{PEnhanced}\left(x,y\right)$$\end{document}. Mathematical model for super-resolution image generation The mathematical details of the GF-CWAO-GAN Processing steps for shallow and deep feature extraction is described in the section below: Generator The generator network comprises several layers named Input layer, Convolutional layer, activation layer and sub-pixel layer and their functionalities as detailed below. Input layer The Patch Enhanced image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{PEnhanced}\left(x,y\right)$$\end{document} is fed into the input layer of the generator where the image is read and regularized before being forwarded to convolution layer. The output produced by this layer, denoted as IGAN is as below:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I_{GAN}=\;{regularise}({P}_{PEnhanced}\left(x,y\right))$$\end{document} Where, regularise brings comprehensibility to the patch enhanced image via regularise() function. Convolution layer The output of the regularized input layer undergoes characteristic/feature mapping in this convolution layer and therefore the resultant output of this convolutional layer denoted as ConvLGAN is shown obtained as indicated below:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{ConvL}_{GAN}=\zeta\:\left({\omega\:}^{o}*{I}_{GAN}+B\right)$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\omega\:}^{o}$$\end{document} signifies the hidden layer weight value that is being optimized using AO function and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\zeta\:$$\end{document}and B denotes the kernel size and bias value. Activation layer The usage of activation layer in the generator model helps detect the hidden patterns in the image. In this layer, the only characteristic features are extracted using Parametric ReLu (PReLu) activation function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi$$\end{document}, expressed in Eq. (15) below is designed to further optimize generator neural network performance. PReLu is an advanced variation of traditional Leaky ReLu and ReLu that is designed for improving optimization performance of neural networks. The resultant output of activation layer denoted ActivationGAN is as given below:15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Activation_{GAN} =\:\chi\:\left({ConvL}_{GAN}\right)=\left\{\begin{array}{c}{ConvL}_{GAN},\:\:\:\:\:\:\:\:\:if{ConvL}_{GAN}>0\\\:w{ConvL}_{GAN},\:\:\:\:if{ConvL}_{GAN}\le\:0\end{array}\right\}$$\end{document} Where, w is the ReLu constant. Sub-pixel Convolution layer: In this layer, mutli-channel features are extracted from ActivationGAN followed by merging operation for generation of SR image. The sub-pixel convolution layer output, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\widehat{SPL}}_{GAN}\right)$$\end{document} is,16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{S}PL}_{GAN}=\varPsi\:\left({Activation}_{GAN}\right)$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varPsi\:$$\end{document} is the merging function. Thus, the resultant SR image generated is represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{G}_{SR}\left(x,y\right)$$\end{document}. Discriminator Remote sensing images comprise of rich texture and ample amount of information in it which can be used to discern the image originality of the rebuilt HR remote sensing images which in turn help in improving the quality of the image. This article proposes Gompertz Function (GF) layer in the discriminator model of GF-CWAO GAN model. The GF layer outputs the prominent value of each discrete image component(pixel) in the image and thus improve the capability of GF-CWAO GAN model to distinguish the image details. Further, the generated image from the generator is passed through the discriminator where originality of the image is verified through the series of layers stacked like convolution layer, activation layer, matching layer and GF layer. The process begins with feature mapping at convolution layer followed by active feature extraction. The resulting feature points are fed into the matching layer at which the following task takes place. Initial matching This stage is determined to establish homography relationship between the resulting features and is denoted by17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{H}_{SR}=\frac{1}{p}\left({\sum\:}_{z}{P}_{SR}^{Yi}\left(x,y\right){P}_{SR}^{Yj}\left(x,y\right)\right)$$\end{document} Where, z denotes p number of active features and (Yi, Yj) refers to the initial matching points. Propagation matching Since the initial matching, also called shallow feature mapping, does not cover the entire feature points in the generated image, propagation matching also referred to as deep feature mapping is done that involves two different steps probability relaxation and geometric correspondence matching respectively. The features points that were not matched in the initial matching stage are selected and the homography relationship is determined between the initial matching output and the selected feature points. In probability relaxation matching, the candidate matching features are computed for the exactly matched feature points. The probability of the exact matched feature points, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\widehat{H}}_{abs}\right)$$\end{document}, is specified as,18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{H}}_{abs}=\varOmega\:\left({H}_{SR}\right)$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\varOmega\:$$\end{document} is the probability function and the matching layer output becomes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{M}}_{SR}$$\end{document}. GF Layer: Among the matching feature points, only the contributing feature points are selected in this layer by using Gompertz Function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(GF_{SR})$$\end{document} by avoiding convergence problem and is denoted as below:19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$GF_{SR}=Fe^{-ge^-{h\tau}}$$\end{document} Where, F means an asymptote, g is the displacement vector and h mean the growth rate at time τ. Finally, the discriminator provides the output as real or fake and is denoted as\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{C}_{DSR}\left(x,y\right).$$\end{document} Convergence War AO (CWAO)-based weight optimization War strategy optimization is a meta-heuristic optimization algorithm that is based on strategic movement of army troops during the war. The war strategy is modelled such that each soldier dynamically reaches the optimal value as a part of optimization process. Weight updating mechanism and weak soldier relocation strategy is followed to improve convergence and robustness of the model25. War strategy optimization is also used for efficient scanning electron microscopy image segmentation through effective optimized weights26. In the proposed architecture, the war strategy optimization idea is incorporated as weight updating mechanism during the training process to achieve the local optimum. Hence Convergence Accelerometric Function (CAF) is proposed for weight adjustment for achieving the global optimal solution by backpropagation method while training the model. This amalgamation of CAF with the traditional WSO is termed as Convergence War Accelerometric Optimization (CWAO). This process is detailed below. Phase 1: Initially, the weight values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\omega^{\widehat{G}})$$\end{document} are randomly distributed on the nodes as follows: 20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega^{\widehat{G}}=\left\{{\omega^{1},\omega^{2},\omega^{3},...,\omega^{N}}\right\}$$\end{document} Then, the generation accuracy (fitness) of each is determined and the one with the better fitness value is selected as king. The fitness determination \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(f(\omega^{\widehat{G}}))$$\end{document} is represented as equation (21).21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(\omega^{\widehat{G}})=\max_{accuracy}(\omega^{\widehat{G}})$$\end{document} Phase 2: Further, the weight of every node is updated depending on the outcome of the current weights. If the image super-resolution improves, then based on the strategy success, the weights of all the nodes are varied. Thus, the updated weights of the nodes become,22\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi(\omega^{\widehat{G}}(t+1))=\xi(\omega^{\widehat{G}}(t))+2\times\varphi\times(\vartheta-\rho)+rnd$$\end{document} Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi(\omega^{\widehat{G}}(t+1))$$\end{document} determines the updated weight, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varphi$$\end{document} signifies the earlier weight and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vartheta-\rho$$\end{document} defines the position of the nodes. Phase 3: Based on the updated image, the weights are also updated. If the current weight (En) is closer to the earlier weight (Ew), then the update equation is as given below:23\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi(\omega^{\widehat{G}}(t+1))=\xi(\omega^{\widehat{G}}(t+1))+(E_n\geq\,E_{\stackrel{-}{w}}+\xi(\omega^{\widehat{G}})$$\end{document} Phase 4: Next as a defence strategy, the position of the nodes in the network is updated. During every phase, the diminishing nodes (that worsen the image quality) are identified and replaced using the CAF as given in the equation below:24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi(\omega^{\widehat{G}}+1)=RND\times\frac{\text{(}{\stackrel{-}{w}}{best}-{\stackrel{-}{w}}_{ff})}{{f}({\stackrel{-}{w}}_{best})+\upsilon}$$\end{document} Where, RND refers to the randomly distributed random variable, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\stackrel{-}{w}}_{best}$$\end{document} is the best weight, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\upsilon$$\end{document} is the CAF. The above process is repeated until the optimal weights are obtained and is represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega^\circ$$\end{document} . The proposed CWAO pseudocode is presented below. Experiments Data preparation The random 128 × 128 pixel crops are extracted from the HR image training dataset. Data Augmentation was applied by randomly flipping the cropped images horizontally, generating around 20 more training samples from each image and following it with normalization to the range [0, 1] which helped prevent over-fitting27. Training details Model training is carried out in dual stages: a PSNR-oriented stage and GF-CWAO-GAN stage. Each stage focused on enhancing different aspects of image SR task. A final Interpolation Model was obtained by taking the weighted average values of both the models. PSNR – oriented training The primary focus of this stage prioritized to attain a high Peak-Signal-to-Noise-Ratio (PSNR) as an objective measure of image quality. The details pertaining to this are as follows: Loss Function: L1 pixel loss was considered as the primary loss function that calculates the pixel wise difference between the generated HR image and the ground truth image. Lower the loss value indicates superior performance thereby resulting in improved PSNR metrics6. 25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:MAE=1/n{\sum\:}_{i=1}^{n}|\text{y}\text{i}-{\widehat{y}}i\:\:|$$\end{document} Loss Function: L1 pixel loss was considered as the primary loss function that calculates the pixel wise difference between the generated HR image and the ground truth image. Lower the loss value indicates superior performance thereby resulting in improved PSNR metrics6. Loss Function: L1 pixel loss was considered as the primary loss function that calculates the pixel wise difference between the generated HR image and the ground truth image. Lower the loss value indicates superior performance thereby resulting in improved PSNR metrics6. Here, yi is the ground truth image value; yˆi is the predicted value; n is the number of samples. Optimizer and Learning Rate Schedule: Here ADAM optimizer is applied with initial LR of 2 × 10−4. The incorporation of multi-step scheduler diminished the learning rate by a factor of 0.5 i.e., half at iterations [1000, 3000, 5000, 7000]9.Training Parameters: The model was trained for 8000 iterations with a batch size of 16.Check-Pointing: Model check points were saved every 1000 iterations that allowed the model to train with limited GPU capabilities by pausing and resuming in two cycles of 4000 iterations each. Optimizer and Learning Rate Schedule: Here ADAM optimizer is applied with initial LR of 2 × 10−4. The incorporation of multi-step scheduler diminished the learning rate by a factor of 0.5 i.e., half at iterations [1000, 3000, 5000, 7000]9. Optimizer and Learning Rate Schedule: Here ADAM optimizer is applied with initial LR of 2 × 10−4. The incorporation of multi-step scheduler diminished the learning rate by a factor of 0.5 i.e., half at iterations [1000, 3000, 5000, 7000]9. Training Parameters: The model was trained for 8000 iterations with a batch size of 16. Training Parameters: The model was trained for 8000 iterations with a batch size of 16. Check-Pointing: Model check points were saved every 1000 iterations that allowed the model to train with limited GPU capabilities by pausing and resuming in two cycles of 4000 iterations each. Check-Pointing: Model check points were saved every 1000 iterations that allowed the model to train with limited GPU capabilities by pausing and resuming in two cycles of 4000 iterations each. GF-CWAO-GAN training The image taken from the dataset undergoes patch extraction as discussed in Sect. 3.3 followed by patch clustering step. Patch Clustering is done by applying KL-KM-KMA. KMA is the most widely used clustering approach due to its better clustering efficiency in reduced time28[,29. Nonetheless, the Euclidean distance-based distance measurement affected the clustering accuracy. Hence, the Krzanowski and Li Index (KLI) and the Kantorovich Metric (KM) are applied for discrepancy/distance measurement, used to describe the similarity between the real and generated images. This combination of KLI and KM to the conventional KMA is termed as KLI-KM-KMA is discussed below. The patches extracted from remote sensing SR \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({R}^{c}\right)$$\end{document} is given as input to KLI-KM-KMA and clustered into N number of cluster centers \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\stackrel{-}{C}}^{N}\right)$$\end{document}. Here the cluster centres are randomly selected, and the number of clusters are selected using KLI\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\stackrel{-}{C}}_{KLI}^{N}\right)$$\end{document} as detailed below.26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\stackrel{-}{C}}_{KLI}^{N}=\left|\frac{diff\left({R}^{c}\right)}{diff\left({R}^{c-1}\right)}\right|$$\end{document} Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:diff\left({R}^{c}\right)\:and\:diff\left({R}^{c-1}\right)$$\end{document} defines the within group dispersion of the cth and c-1th feature. Then, the distance between the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({R}^{c}\right)$$\end{document} and the selected cluster centroid \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\stackrel{-}{C}}^{N}\right)$$\end{document} is evaluated by KM function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(k\left({R}^{c},{\stackrel{-}{C}}^{N}\right)\right)$$\end{document}, defined as,27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:k\left({R}^{c},{\stackrel{-}{C}}^{N}\right)=SUP\left\{\int\:{R}^{c}dc-\int\:{\stackrel{-}{C}}^{N}.{R}^{c}dc\right\}$$\end{document} Where, SUP demonstrates the real-valued function. Further, the extracted patches are clustered to the nearest centroid with a minimal central difference. Hence the objective function (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathcal{F}$$\end{document}) of KLI-KM-KMA can be defined as,28\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathcal{F}={{\sum\:}_{c=1}^{e}{\sum\:}_{N=1}^{l}\left|k\left({R}^{c},{\stackrel{-}{C}}^{N}\right)\right|}^{2}$$\end{document} The above steps are repeated until the cluster centres remain unchanged. Therefore, the final \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\stackrel{\sim}{m}}-$$\end{document} number of patch clusters formed using KLI-KM-KMA \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({\nu\:}^{p}\right)$$\end{document} is represented as the following set as given below,29\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\nu\:}^{p}=\left\{{\nu\:}^{1},{\nu\:}^{2},{\nu\:}^{3},{\nu\:}^{4},...,{\nu\:}^{\stackrel{\sim}{m}}\right\}$$\end{document} The pseudocode for the proposed KLI-KM-KMA is presented as below, Generator Initialization: The proposed work implemented Transfer Learning via initializing the generator here by loading the weights from the PSNR-Oriented model.Discriminator Network: The Discriminator Network was trained adversarial alongside the Generator Network. PatchGAN architecture network was used in discriminator that guide in distinguishing between real and generated HR images.Loss Function: The loss function used here during training is a combination of L1 Pixel Loss (Mean Absolute Error) and VGG (Perceptual Loss).L1 Pixel Loss: Assigned a lower weight i.e., 1 × 10−2 to maintain a minimum image quality. 30\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\text{W}\text{e}\text{i}\text{g}\text{h}\text{t}\text{e}\text{d}\:\text{L}1\:\text{P}\text{i}\text{x}\text{e}\text{l}\:\text{L}\text{o}\text{s}\text{s}=1\:\text{x}\:{10-}^{2}\:x\:1/N{\sum\:}_{i=1}^{n}|\text{y}\text{i}-{\widehat{y}}i\:\:|$$\end{document} Generator Initialization: The proposed work implemented Transfer Learning via initializing the generator here by loading the weights from the PSNR-Oriented model. Generator Initialization: The proposed work implemented Transfer Learning via initializing the generator here by loading the weights from the PSNR-Oriented model. Discriminator Network: The Discriminator Network was trained adversarial alongside the Generator Network. PatchGAN architecture network was used in discriminator that guide in distinguishing between real and generated HR images. Discriminator Network: The Discriminator Network was trained adversarial alongside the Generator Network. PatchGAN architecture network was used in discriminator that guide in distinguishing between real and generated HR images. Loss Function: The loss function used here during training is a combination of L1 Pixel Loss (Mean Absolute Error) and VGG (Perceptual Loss). Loss Function: The loss function used here during training is a combination of L1 Pixel Loss (Mean Absolute Error) and VGG (Perceptual Loss). L1 Pixel Loss: Assigned a lower weight i.e., 1 × 10−2 to maintain a minimum image quality. L1 Pixel Loss: Assigned a lower weight i.e., 1 × 10−2 to maintain a minimum image quality. VGG Feature Loss (Perceptual Loss): Pretrained VGG model was used with the weight assignment of 1.0 for feature maps extraction. This loss compares the loss in the feature maps extracted for both generated HR and ground truth images. 31\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$VGG \;Feature \;Loss\;\:=\:\lambda\:\:\mathsf{{\rm\:X}}{\sum\:}_{i=1}^{N}\left|\right|{\upphi\:}\text{i}({\widehat{y}})-\:{\upphi\:}\text{i}\left(\text{y}\right)\left|\right|2$$\end{document} VGG Feature Loss (Perceptual Loss): Pretrained VGG model was used with the weight assignment of 1.0 for feature maps extraction. This loss compares the loss in the feature maps extracted for both generated HR and ground truth images. VGG Feature Loss (Perceptual Loss): Pretrained VGG model was used with the weight assignment of 1.0 for feature maps extraction. This loss compares the loss in the feature maps extracted for both generated HR and ground truth images. Where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\lambda\:$$\end{document} is the weight assigned as 1.0; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\upphi\:}\text{i}({\widehat{y}})$$\end{document} - the feature maps extracted from the generated image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{y}}$$\end{document} by the VGG model; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\upphi\:}\text{i}\left(\text{y}\right)$$\end{document} – the feature maps extracted from the ground truth image y; N – total number of layers in the VGG model. Optimizers and LR Schedule: Like the earlier training, ADAM optimizers were used for both generator and discriminator with the difference being different LR Schedules for each network. The initial LR was set to 1 × 10 − 4 for both, with a decay factor of 0.5 i.e., half with multistep scheduler at iterations [500, 1000, 2000,3000].Training Parameters: Trained the model for 4000 iterations with a batch size of 16. Here the number of iterations were reduced since this model is built upon the earlier trained weights from the PSNR-Oriented model.Check-pointing: While training this model, check points were saved every 500 iterations, and the training was carried out along 2 cycles comprising 2000 iterations each. Optimizers and LR Schedule: Like the earlier training, ADAM optimizers were used for both generator and discriminator with the difference being different LR Schedules for each network. The initial LR was set to 1 × 10 − 4 for both, with a decay factor of 0.5 i.e., half with multistep scheduler at iterations [500, 1000, 2000,3000]. Optimizers and LR Schedule: Like the earlier training, ADAM optimizers were used for both generator and discriminator with the difference being different LR Schedules for each network. The initial LR was set to 1 × 10 − 4 for both, with a decay factor of 0.5 i.e., half with multistep scheduler at iterations [500, 1000, 2000,3000]. Training Parameters: Trained the model for 4000 iterations with a batch size of 16. Here the number of iterations were reduced since this model is built upon the earlier trained weights from the PSNR-Oriented model. Training Parameters: Trained the model for 4000 iterations with a batch size of 16. Here the number of iterations were reduced since this model is built upon the earlier trained weights from the PSNR-Oriented model. Check-pointing: While training this model, check points were saved every 500 iterations, and the training was carried out along 2 cycles comprising 2000 iterations each. Check-pointing: While training this model, check points were saved every 500 iterations, and the training was carried out along 2 cycles comprising 2000 iterations each. Network interpolation Though both PSNR-Oriented and GF-CWAO-GAN model have provided remarkable results as individual models, yet it implemented network interpolation by taking weighted average of both the models to improve visual perception of image quality. The interpolation factor, α, varied to control the weight of the model where α was set to 0.8 (higher) for GF-CWAO-GAN model while taking influence from PSNR-Oriented model too. Both models used ADAM optimizers with β1 and β2 set to 0.9 and 0.99 values for PSNR-Oriented and GF-CWAO-GAN model respectively – implemented using Tensorflow and trained on Colab’s TPU offerings. Results and discussion This segment discusses the corresponding results obtained from the experiments conducted as mentioned in earlier section using the dataset mentioned below under section A. Dataset The WorldStrat dataset is used for training, SR image generation and evaluation. This dataset is commonly used in deep learning for remote sensing image SR. This dataset originally contains HR images of 1.5 m/pixel which were resized into dimensions [512,512] and around 405 such HR images are considered for experiment. The images were downsampled by applying Lanczos resampling during this process and resized into LR images of dimensions [128, 128] for testing the model. Further, the downsampled images are randomly assigned to the training set (80%) and the test set (20%). Evaluation metrics used The performance of the proposed model was evaluated using the following metrics. Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Inception Score, Fréchet Inception Distance, Accuracy, Precision and Sensitivity. Quantitative evaluation Quantitative evaluation was carried out using four metrics mentioned above in sub section B of Sect. 5. Table II below shows the PSNR and SSIM scores achieved for GF-CWAO-GAN, Medi-ESRGAN, SRGAN and ESRGAN. It can be observed that GF-CWAO-GAN outperforms over the other models on all images, both in terms of PSNR and SSIM metrics. As shown in Table 1, it can be observed that the results of the four GAN-based SR models for PSNR and SSIM are calculated for 5 images and averaging them (as shown in last column and corresponding graph in Fig. 5 below) across multiple images to obtain reliable and robust measure of overall performance of proposed SR model by smoothing out individual variability, thereby facilitating consistent comparison with other state-of-the-art models. It is observed that GF-CWAO-GAN (35.86dB, 0.903) and Medi-ESRGAN (34.83dB, 0.896) have outperformed PSNR and SSIM parameter values respectively. The good performance of GF-CWAO-GAN model expresses that it is effective and feasible to use this model for SR reconstruction of remote sensing hyperspectral images. Table 1Quantitative evaluation of proposed model with other gan models based on psnr and ssim.PSNR (dB)Image 1Image 2Image 3Image 4Image 5AverageGF-CWAO-GAN35.9735.6235.9335.9135.8835.86Medi-ESRGAN35.8935.8935.5936.132.6634.83SRGAN33.9334.3732.5735.3531.4433.53ESRGAN31.8531.9731.4232.2330.0831.51 SSIM Image 1 Image 2 Image 3 Image 4 Image 5 Average GF-CWAO-GAN0.8920.8780.9320.9160.8970.903Medi-ESRGAN0.9120.9080.9710.9180.8720.896SRGAN0.8710.8680.8440.8820.8540.864ESRGAN0.6710.7170.7480.7050.7140.711 Table 1Quantitative evaluation of proposed model with other gan models based on psnr and ssim.PSNR (dB)Image 1Image 2Image 3Image 4Image 5AverageGF-CWAO-GAN35.9735.6235.9335.9135.8835.86Medi-ESRGAN35.8935.8935.5936.132.6634.83SRGAN33.9334.3732.5735.3531.4433.53ESRGAN31.8531.9731.4232.2330.0831.51 SSIM Image 1 Image 2 Image 3 Image 4 Image 5 Average GF-CWAO-GAN0.8920.8780.9320.9160.8970.903Medi-ESRGAN0.9120.9080.9710.9180.8720.896SRGAN0.8710.8680.8440.8820.8540.864ESRGAN0.6710.7170.7480.7050.7140.711 Quantitative evaluation of proposed model with other gan models based on psnr and ssim. Fig. 5Performance analysis of proposed model for parameters (a) PSNR and (b) SSIM. Performance analysis of proposed model for parameters (a) PSNR and (b) SSIM. We also correlated the proposed model with the other three state-of-the-art models’ perceptual image quality, training time and resource requirements as shown in Table 2 below. The training time was measured from the start of the training stage until the point of convergence where the predefined validation loss threshold is reached. Our model demonstrated a significant reduction in training time, completing the training stage in 6–7 h compared to the other reference/guiding models like ESRGAN and SRGAN. This improvement in training time is attributed to the use of CWAO weight optimization, ADAM optimizer and LR schedule in turn providing enhanced efficiency. All the experiments are implemented on Colab T4 GPU (16GB VRAM) to combat the computationally expensive physical resource limitations. Table 2Comparison of different gan models.ModelDatasetImage Quality (PSNR/SSIM)TrainingTimeResource RequirementsGF-CWAO-GANWorldStrat35.86/0.9036–7 hColab T4 GPU (16GB VRAM)Medi-ESRGANSTARE (8000 + 4000)34.83/0.8966–7 hColab T4 GPU (16GB VRAM)SRGANCelebA/1033.53/0.864~ 10 hNVIDIA GTX 1080 Ti (11GB VRAM)ESRGANDiv2k/10031.51/0.71112–24 hNVIDIA RTX 3080 Ti (10GB VRAM) Table 2Comparison of different gan models.ModelDatasetImage Quality (PSNR/SSIM)TrainingTimeResource RequirementsGF-CWAO-GANWorldStrat35.86/0.9036–7 hColab T4 GPU (16GB VRAM)Medi-ESRGANSTARE (8000 + 4000)34.83/0.8966–7 hColab T4 GPU (16GB VRAM)SRGANCelebA/1033.53/0.864~ 10 hNVIDIA GTX 1080 Ti (11GB VRAM)ESRGANDiv2k/10031.51/0.71112–24 hNVIDIA RTX 3080 Ti (10GB VRAM) Comparison of different gan models. The performance of the proposed approach is computed for evaluation metrics Fréchet Inception Distance (FID), Inception Score (IS), Accuracy, Precision and Sensitivity. Table 3below unveils the comparative measure of the proposed technique in terms of IS and FID. As per the studies, while the original paper ESRGAN has not reported IS and FID values, the other literature have indicated that higher the values of IS and lower the FID, the better the GANs are suited for image generation10,11. From Table 3, it can be noticed that the proposed model surpasses with the IS value of 8.71 (higher) and FID value of 36.4 (lower) over the GANs and CNNs/DNNs. Thus, the proposed method generates noise free SR images when compared to other traditional methodologies. Table 3Comparitive measure of IS and FID.ModelInception Score (IS)Fréchet Inception Distance (FID)GF – CWAO - GAN8.7136.4Improved GAN8.0937.7DNN/CNN6.1629.3 Table 3Comparitive measure of IS and FID.ModelInception Score (IS)Fréchet Inception Distance (FID)GF – CWAO - GAN8.7136.4Improved GAN8.0937.7DNN/CNN6.1629.3 Comparitive measure of IS and FID. Following further, the performance measure of the proposed CWAO in terms of Fitness (as shown in Fig. 6 below) is presented to substantiate its superiority while comparing it with the existing Particle Swarm Optimization Algorithm (PSOA), Salp Swarm Optimization Algorithm (SSOA), Whale Optimization Algorithm (WOA) and White Shark Optimization Algorithm (WSOA). Fitness is typically evaluated using the combination of IS, FID and perceptual quality metrics (i.e., SSIM) to provide comprehensive assessment of quality of generated images guiding optimization of proposed model during training process. The CWAO incorporation into the proposed model achieves better performance gains by preventing premature convergence that is prevalent in basic WOA and PSOA. However, the usage of Gompertz function-based GAN permits CWAO to remain stable while learning high-frequency details crucial to SR tasks. CWAO converges faster for SR task when compared to SSOA, WSOA and WOA when handling large datasets validating the efficiency of Image SR task. Fig. 6Fitness performance measure for the proposed technique. Fitness performance measure for the proposed technique. Advancing further, performance analysis of the proposed KL-KM-KMA in terms of clustering time with the other state-of-the-art clustering algorithms like K-Means (KMA), Fuzzy C Means (FCM), Mean Shift (MS) and Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) is made and is tabulated in Table 4 as below. Table 4Performance analysis of kl-km-kma with other state-of_the_art clustering algorithms.ModelClustering Time (ms)Proposed KL-KM-KMA16,523KMA24,879FCM32,543MS5980BIRCH7214 Table 4Performance analysis of kl-km-kma with other state-of_the_art clustering algorithms.ModelClustering Time (ms)Proposed KL-KM-KMA16,523KMA24,879FCM32,543MS5980BIRCH7214 Performance analysis of kl-km-kma with other state-of_the_art clustering algorithms. It is clear from the above Table 4 that the time taken to form clusters by the proposed approach is reduced (16523ms) compared to the conventional K-Means algorithm and FCM which confirms that the usage of KL-KM approach has avoided the issue of convergence hence improving the SR image generation performance. MS and BIRCH have lower Clustering timing against the compromise on other metrics. The accomplishment of the proposed GF-CWAO-GAN approach with conventional GAN in terms of accuracy, precision, recall and sensitivity is shown in Table 5 and Fig. 7 below. It can be noticed that the proposed approach is superior in performance for all the metric listed above by the virtue of usage of GF and CWAO. Table 5Comparison of performance measures.ModelAccuracy(%)Precision(%)Recall(%)Sensitivity(%)GF-CWAO-GAN98.0597.9897.8697.88Conventional GAN93.4292.7892.5492.63 Table 5Comparison of performance measures.ModelAccuracy(%)Precision(%)Recall(%)Sensitivity(%)GF-CWAO-GAN98.0597.9897.8697.88Conventional GAN93.4292.7892.5492.63 Comparison of performance measures. Fig. 7Comparison of Performance Measures for GF-CWAO-GAN and Conventional GAN. Comparison of Performance Measures for GF-CWAO-GAN and Conventional GAN. Figure 8 below depicts the training versus test generator loss indicating that the training and test losses converge with the test loss being stable, indicating better generalization of model. Figure 9 below depicts the total generator loss curve decreasing over the training steps, indicating that GF-CWAO-GAN is successfully learning to generate realistic HR images. The fluctuations noticed in the plot are a result of adversarial training process. The stabilizing loss around step 1500 indicates that the model has entered a stable learning phase. Fig. 8Training versus Test generator loss. Training versus Test generator loss. Fig. 9Total Loss. Total Loss. Qualitative evaluation Qualitative evaluation focusses on perceptual quality of SR image generated by the proposed model. It complements quantitative metrics like PSNR and SSIM to assess whether the generated images are realistic, sharp and visually appealing. The method focusses on restoring edges, textures and high-frequency information accurately, while avoiding blurriness. The higher Inception Score and lower Fréchet Inception Distance values achieved, and side-by-side visual comparisons are made as shown in figures below ensuring the visible improvements in the perceptual quality with respect to features like sharpness and texture consistency for LR Images, generated images and ground truth images. Visual images generated showed sharp edges with minimal artifacts that are closer to the ground truth image as shown in the Figs. 10, 11 and 12 below. Fig. 10Sample image results for (a) Input image (b) Noise removed image (c)Tone mapped image (d) Skew corrected image (e) Boundary and edge enhanced image (f) Patch extraction (g) Patch enhancement. Sample image results for (a) Input image (b) Noise removed image (c)Tone mapped image (d) Skew corrected image (e) Boundary and edge enhanced image (f) Patch extraction (g) Patch enhancement. Fig. 11Sample Output 2. Sample Output 2. Fig. 12Sample Output 3. Sample Output 3. Conclusion In this paper, a novel deep learning-based unsupervised GF-CWAO-GAN is proposed for SISR of hyperspectral remote sensing image accomplished by training the model using KL-KM-KMA based clustering. Experiments were conducted on the benchmark dataset to show the performance and comparative analysis of the proposed model. It outperformed comparatively better over existing state-of-the-art GAN-based models both qualitatively and quantitatively with metric values, accuracy – 98.05%, precision – 97.98%, recall – 97.86%, PSNR – 35.86, SSIM – 0.903, Inception Score – 8.71, Fréchet Inception Distance – 36.4, clustering time – 16523ms and reduced model training time of 6–7 h. This work also proposed a solution to mitigate the necessity of expensive hardware and computational resources thereby improving SR image quality by utilizing more readily available resources. Even though the integration of Gompertz function to the model for Image SR has introduced certain merits like better convergence, quantitative metrics, reduced training time and learning rate, nonetheless it has introduced limitations like slow initial training, generalization issues with diverse datasets, reconstruction accuracy of complex textures, optimization and resource requirements that could be overcome by exploring adaptive mechanisms to mitigate these shortcomings.