Uncertainty Estimation for Super-Resolution using ESRGAN
Maniraj Sai Adapa 1 , Marco Zullich 1 and Matias Valdenegro-Toro 1 1 Department of AI, University of Groningen, Nijenborgh 9, 9747AG, Groningen, Netherlands [email protected], m.zullich, [email protected] · 2024
Read Uncertainty Estimation for Super-Resolution using ESRGAN by Maniraj Sai Adapa 1 , Marco Zullich 1 and Matias Valdenegro-Toro 1 1 Department of AI, University of Groningen, Nijenborgh 9, 9747AG, Groningen, Netherlands [email protected], m.zullich, [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15439.txt.
Introduction
Maniraj Sai Adapa, Marco Zullich, and Matias Valdenegro-Toro add doubt estimates to image sharpening in a 2024 paper. Deep learning tools that upscale images rank among the best, but they give no sign of where their output may be wrong. A user sees a sharp image with no warning on the risky parts.
Two known tricks were added to a top sharpening model: random dropout at test time and a set of models whose outputs are averaged. Both let the system mark pixels where it is unsure. Tests showed those marks were fairly well matched to real error, with no drop in image quality.
Catalog record
| CREATOR | Maniraj Sai Adapa 1 , Marco Zullich 1 and Matias Valdenegro-Toro 1 1 Department of AI, University of Groningen, Nijenborgh 9, 9747AG, Groningen, Netherlands [email protected], m.zullich, [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertainty Estimation for Super-Resolution using ESRGAN Maniraj Sai Adapa 1 , Marco Zullich 1 and Matias Valdenegro-Toro 1 1 Department of AI, University of Groningen, Nijenborgh 9, 9747AG, Groningen, Netherlands [email protected], m.zullich, [email protected] Abstract Deep Learning–based image super-resolution (SR) has been gaining traction with the ad of Generative Adversarial Networks. Models like SRGAN and ESRGAN are constantly ranked between the best image SR tools. However, they lack principled ways for estimating predictive uncertainty. In the present work, we enhance these models using Monte Carlo–Dropout and Deep Ensemble, allowing the computation of predictive uncertainty. When coupled with a prediction, uncertainty estimates can provide more information to the model users, highlighting pixels where the SR output might be uncertain, hence potentially inaccurate, if these estimates were to be reliable. Our findings suggest that these uncertainty estimates are decently calibrated and can hence fulfill this goal, while providing no performance drop with respect to the corresponding models without uncertainty estimation. 1 INTRODUCTION Super-Resolution (SR) is an important computer vision task, where a low-resolution image is upscaled to a higher resolution one. It is fundamentally an inverse problem, where missing information needs to be filled by making assumptions encoded in a model, which can lead to errors, as shown in Figure 1 . Many efforts are made to improve SR models to increase their accuracy, but any model will tend to produce erroneous outputs if the input is outside the training distribution. An important task is then provide feedback to a human user on which pixels or regions of the SR output image are likely to be incorrect or imprecise. In this paper we combine two uncertainty estimation methods with a state of the art SR model—Super Resolution Generative Adversarial Network (SRGAN) [ Ledig et al., 2017 ] and Enhanced SRGAN (ESRGAN) [ Wang et al., 2018 ] —, to build a SR model with epistemic uncertainty estimation, which outputs a SR image and a uncertainty map, indicating which regions are likely to be incorrect. We evaluate the performance of uncertainty estimation, noting that an ensemble of 5 ESRGAN generators works the best, and provide extensive quantitative and qualitative results, showcasing the usefulness of uncertainty estimation in the SR domain. We posit our results show that uncertainty estimation, in particular ensembles, can provide useful feedback to a human using SR results, and the standard deviation produced by the model can work as a per-pixel error proxy. Image SR Uncert Overlay Error vs Std Crop SR Crop Uncert Image SR Uncert Overlay Error vs Std Crop SR Crop Uncert Image SR Uncert Overlay Error vs Std Crop SR Crop Uncert Figure 1: Ensemble Results using ESRGAN with Uncertainty, including error vs standard deviation plots. This figure shows how SR uncertainty correlates with SR reconstruction errors and can be used to detect possible errors at inference time. Error vs Std plots show that uncertainty correlates very well with absolute errors at the pixel level. The contributions of this paper are: we build SRGAN and ESRGAN models with uncertainty estimation, we evaluate uncertainty performance in several well known datasets, and validate that model standard deviation can be used as a proxy for test time error. This work expands the state of the art by building simple combinations of a state of the art SR model (ESRGAN) with Monte Carlo Dropout and Ensembles, with an explicit focus on qualitative and quantitative uncertainty estimation, showing that uncertainty can be used as a proxy for error at inference time, for natural color images. Note that our work is not about improving the super-resolution task performance, but we argue that fundamentally any super-resolution model will make mistakes at some point, especially with out of distribution images, and uncertainty estimation is a key component to notify the end user about these mistakes via higher per-pixel uncertainty. 2 State of the Art The literature for super resolution uncertainty is relatively underexplored. Kar and Biswas [ Kar and Biswas, 2021 ] use stochastic Batch Normalization for uncertainty estimation in deep SISR models. Liu et al. [ Liu et al., 2023 ] estimate uncertainty in spectral domain instead of the common spatial domain for the DDL-EDSR model. SR Uncertainty outside of natural images is also present. Tanno et al [ Tanno et al., 2017 ] use variational dropout for SR of 3D Diffusion MRI brain images, while Song and Yang [ Song and Yang, 2023 ] use Bayesian Neural Networks for SR in wave array imaging and making separate predictions of aleatoric and epistemic uncertainty. Most previous research on SR uncertainty focuses on improving SR accuracy using uncertainty estimation ( [ Kar and Biswas, 2021 ] and [ Liu et al., 2023 ] ), or are applied to domains outside of natural images. There is often not a deep focus on uncertainty quantification for SR and its consequences. We perform a deep evaluation of uncertainty quality for SR, as we assume that SR models will always make errors in out of distribution settings, and per-pixel output uncertainty can guide the human user to detect these errors. 3 ESRGAN WITH UNCERTAINTY 3.1 Image SR Using GAN-based Models SRGAN [ Ledig et al., 2017 ] and ESRGAN [ Wang et al., 2018 ] have emerged as industry-standard architectures for image super-resolution. Both techniques make use of a Generative Adversarial Network (GAN) framework, whereas a generator is trained to super-resolve low-resolution images, while a discriminator is simultaneously trained to distinguish between real high-resolution images and the output of the generator. SRGAN demonstrates the potential of GANs for super-resolution by utilizing an adversarial loss versus the discriminator. Building upon these foundations, ESRGAN further enhances the approach with improvements in both generator and discriminator architecture, focusing on optimizing perceptual quality. The success of SRGAN and ESRGAN has made them widely adopted baseline approaches, with code and pre-trained models readily available. While these pre-trained GAN models offer strong performance, training customized models from scratch can provide advantages when exploring specific techniques like uncertainty estimation and also provide architectural similarity. Formally, we call D the discriminator and G the generator. Each training data point is composed by a high-resolution image Y\in\mathbb{R}^{h\times w} acting as ground truth and its low-resolution equivalent X\in\mathbb{R}^{h^{\prime}\times w^{\prime}} , with h>h^{\prime},w>w^{\prime} , acting as input. Depending on the specific dataset employed, X is usually obtained from Y by applying a specific downsampling technique, such as bicubic interpolation. The generator G takes as input X and produces a super-resolved image \hat{Y}\in\mathbb{R}^{h\times w} . The discriminator D is fed a high-resolution image (either Y or \hat{Y} ) and outputs a scalar r\in(0,1) , which can be interpreted as a probability value of that image being a real high-resolution image. 3.1.1 SRGAN The discriminator of the SRGAN plays a crucial role in adversarial training. As the discriminator becomes effective in distinguishing the super-resolved images, the adversarial process forces the generator to produce images that are increasingly realistic. The discriminator is a simple convolutional neural network classifier that consists of a series of strided convolution layers that are responsible for extracting hierarchical features and a final classification layer which produces the scalar output. SRGAN uses a loss function for the generator l_{G}^{\text{SRGAN}} for improving the perceptual quality of super-resolution results. It is