Auto-Encoded Supervision for Perceptual Image Super-Resolution
MinKyu Lee, Sangeek Hyun, Woojin Jun, Jae-Pil Heo · 2024
Read Auto-Encoded Supervision for Perceptual Image Super-Resolution by MinKyu Lee, Sangeek Hyun, Woojin Jun, Jae-Pil Heo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00124.txt.
Introduction
MinKyu Lee and co-authors sharpen AI photo enlargement in a 2024 paper. Super-resolution rebuilds a large, detailed image from a small one, and generative methods make results look real. But the loss that enforces closeness to the original photo also blurs, a known tug of war between fidelity and looks.
The team dissect that loss and isolate exactly which part causes the blur. An auto-encoder, trained with the same loss, separates the blurring part from the harmless part, so their new loss guides training only by the safe part. The result aims at enlargements that stay both faithful and crisp, without the usual workarounds that never fixed the root cause.
Catalog record
| CREATOR | MinKyu Lee, Sangeek Hyun, Woojin Jun, Jae-Pil Heo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00124
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Auto-Encoded Supervision for Perceptual Image Super-Resolution MinKyu Lee, Sangeek Hyun, Woojin Jun, Jae-Pil Heo Abstract This work tackles the fidelity objective in the perceptual super-resolution (SR) task. Specifically, we address the shortcomings of pixel-level ℒ p \mathcal{L}_{\text{p}} loss ( ℒ pix \mathcal{L}_{\text{pix}} ) in the GAN-based SR framework. Since ℒ pix \mathcal{L}_{\text{pix}} is known to have a trade-off relationship against perceptual quality, prior methods often multiply a small scale factor or utilize low-pass filters. However, this work shows that these circumventions fail to address the fundamental factor that induces blurring. Accordingly, we focus on two points: 1) precisely discriminating the subcomponent of ℒ pix \mathcal{L}_{\text{pix}} that contributes to blurring, and 2) only guiding based on the factor that is free from this trade-off relationship. We show that they can be achieved in a surprisingly simple manner, with an Auto-Encoder (AE) pretrained with ℒ pix \mathcal{L}_{\text{pix}} . Accordingly, we propose the Auto-Encoded Supervision for Optimal Penalization loss ( ℒ AESOP \mathcal{L}_{\text{AESOP}} ), a novel loss function that measures distance in the AE space 1 1 1 AE space indicates the space after the decoder, not the bottleneck. , instead of the raw pixel space. By simply substituting ℒ pix \mathcal{L}_{\text{pix}} with ℒ AESOP \mathcal{L}_{\text{AESOP}} , we can provide effective reconstruction guidance without compromising perceptual quality. Designed for simplicity, our method enables easy integration into existing SR frameworks. Extensive experiments demonstrate the effectiveness of AESOP. Figure 1 : Conceptual illustration of the proposed AESOP loss and the pixel-level \mathcal{L}_{\text{p}} reconstruction guidance employed in typical perceptual SR methods. (a) Fidelity oriented SR network trained with \mathcal{L}_{\text{pix}} estimates the average over plausible solutions (i.e., the optimal fidelity point). Meanwhile, perceptual SR involves a range of multiple solutions, standing around the optimal fidelity point. Thus, we identify two fundamental components of a perceptual SR image as 1) the perceptual variance factor (red line), a factor that possesses randomness and contributes to realistic textures, and 2) the fidelity bias term (orange dot), the residual blurry component of an SR image, contributing to the overall fidelity, apart from the perceptual variance. (b) Typical perceptual SR methods adopt \mathcal{L}_{\text{pix}} for reconstruction guidance, which pushes the perceptual variance factor to vanish. Thus, when combined with perceptual quality oriented losses that encourage this variance factor, conflict arises, leading to suboptimal performance. (c) In contrast, \mathcal{L}_{\text{AESOP}} only penalizes the fidelity bias-induced error, while preserving these critical perceptual variance factors. This ensures improved fidelity without sacrificing perceptual quality. 1 Introduction Image Super-Resolution is a fundamental challenge in image processing, where the goal is to reconstruct an unknown high-resolution (HR) image from its low-resolution (LR) counterpart. Recent advances in this field have branched into two distinct mainstreams; fidelity-oriented SR and perceptual quality oriented SR (perceptual SR). Fidelity-oriented SR methods aim for high fidelity towards the HR image on a pixel-wise basis. These methods generally adopt the per-pixel reconstruction loss \mathcal{L}_{\text{pix}} (i.e., pixel-level \mathcal{L}_{\text{p}} loss), thereby regressing the unique point that minimizes the expected error. This unique point of minimum expected error is known as the average point over multiple plausible solutions Lee and Heo 2024 , which we will refer to as the optimal fidelity point (brown dot in Fig. 1 .(a)) throughout this work. Another branch of research is the perceptual SR task where the emphasis is on generating visually plausible SR images rather than mere minimum pixel-wise error. Notably, the inherent ill-posedness leads perceptual SR to exhibit a range of variant realistic solutions (multiple purple points in Fig. 1 .(a)), each pivotal to the aforementioned optimal fidelity point. Here, the representative framework in perceptual SR is the SRGAN-based Ledig et al. 2017 framework, which utilizes perceptual quality oriented losses Wang et al. 2018 ; Johnson et al. 2016 together with \mathcal{L}_{\text{pix}} , the de facto training scheme. Yet, since these methods rely on \mathcal{L}_{\text{pix}} as their fidelity loss term, they cannot avoid the blurring phenomenon as shown in the perception-distortion (PD) trade-off Blau and Michaeli 2018 . To address this, they either apply a small coefficient Ledig et al. 2017 ; Wang et al. 2018 to \mathcal{L}_{\text{pix}} or use low-pass filters (LPF) Zhang et al. 2022b ; Deng et al. 2019 before calculating \mathcal{L}_{\text{pix}} . However, we point out that these circumventions result in suboptimal performance, as they misinterpret the implications of \mathcal{L}_{\text{pix}} and fail to distinguish between factors that cause blurring and those that do not. Accordingly, this work revisits \mathcal{L}_{\text{pix}} and aims to correctly analyze the implications of it, by introducing two key factors of an SR image: 1) the perceptual variance factor, and 2) the fidelity bias factor. Here, the perceptual variance is a necessary variance that captures realistic textures and fine details (red line in Fig. 1 .(a)). Meanwhile, the fidelity bias is the residual component of the SR image, apart from the perceptual variance factor. This can be understood as the blurry average solution without the fine-grained texture that possesses randomness (no variance, Fig. 3 .(d)), or can also be understood as the centroid of the distribution where the SR image was originally expected to be sampled from (orange dot in Fig. 1 .(a)). For an optimal SR image (or the HR), the fidelity bias of itself is the optimal fidelity point. In terms of these two key components defined above, we will show that \mathcal{L}_{\text{pix}} is identical to minimizing both the fidelity bias induced errors and the perceptual variance factor. However, while minimizing the fidelity bias induced error is an intended aspect of \mathcal{L}_{\text{pix}} , vanishing perceptual variance is not suitable in perceptual SR. Specifically, when the perceptual variance factor is minimized, the prediction space degenerates and the SR image converges to the blurry average image as in Fig. 1 .(b). Meanwhile, perceptual quality oriented losses aim to preserve this necessary perceptual variance, which indicates a conflict against \mathcal{L}_{\text{pix}} . Consequently, as long as the SR network receives reconstruction guidance from \mathcal{L}_{\text{pix}} , the perceptual quality oriented losses cannot converge to an optimal point, limiting the visual quality. Now, we focus on the other counterpart of the SR image, the fidelity bias, and the error induced by it. The fidelity bias induced error indicates the overall degree of misalignment between the prediction space and the solution space. Accordingly, reducing the fidelity bias induced error is identical to aligning the centroids of the prediction space and the solution space, without altering the range of the prediction space (i.e., preserving perceptual variance), as in Fig. 1 .(c). Thus, by reducing the fidelity bias induced error, we can achieve improved fidelity without degrading perceptual variance, which is a highly desired aspect for optimal reconstruction guidance in the perspective of perceptual SR. Accordingly, this motivates us to design a novel reconstruction loss that can precisely discriminate the fidelity bias and the perceptual variance, and solely penalize based on the fidelity bias term. Notably, we will show that this can be surprisingly simplified with a pretrained Auto-Encoder (AE). We will pretrain an AE with \mathcal{L}_{\text{pix}} , and take \mathcal{L}_{\text{p}} in the AE space instead of raw pixel space. Since \mathcal{L}_{\text{pix}} removes perceptual variances, loss in the AE space will enable us to penalize solely based on fidelity biases. Importantly, we are conversely taking advantage of the vanishing perceptual variance phenomenon