Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach
Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, Lei Zhang · 2024
Read Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach by Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, Lei Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03017.txt.
Introduction
Lingchen Sun and five colleagues present PiSA-SR, an image super-resolution method that lets users trade pixel fidelity against semantic detail at inference time. Diffusion-based methods usually entangle those two goals during training, forcing one fixed balance. Different users, however, prefer different balances, and retraining for each is impractical.
The method learns two separate low-rank adapter modules on a pre-trained diffusion model. One adapter handles pixel-level regression; the other extracts semantic information using perceptual and classifier-based losses. Two guidance scales then adjust each module's strength during inference. In its default setting the method runs in a single diffusion step with leading quality and efficiency on real-world benchmarks.
Catalog record
| CREATOR | Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, Lei Zhang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03017
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, Lei Zhang Abstract Diffusion prior-based methods have shown impressive results in real-world image super-resolution (SR). However, most existing methods entangle pixel-level and semantic-level SR objectives in the training process, struggling to balance pixel-wise fidelity and perceptual quality. Meanwhile, users have varying preferences on SR results, thus it is demanded to develop an adjustable SR model that can be tailored to different fidelity-perception preferences during inference without re-training. We present Pi xel-level and S emantic-level A djustable SR ( PiSA-SR ), which learns two LoRA modules upon the pre-trained stable-diffusion (SD) model to achieve improved and adjustable SR results. We first formulate the SD-based SR problem as learning the residual between the low-quality input and the high-quality output, then show that the learning objective can be decoupled into two distinct LoRA weight spaces: one is characterized by the ℓ 2 \ell_{2} -loss for pixel-level regression, and another is characterized by the LPIPS and classifier score distillation losses to extract semantic information from pre-trained classification and SD models. In its default setting, PiSA-SR can be performed in a single diffusion step, achieving leading real-world SR results in both quality and efficiency. By introducing two adjustable guidance scales on the two LoRA modules to control the strengths of pixel-wise fidelity and semantic-level details during inference, PiSA-SR can offer flexible SR results according to user preference without re-training. The source code of our method can be found at https://github.com/csslc/PiSA-SR . 1 Introduction Figure 1 : Visual illustration of our pixel- and semantic-level adjustable method for real-world SR. By increasing the pixel-level guidance scale \lambda_{pix} , the image degradations such as noise and compression artifacts can be gradually removed; however, a too-strong \lambda_{pix} will make the SR image over-smoothed. By increasing the semantic-level guidance scale \lambda_{sem} , the SR images will have more semantic details; nonetheless, a too-high \lambda_{sem} will generate visual artifacts. Please zoom in for a better view. Single image super-resolution (SR) Wang et al. 2021c aims to reconstruct a high-quality (HQ) image from its low-quality (LQ) counterpart suffering from various degradations, e.g . , noise, blur, downsampling, etc . SR is a challenging ill-posed problem due to the inherent ambiguity in reconstructing details Blau and Michaeli 2018 ; Liang et al. 2022b , and various deep neural networks (DNNs) Chao et al. 2014 ; Haris et al. 2018 ; Kim et al. 2016 ; Liang et al. 2021 ; Zhang et al. 2022a ; Chen et al. 2023 have been developed to address it. The pixel-level regression losses, such as \ell_{1} and \ell_{2} losses, are essential to keep the pixel-level fidelity of SR outputs, yet they tend to produce over-smoothed details Blau and Michaeli 2018 ; Liang et al. 2022b . The SSIM Wang et al. 2004 and perceptual losses can alleviate this problem. In specific, SSIM measures the local structural similarity between the SR and ground-truth (GT) images, and the perceptual loss Johnson et al. 2016 extracts semantic features with a pre-trained classification model Simonyan and Zisserman 2014 to improve the perceptual quality of SR results. The adversarial loss associated with the generative adversarial network (GAN) Ledig et al. 2017 ; Wang et al. 2018 ; Wang et al. 2021b ; Liang et al. 2022b ; Liang et al. 2022a provides a more effective solution than SSIM/perceptual losses to align SR images with natural image distribution, thereby leading to perceptually more realistic SR results. However, it simultaneously introduces many undesirable visual artifacts Liang et al. 2022b because the adversarial training is unstable and the limited capacity of GAN models in characterizing the space of natural images. Recently developed large-scale stable diffusion (SD) models Rombach et al. 2022 , which are pre-trained on text-to-image (T2I) tasks, have demonstrated impressive semantic understanding and have been successfully applied to many downstream tasks Zhang et al. 2023a ; Mou et al. 2023 , including SR Wang et al. 2023b ; Lin et al. 2023 ; Yang et al. 2023 ; Wu et al. 2024b ; Wu et al. 2024a ; Sun et al. 2023 . While SD-based SR approaches have exhibited perceptually more realistic SR results than GAN-based methods Chen et al. 2025 , they commonly entangle the pixel-level fidelity and semantic-level enhancement objectives in the diffusion process Wang et al. 2023b ; Wu et al. 2024b ; Wu et al. 2024a , which can be contradictory in optimization Blau and Michaeli 2018 ; Sun et al. 2024 . As a result, they struggle to balance the pixel-wise fidelity and semantic-level perception in the final SR output. While some methods Lin et al. 2023 ; Yang et al. 2023 ; Sun et al. 2023 perform pixel-level restoration before semantic-level enhancement, the performance of the latter stage depends heavily on the accuracy of the earlier stage. Besides the trade-off between pixel-level fidelity and semantic -level details, users in practice often have different preferences on the SR results: some prioritize content fidelity over detail generation, while others favor semantically rich details over pixel-wise fidelity. This diversity in user preferences underscores the importance and demands for a more flexible SR approach that can accommodate individual tastes during inference. Although some previous methods He et al. 2021 ; Mou et al. 2022 have explored interactive SR methods to control the restoration strength, they are primarily limited to degradation-level adjustments (such as noise and blur), resulting in either smoother or sharper outputs. Several multi-step SD-based SR methods Wu et al. 2024b ; Yang et al. 2023 ; Lin et al. 2023 ; Yu et al. 2024a employ guidance-based strategies Ho and Salimans 2022 at each sampling step to achieve varying levels of semantic richness, but these methods often struggle with precise control and efficiency. In this paper, we present Pi xel-level and S emantic-level A djustable S uper- R esolution, namely PiSA-SR , which separates pixel-level and semantic-level enhancement into two distinct Low-Rank Adapter (LoRA) Hu et al. 2021 weight spaces with the pre-trained SD model, providing an effective SR model for various user preferences. We begin by formulating the SD-based SR problem to learn the residual between LQ and HQ latent features He et al. 2016 ; Zhang et al. 2017 ; Zhang et al. 2018b . With this formulation, we can not only accelerate the convergence of model training but also introduce scale factors on the model outputs, enabling flexible adjustment on the SR results during the inference stage without re-training. Then, we fine-tune the pre-trained SD model by introducing two separate LoRA modules dedicated to pixel-level regression and semantic-level enhancement. In specific, the \ell_{2} loss is employed for the LoRA module of pixel-level regression, while the LPIPS Zhang et al. 2018a and classifier score distillation (CSD) Yu et al. 2024b losses are employed for another LoRA module, leveraging the semantic priors encoded in the pre-trained VGG classification model Simonyan and Zisserman 2014 and SD image generation model Rombach et al. 2022 . A decoupled training approach is proposed to train pixel- and semantic-level LoRAs, effectively improving semantic information while maintaining pixel-level fidelity. Our experiments demonstrate that PiSA-SR not only achieves superior SR performance to existing SD-based models but also