UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior

I-Hsiang Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang · 2025

Read UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior by I-Hsiang Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13134.txt.

Introduction

I-Hsiang Chen and co-authors unify photo repair in a 2025 paper. Restoring damaged images splits into two camps: making pictures look good to human eyes, and making them useful for machine tasks like detection. Diffusion-based repair pleases eyes but often fails the machines.

Their model adapts a diffusion prior with features drawn from an autoencoder. One module rebuilds the damaged features, and another fuses task cues into the decoding stage. The same system can thus serve both beauty and utility. Wide tests show top results on both fronts. The authors present the design as a flexible base for many repair jobs.

Catalog record

CREATORI-Hsiang Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior I-Hsiang Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang Abstract Image restoration aims to recover content from inputs degraded by various factors, such as adverse weather, blur, and noise. Perceptual Image Restoration (PIR) methods improve visual quality but often do not support downstream tasks effectively. On the other hand, Task-oriented Image Restoration (TIR) methods focus on enhancing image utility for high-level vision tasks, sometimes compromising visual quality. This paper introduces UniRestore, a unified image restoration model that bridges the gap between PIR and TIR by using a diffusion prior. The diffusion prior is designed to generate images that align with human visual quality preferences, but these images are often unsuitable for TIR scenarios. To solve this limitation, UniRestore utilizes encoder features from an autoencoder to adapt the diffusion prior to specific tasks. We propose a Complementary Feature Restoration Module (CFRM) to reconstruct degraded encoder features and a Task Feature Adapter (TFA) module to facilitate adaptive feature fusion in the decoder. This design allows UniRestore to optimize images for both human perception and downstream task requirements, addressing discrepancies between visual quality and functional needs. Integrating these modules also enhances UniRestore’s adaptability and efficiency across diverse tasks. Extensive experiments demonstrate the superior performance of UniRestore in both PIR and TIR scenarios. † † Project page: https://unirestore.github.io/ † † * Indicates equal contribution. 1 Introduction Image restoration [ 55 , 42 ] aims to restore content degraded by various factors, such as adverse weather, blur, and noise. These factors often reduce image perceptual visibility [ 60 , 28 ] and negatively impact the performance of high-level vision applications, such as object detection [ 73 , 66 ] and semantic segmentation [ 49 , 43 ] . Various restoration methods have been developed and studied over the past decades to address this wide range of image restoration challenges. These include methods focused on improving perceptual quality [ 44 , 72 ] as well as those designed to enhance the performance of downstream tasks [ 52 , 69 ] . (a) Perceptual IR (b) Task-oriented IR (c) UniRestore Figure 1 : Illustration of UniRestore’s capabilities: (a) PIR - Comparison with existing methods ( e.g . , URIE [ 52 ] , PromptIR [ 42 ] ) under adverse conditions. (b) TIR - Demonstrating UniRestore’s robustness for downstream tasks such as classification and segmentation. (c) Unified Restoration - UniRestore’s versatility in addressing PIR and TIR. Perceptual image restoration (PIR) algorithms focus on improving visual clarity and fidelity of images by removing or reducing visible artifacts that affect their aesthetic quality. These methods focus on dealing with noise reduction [ 72 , 8 ] , low-light enhancement [ 70 , 75 ] , deblurring [ 38 , 44 ] , dehazing [ 6 , 72 ] , deraining [ 57 , 67 ] , and desnowing [ 59 , 7 , 35 ] . While these algorithms can enhance the visual quality of images, they do not always improve performance in downstream tasks [ 21 , 4 , 5 ] . This is because the factors contributing to visual quality often differ from those determining recognition quality [ 52 , 69 , 41 ] . On the other hand, task-oriented image restoration (TIR) [ 52 , 69 ] is specifically designed to optimize images for applications that rely on computer vision, such as object detection [ 73 , 66 ] , classification [ 51 , 52 ] , and autonomous driving systems [ 49 , 43 ] . This approach aligns the restoration process with the specific requirements of neural network models used in these applications, ensuring that the restored images are suitable for effective machine interpretation. Although these methods improve the performance of downstream tasks, they often produce results that are less visually appealing [ 73 , 34 ] . Based on the analysis above, existing image restoration algorithms often face a trade-off, highlighting the challenge of balancing technical functionality with aesthetic quality. This dual functionality is crucial because real-world applications often require restoration processes to be specifically adapted to different scenarios. Thus, developing a model that can simultaneously enhance perceptual quality and improve performance for downstream tasks is essential. Such a unified image restoration framework can effectively perform across diverse settings, reducing system redundancy and boosting operational efficiency, as shown in Figure 1 . In this paper, we propose a unified image restoration paradigm, UniRestore, which simultaneously improves the performance of downstream tasks and the human perceptual quality of degraded images. UniRestore leverages a diffusion prior [ 45 ] as the backbone, recognized for its generative capabilities in producing high-quality images. However, these images are typically optimized for human aesthetics, which may not align with downstream task requirements. UniRestore addresses this limitation by adapting the diffusion prior to meet both perceptual and functional needs, enabling the model to effectively bridge the gap between visual quality and downstream task performance. To bridge this gap, we exploit the encoder features from the autoencoder within the diffusion model as complementary elements to tailor the diffusion prior to specific tasks. We introduce a Complementary Feature Restoration Module designed to reconstruct degraded features in the encoder. Subsequently, we propose a Task Feature Adapter, which harmonizes the diffusion features with the restored features within the decoder for various downstream tasks. Given the diversity of downstream tasks and the frequent necessity to adapt these tasks within existing models, the TFA module offers extendability to accommodate new tasks. Extensive experiments validate UniRestore’s effectiveness, demonstrating enhancements in both PIR and TIR, with the potential for expansion to additional downstream tasks. The contributions of this work are: • We introduce UniRestore, a unified image restoration model that addresses both perceptual image restoration and task-oriented image restoration within a single framework. Experimental results show that UniRestore surpasses existing methods in both visual quality and downstream task performance. • We propose two components for UniRestore: the CFRM and the TFA. These modules work together to adaptively complement the diffusion prior, enabling simultaneous restoration across diverse tasks. 2 Related Work 2.1 Perceptual Image Restoration Perceptual image restoration aims to enhance the visual quality of images as perceived by humans, and it can be categorized into single degradation and multiple degradation restoration. Early work addressed single degradation such as denoising [ 72 , 8 ] , dehazing [ 13 , 6 , 72 ] , deraining [ 57 , 67 ] , low-light enhancement [ 70 , 75 ] , and deblurring [ 38 , 44 ] . To tackle multiple degradations, methods like MPRNet [ 72 ] and NAFNet [ 2 ] introduced unified solutions, while transformer-based approaches such as SwinIR [ 29 ] and Restormer [ 71 ] gained traction for their versatility. Additionally, holistic approaches like All-in-One [ 28 ] , TransWeather [ 55 ] and PromptIR [ 42 ] focused on improving visual quality across a wide range of conditions while providing enhanced adaptability and performance. While these methods excel in enhancing visual quality, they do not always improve downstream vision tasks. 2.2 Task-oriented Image Restoration Task-oriented image restoration aims to enhance downstream task performance, as studies [ 25 , 4

Read the complete text