Dual Prompting Image Restoration with Diffusion Transformers
Dehong Kong, Fan Li, Zhixin Wang, Jiaqi Xu, Renjing Pei, Wenbo Li, WenQi Ren · 2025
Read Dual Prompting Image Restoration with Diffusion Transformers by Dehong Kong, Fan Li, Zhixin Wang, Jiaqi Xu, Renjing Pei, Wenbo Li, WenQi Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17825.txt.
Introduction
Dehong Kong and co-authors present DPIR, a method for restoring damaged images with diffusion transformers. Their 2025 paper notes that most top restorers still use older U-Net designs inside diffusion models. Transformer versions promise better quality and scaling, but they need the right guidance from the damaged input.
DPIR feeds the model two kinds of prompts at once. One branch adds image cues through a light module, while a second branch builds visual prompts that capture both the whole scene and fine local detail. Text prompts join these visual ones, giving dual control. Wide tests showed restoration quality above rival methods, the authors report.
Catalog record
| CREATOR | Dehong Kong, Fan Li, Zhixin Wang, Jiaqi Xu, Renjing Pei, Wenbo Li, WenQi Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Prompting Image Restoration with Diffusion Transformers Dehong Kong, Fan Li, Zhixin Wang, Jiaqi Xu, Renjing Pei, Wenbo Li, WenQi Ren Abstract † † ∗ Contribute Equally. † Corresponding Author. Recent state-of-the-art image restoration methods mostly adopt latent diffusion models with U-Net backbones, yet still facing challenges in achieving high-quality restoration due to their limited capabilities. Diffusion transformers (DiTs), like SD3, are emerging as a promising alternative because of their better quality with scalability. In this paper, we introduce DPIR (Dual Prompting Image Restoration), a novel image restoration method that effectivly extracts conditional information of low-quality images from multiple perspectives. Specifically, DPIR consits of two branches: a low-quality image conditioning branch and a dual prompting control branch. The first branch utilizes a lightweight module to incorporate image priors into the DiT with high efficiency. More importantly, we believe that in image restoration, textual description alone cannot fully capture its rich visual characteristics. Therefore, a dual prompting module is designed to provide DiT with additional visual cues, capturing both global context and local appearance. The extracted global-local visual prompts as extra conditional control, alongside textual prompts to form dual prompts, greatly enhance the quality of the restoration. Extensive experimental results demonstrate that DPIR delivers superior image restoration performance. 1 Introduction The persistent demand for high-quality images across diverse fields, such as digital art and medical imaging, drives research in image restoration and super-resolution. Image restoration (IR) aims to reconstruct high-quality (HQ) images from low-quality (LQ) inputs, addressing real-world degradations of varying complexity, such as noise, blur, and compression artifacts. Recently, diffusion models-based methods [ 45 , 10 , 47 , 42 , 49 , 48 ] have shown the advantages of leveraging the powerful pre-trained text-to-image (T2I) models to enhance the performance of real-world IR. These existing IR methods rely on latent diffusion models [ 35 ] with the U-Net [ 36 ] architecture, which, while effective, still encounter challenges in restoration quality. As an alternative, diffusion transformers [ 30 ] (DiTs) recently show promising generative capabilities due to their ability to capture long-range dependencies and their scalability potential, which are crucial for the quality and fidelty of restoration across a wide range of real-world situations. Despite these advantages, effectively incorporating LQ information into DiTs remains underexplored. Existing IR methods [ 21 , 52 ] based on the U-Net backbone typically utilize ControlNet [ 58 ] , which incorporates the LQ image by employing a trainable copy of the U-Net’s encoding layers. In parallel, StableSR [ 42 ] adopts a lightweight trainable control network, akin to the T2I Adaptor [ 29 ] , but suffers from moderate performance. However, these conditional control methods are not optimized for DiTs composed of ViT [ 6 ] blocks. This discrepancy necessitates a careful design of conditioning mechanisms for DiTs in the IR task. In this work, we present a dual prompting image restoration model (DPIR), combined with a DiT-based diffusion model Stable Diffusion 3 (SD3) [ 8 ] . We propose several key techniques in DPIR to effectively integrate control signals from LQ images into the DiT, enabling natural and faithful image restoration. DPIR includes a low-quality image conditioning branch and a dual prompting control branch, which balances model efficiency and effectiveness. Specifically, the first branch is a lightweight module designed to efficiently inject the LQ image prior into the DiT. Inspired by ControlNeXt [ 31 ] , this branch extracts conditional features of LQ images by a few of convolution layers rather than heavy backbone copies like ControlNet. Notably, the lightweight module and the DiT backbone are jointly trained to handle diverse real-world scenarios. In IR tasks, visual cues of LQ images, including the global context information and the local texture infomation, are crucial for desirable image restoration. These visual signals cannot be fully captured by text descriptions alone, especially considering that the DiT backbone, without a skip-connection mechanism like U-Net, hardly keeps the information of input LQ images, which are the conditioning in text-to-image models across all DiT blocks. Hence, we design a dual prompting restoration branch to incorporate LQ information into DiT from a more comprehensive visual perspective, leveraging the knowledge of pre-trained T2I models and complementing our lightweight conditioning branch. In detail, given a cropped patch of the LQ image, we first feed it into CLIP [ 33 ] to extract local visual features. Additionally, global visual features from the surrounding regions, rich in contextual information, are also extracted. These global-local features function as visual tokens, replacing the original CLIP text features in SD3 and serving as conditioning, alongside T5 [ 34 ] text prompts as visual-text dual prompts, for each block of DiT. As shown in Figure , DPIR with the dual prompting strategy demonstrates excellent restoration performance and shows a clear advantage over the simple text prompts as DiT conditioning. Additionally, to further exploit the scalability of DiTs, we use over 20 million high-quality images as training data. Experimental results validate the effectiveness of the proposed DPIR, showing compelling image restoration performance compared to recent state-of-the-art methods and achieving impressive results across a broad range of image restoration tasks. Figure 2 : Framework of our proposed DPIR. Given a low-quality (LQ) image, a lightweight conditioning branch efficiently introduces the LQ information into the DiT backbone. Additionally, a dual prompting restoration branch extracts global and local visual information, alongside text prompts, to form visual-text dual prompts, which greatly enhances the restoration quality and fidelity. 2 Related Work Image Restoration. IR aims to recover high-quality, degradation-free images from degraded inputs [ 9 , 13 , 55 ] . Early research focus on addressing specific types of image degradation, such as denoising, deblurring, and super-resolution (SR). Since the pioneering work of SRCNN [ 5 ] , numerous deep learning-based approaches [ 3 , 19 , 37 , 56 , 14 , 54 , 60 , 20 , 59 ] , have been developed. Recently, generative model-based IR methods are evolving, including GAN-based and diffusion-based methods, to tackle real-world IR tasks involving blind and complex degradations. DACLIP [ 27 ] transfers pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. GAN-based IR. The application of GANs for IR or SR dates back to SRGAN [ 18 ] , where image degradation is modeled using bicubic downsampling. Subsequently, BSRGAN [ 57 ] and Real-ESRGAN [ 43 ] show that IR GANs can handle more complex image degradations by synthesizing more realistic HQ-LQ pairs, showing convincing results in real-world IR. Diffusion-based IR. Early diffusion-based works [ 15 , 16 , 44 ] use pre-trained diffusion models [ 12 , 39 , 4 ] to tackle the IR problem with simple degradation modeling. Recent approaches [ 42 , 48 , 51 , 21 , 52 , 26 ] use advanced pre-trained T2I models, such as Stable Diffusion [ 35 ] , for real-world image restoration. These methods typically introduce adapters to incorporate LQ images as control signals into diffusion models to reconstruct HQ images. StableSR [ 42 ] redesigns a time-aware encoder with LQ embedding. SUPIR [ 52 ] capitalizes on the generative capability of SDXL [ 32 ] and the captioning strength of LLaVA [ 23 ] to synthesize