Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration
Xingyu Jiang, Ning Gao, Hongkun Dou, Xiuhui Zhang, Xiaoqing Zhong · 2025
Read Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration by Xingyu Jiang, Ning Gao, Hongkun Dou, Xiuhui Zhang, Xiaoqing Zhong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13663.txt.
Introduction
Xingyu Jiang leads a team in a 2025 paper arguing that global modeling matters for fast, light image restoration. Rain, haze, and snow dull real photos and hurt the vision tools that read them. Transformer models clean images well, but their growing size and cost block real-time use.
Most speed-ups trim the attention step and leave the rest of the design alone. The team instead start from what restoration itself needs: a view of the whole scene, not just near pixels. Their baseline keeps that global view while staying small and quick. The design, they show, restores weather-worn images well at a fraction of the usual cost.
Catalog record
| CREATOR | Xingyu Jiang, Ning Gao, Hongkun Dou, Xiuhui Zhang, Xiaoqing Zhong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration Thanks: Xingyu Jiang, Ning Gao, Hongkun Dou, Xiuhui Zhang, Yue Deng and Hongjue Li are with Beihang University, Beijing 100191, China; Xiaoqing Zhong is with the China Academy of Space Technology. Xingyu Jiang, Ning Gao, Hongkun Dou, Xiuhui Zhang, Xiaoqing Zhong Abstract Natural image quality is often degraded by adverse weather conditions, significantly impairing the performance of downstream tasks. Image restoration has emerged as a core solution to this challenge and has been widely discussed in the literature. Although recent transformer-based approaches have made remarkable progress in image restoration, their increasing system complexity poses significant challenges for real-time processing, particularly in real-world deployment scenarios. To this end, most existing methods attempt to simplify the self-attention mechanism, such as by channel self-attention or state space model. However, these methods primarily focus on network architecture while neglecting the inherent characteristics of image restoration itself. In this context, we explore a pyramid Wavelet-Fourier iterative pipeline to demonstrate the potential of Wavelet-Fourier processing for image restoration. Inspired by the above findings, we propose a novel and efficient restoration baseline, named Pyramid Wavelet-Fourier Network (PW-FNet). Specifically, PW-FNet features two key design principles: 1) at the inter-block level, integrates a pyramid wavelet-based multi-input multi-output structure to achieve multi-scale and multi-frequency bands decomposition; and 2) at the intra-block level, incorporates Fourier transforms as an efficient alternative to self-attention mechanisms, effectively reducing computational complexity while preserving global modeling capability. Extensive experiments on tasks such as image deraining, raindrop removal, image super-resolution, motion deblurring, image dehazing, image desnowing and underwater/low-light enhancement demonstrate that PW-FNet not only surpasses state-of-the-art methods in restoration quality but also achieves superior efficiency, with significantly reduced parameter size, computational cost and inference time. The code is available at: https://github.com/deng-ai-lab/PW-FNet. I Introduction Natural image quality deteriorates significantly under adverse weather conditions, severely impacting the performance of subsequent high-level vision tasks. This degradation not only reduces visibility but also disrupts critical feature extraction, which poses challenges for tasks such as object detection [ 1 ] , segmentation [ 2 ] , and tracking [ 3 ] . As a result, image restoration has been proposed as a preprocessing solution aimed at restoring degraded images to their clean counterparts, drawing considerable attention from researchers. However, due to the inherently ill-posed nature of image restoration, traditional prior-based methods [ 4 , 5 ] struggle to handle complex real-world scenarios effectively, as their predefined constraints often fail to generalize across diverse degradation patterns and background textures. In recent years, deep learning-based approaches [ 6 , 7 , 8 ] have achieved remarkable success. Specifically, convolutional neural networks (CNNs), as a type of local perceptual network, have demonstrated strong restoration capabilities through end-to-end learning. More recently, transformer-based models [ 9 , 10 , 11 ] have significantly advanced the field by leveraging self-attention mechanisms to incorporate global contextual information, surpassing CNN-based restoration performance in challenging conditions. Fig. 1: Comparison with different state-of-the-art approaches on the Rain200L [ 12 ] dataset: (a) FLOPs vs PSNR; (b) Parameters vs PSNR; (c) Inference time vs PSNR; (d) FPS vs PSNR. Our method achieves better performance and all calculations are performed at a resolution of 256\times 256 . Fig. 2: The pyramid wavelet-Fourier iterative pipeline for image restoration. Rain degradation is concentrated on HL sub-bands. Despite above advancements, due to the quadratic computational complexity of self-attention, transformer-based models [ 10 , 11 ] require high system complexity (see Fig. 1 (a) and Fig. 1 (b)), which limits their practical applicability in real-time or resource-constrained environments (see Fig. 1 (c) and Fig. 1 (d)), such as autonomous driving [ 13 ] . Consequently, existing methods predominantly focus on simplifying self-attention mechanisms in two primary ways: 1) simplifying the self-attention mechanism itself , where various methods aim to restructure the self-attention computation to reduce complexity. Notable examples include Restormer, which introduces channel self-attention, and GRL, which incorporates anchor self-attention; 2) replacing the self-attention mechanism , inspired by the Metaformer concept. For instance, NAFNet replaces self-attention with convolution, while the latest Mamba-based methods substitute self-attention with state space models. We follow the latter manner, as we believe the key to the success of transformers in image restoration lies in their global modeling capability, with self-attention being merely one possible avenue for achieving this. In this context, rather than exclusively focusing on model architecture design, we investigate efficient global modeling strategies grounded in the inherent priors of the image restoration tasks themselves to develop a lightweight and effective restoration baseline. To this end, we present the pyramid wavelet-Fourier iterative pipeline, without any deep learning techniques, to efficiently separate the groundtruth and degraded components in the Wavelet-Fourier domain, as illustrated in Fig. 2 . Taking deraining as an example, in Level-1 of Fig. 2 , we begin by applying wavelet transform (WT) to the input rainy and clear images, decomposing them into four sub-bands: [LL, LH, HL, HH]. Subsequently, we apply fast Fourier transform (FFT) to each sub-band. The Fourier transform’s global operation reveals that degradation primarily concentrates in the compact high-frequency regions of the [LL, HL] sub-bands. In Level-2, we take the LL sub-band from Level-1 as input and repeat the process, observing a similar pattern: the degradation differences remain localized in the high-frequency regions of the [LL, HL] sub-bands. This iterative decomposition continues until the resolution of the LL sub-band becomes sufficiently small, at which point we swap the HL sub-band between the rainy and clear images. The derained image is then reconstructed through inverse Fourier and inverse wavelet transforms. Additionally, we present the quantitative results of swapping different sub-bands in the bottom-right corner of Fig. 2 . This wavelet-Fourier pipeline highlights two key observations: 1) wavelet iterative decomposition effectively isolates degradation components into specific sub-band, thereby reducing the feature processing space; 2) Fourier transform’s global operation further concentrates degradation into compact high-frequency regions, narrowing the feature extraction scope. By precisely localizing degradation-related features, the Wavelet-Fourier pipeline demonstrates its potential for achieving effective global restoration capabilities while maintaining computational efficiency. Building upon the aforementioned findings, we propose a powerful and efficient restoration baseline, the Pyramid Wavelet-Fourier Network (PW-FNet), which integrates pyramid wavelet-Fourier transforms into the model architecture to achieve efficient image restoration. Specifically, at the inter-block level, we incorporate a pyramid wavelet multi-input multi-output structure, enabling multi-scale and multi-frequency bands decomposition, while allowing for dynamic adaptive output. This design enables a single trained model to generate small, medium and large versions without additional training. At the intra-block level, we replace self-attention mechanisms with Fourier transforms, enabling efficient global feature modeling while significantly reducing network complexity. To validate the effectiveness of PW-FNet, we conduct extensive experiments on a range of image restoration tasks, including deraining, raindrop removal, super-resolution, motion deblurring, dehazing, desnowing, and underwater/low-light enhancement and demonstrate that PW-FNet not only outperforms state-of-the-art methods in restoration quality but also achieves superior efficiency, with significantly reduced parameter size, computational cost and inference time. We summarize the mian contributions as three-folds: \bullet We show the pyramid Wavelet-Fourier iterative pipeline, which effectively decomposes degradation into distinct sub-bands through wavelet transforms and isolates compact feature regions using Fourier transforms. \bullet We propose a powerful and efficient restoration baseline, which replaces