FourierSR: A Fourier Token-based Plugin for Efficient Image Super-Resolution

Wenjie Li, Heng Guo, Yuefeng Hou, Zhanyu Ma · 2025

Read FourierSR: A Fourier Token-based Plugin for Efficient Image Super-Resolution by Wenjie Li, Heng Guo, Yuefeng Hou, Zhanyu Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10043.txt.

Introduction

Wenjie Li and co-authors propose a plug-in that improves image super-resolution at very low extra cost, published in 2025. Super-resolution rebuilds a sharp image from a blurry, low-resolution one. Common building blocks, such as convolutions and window-based Transformers, see only a limited area at once, which limits gains under tight compute budgets.

Their plug-in, FourierSR, uses Fourier tokens instead of those blocks. It relies only on Fourier transforms and multiplication, which cuts complexity while giving a global view of the image. Added to existing efficient methods, it raised a standard quality score by a clear margin on a test set. The added parameters and compute were a tiny fraction of the original models' size, the authors report.

Catalog record

CREATORWenjie Li, Heng Guo, Yuefeng Hou, Zhanyu Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FourierSR: A Fourier Token-based Plugin for Efficient Image Super-Resolution Wenjie Li, Heng Guo, Yuefeng Hou, Zhanyu Ma Abstract Image super-resolution (SR) aims to recover low-resolution images to high-resolution images, where improving SR efficiency is a high-profile challenge. However, commonly used units in SR, like convolutions and window-based Transformers, have limited receptive fields, making it challenging to apply them to improve SR under extremely limited computational cost. To address this issue, inspired by modeling convolution theorem through token mix, we propose a Fourier token-based plugin called FourierSR to improve SR uniformly, which avoids the instability or inefficiency of existing token mix technologies when applied as plug-ins. Furthermore, compared to convolutions and windows-based Transformers, our FourierSR only utilizes Fourier transform and multiplication operations, greatly reducing complexity while having global receptive fields. Experiments show that our FourierSR as a plugin brings an average PSNR gain of 0.34dB for existing efficient SR methods on Manga109 test set at the scale of × 4 \times 4 , while the average increase in the number of Params and FLOPs is only 0.6% and 1.5% of original sizes. Code link: https://github.com/PRIS-CV/FourierSR . I Introduction IMAGE super-resolution (SR) aims to reconstruct a low-resolution (LR) image to a high-resolution (HR) image, which plays an important role in computer vision, such as visual quality improvement [ 1 ] and security surveillance [ 2 ] . To promote the application of SR on devices with limited computational resources, efficient SR has been continuously studied. However, existing methods face a trade-off between performance and computational cost. Therefore, a plug-and-play unit with the ability to efficiently enhance performance while consuming minimal computational costs is desired. The challenge of the plug-and-play unit for efficient SR lies in balancing complexity and performance. Existing CNN-based methods utilize techniques such as feature distillation [ 3 ] and wide activation [ 4 ] to reduce model complexity, of which 3 \times 3 convolutions are widely used. However, compared to the local receptive fields from convolution, larger receptive fields are preferred for improving the performance of SR. Transformer can be one solution for obtaining global receptive fields, but it also introduces high computational costs. Existing Transformer-based efficient SR methods [ 5 ] reduce the complexity by dividing inputs into small windows. Due to limited connections between different windows, the receptive field of models is limited. Therefore, this paper aims to design an efficient SR plugin with low costs and global receptive fields. (a) Comparisons of PSNR on Manga109 [ ] test set. (b) Comparisons of the number of Params and FLOPs at the scale of . [formula] (c) Comparisons of the effective receptive field (ERF) [ ] visualization. Fig. 1 : ( Top ) The performance and efficiency improvement embedded with our FourierSR (-O), a plugin for improving existing efficient SR methods [ 8 , 9 , 10 ] . ( Bottom ) With our FourierSR plugin, the receptive field of existing efficient SR methods [ 8 , 9 , 10 ] can be extend effectively. Inspired by performing token mixing in the Fourier domain to model global convolution via convolution theorem [ 11 ] at low costs in image recognition and detection tasks [ 12 ] , token mix can be one solution for designing efficient SR plugins, where token mix refers to exchange information contained in divided features of inputs. Despite the potential of applying token mix techniques to SR tasks, it is non-trivial to apply the token mix in efficient SR plugins where expensive costs or instability could be introduced. For example, AFNO [ 13 ] utilizes a set of matrix multiplications to mix channel tokens in the Fourier domain, bringing computational costs to be too expensive for efficient SR. GFNet [ 14 ] defines global filters the same size as inputs to perform token mix. However, in the SR task, the input size differs between training and testing, making it unstable to apply this approach consistently. AFFNet [ 15 ] multiplies mixed tokens with Fourier features. However, the multiplication of complex tensors produces unstable values, leading to gradient explosion in SR training. To construct improved Fourier tokens suitable for the efficient SR plugin, we propose FourierSR to boost off-the-shelf efficient SR methods. Similar to existing token mix methods, we simulate the global convolution based on the convolution theorem [ 11 ] , which states that element-wise products in the frequency domain are equivalent to convolutions in the time domain. The key difference is that we find using defined local kernels to perform token mixing on the real and imaginary parts of Fourier features results in more robust and superior performance. Specifically, we extend defined local kernels to obtain simulated global kernels, which allows FourierSR to avoid the size inconsistency caused by defining global filters as kernels for SR training and testing in GFNet [ 14 ] , and the gradient explosion resulting from defining Fourier features as kernels in AFFNet [ 15 ] . Furthermore, we demonstrate that multiplying the real and imaginary parts of Fourier features with defined filters further results in enhanced global receptive fields. Additionally, such operation is lightweight, avoiding the slow inference and large number of parameters associated with heavy computational operations in AFNO [ 16 ] and FFC [ 17 ] . Due to its global receptive field and its absence of heavy computational operations, our FourierSR is ideally suited as a plug-and-play unit to enhance efficient SR methods. Experiments show our FourierSR as a plugin can be easily adapted to off-the-shelf efficient methods and improve their performance while avoiding the significant gain of computational cost. As shown in Fig. 1 (a) and (b), our FourierSR improves PSNR by more than 0.34dB while increasing the average number of parameters by only 0.6% and FLOPs by only 1.5%, which benefits from our FourierSR’s ability to expand the limited receptive field significantly of existing efficient CNN [ 8 , 9 ] and windows-based Transformer [ 10 ] methods, as shown in Fig. 1 (c). Our contributions can be summarized as: • We propose a plugin called FourierSR, which improves off-the-shelf efficient SR by simulating global convolution based on the token mix and convolution theorem. • We propose a token mix scheme that uses local filters to modulate real and imaginary parts of Fourier features, efficiently and robustly achieving global receptive fields. • Experiments show FourierSR can be a plug-and-play unit that efficiently expands receptive fields of existing methods and thus improves SR quality with minimal costs. II Related Work II-A Efficient SR Methods To reduce the high computational costs of general SR methods [ 18 , 19 , 20 , 21 , 22 ] , efficient SR has been developed recently. CARN [ 9 ] utilizes group convolution and recursive mechanisms to save SR costs. s-LWSR [ 23 ] designs a lightweight network with a U-shaped backbone plus depth-separable convolution. IDN [ 24 ] introduces the concept of information distillation to improve model speed while maintaining model performance. Subsequently, RFDN [ 25 ] , FDIWN [ 4 ] , and FIWHN [ 26 ] further enhance the model’s representation ability by improved information distillation. To obtain a larger receptive field, ShuffleMixer [ 27 ] and CFSR [ 28 ] utilize separable large

Read the complete text