Emulating Self-attention with Convolution for Efficient Image Super-Resolution
Dongheon Lee, Seokju Yun, Youngmin Ro * · 2025
Read Emulating Self-attention with Convolution for Efficient Image Super-Resolution by Dongheon Lee, Seokju Yun, Youngmin Ro * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06671.txt.
Introduction
Dongheon Lee, Seokju Yun, and Youngmin Ro mimic the self-focus of transformer nets with plain convolution, to sharpen images at low cost. Their paper is from two thousand twenty-five.
Transformers are strong but heavy for this task. The team notes that self-focus traits repeat from layer to layer. Their part, called ConvAttn, uses one big shared kernel plus small dynamic ones to copy both the long reach and the case-by-case weight of self-focus. That cuts the need for true self-focus and its costly memory work. They also fit flash focus into the light regime, and grow the window a lot. Tests show a clear gain in score with far less lag and memory.
Catalog record
| CREATOR | Dongheon Lee, Seokju Yun, Youngmin Ro * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Emulating Self-attention with Convolution for Efficient Image Super-Resolution Dongheon Lee, Seokju Yun, Youngmin Ro * Abstract In this paper, we tackle the high computational overhead of Transformers for efficient image super-resolution (SR). Motivated by the observations of self-attention’s inter-layer repetition, we introduce a convolutionized self-attention module named Convolutional Attention (ConvAttn) that emulates self-attention’s long-range modeling capability and instance-dependent weighting with a single shared large kernel and dynamic kernels. By utilizing the ConvAttn module, we significantly reduce the reliance on self-attention and its involved memory-bound operations while maintaining the representational capability of Transformers. Furthermore, we overcome the challenge of integrating flash attention into the lightweight SR regime, effectively mitigating self-attention’s inherent memory bottleneck. We scale up the window size to 32 × \times 32 with flash attention rather than proposing an intricate self-attention module, significantly improving PSNR by 0.31dB on Urban100 × \times 2 while reducing latency and memory usage by 16 × \times and 12.2 × \times . Building on these approaches, our proposed network, termed Emulating Self-attention with Convolution (ESC), notably improves PSNR by 0.27 dB on Urban100 × \times 4 compared to HiT-SRF, reducing the latency and memory usage by 3.7 × \times and 6.2 × \times , respectively. Extensive experiments demonstrate that our ESC maintains the ability for long-range modeling, data scalability, and the representational power of Transformers despite most self-attention being replaced by the ConvAttn module. 1 1 footnotetext: Corresponding author. 1 Introduction Image Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs and remains an active area of research in computer vision. Recently, the demand for multimedia content and generative models has increased significantly, making SR particularly noteworthy as it enables users to enjoy high-quality content under resource-constrained conditions. Consequently, practical deployment has emerged as a critical consideration in SR tasks, motivating numerous SR studies to enhance performance while reducing computational complexity (FLoating point OPerations; FLOPs) and parameter size. Therefore, Transformers have gained significant attention in the SR task since they have achieved superior performance over Convolutional Neural Networks (CNNs) while requiring lower FLOPs and fewer parameters. By capturing long-range dependencies and performing input-dependent weighting through self-attention, Transformers exhibit high representational capacity with enhanced performance, especially as the training data volume increases [ 15 , 5 ] . Accordingly, numerous transformer-based methods have been proposed [ 75 , 56 , 47 ] to capitalize on the advantages of self-attention while reducing FLOPs and parameter counts. Figure 1 : Comparison of performance, latency, and memory usage. Our methods are evaluated against representative SR models, including CNNs, Transformers, and SSMs. However, many of these works overlook the excessive memory access caused by self-attention, which arises from materializing the score matrix ( S=QK^{T} ) [ 10 ] and leveraging memory-bound operations such as tensor reshaping and window masking [ 68 , 39 ] . The memory access issue is further worsened in the SR architecture, which deals with large feature maps without patchify stem or downsampling stage. For example, SwinIR-light [ 36 ] is 4.7 \times slower latency and 2 \times higher memory usage than CNNs [ 77 ] reconstructing an HD image at the \times 2 scale, even though it requires 14.5 \times and 17 \times fewer FLOPs and parameter sizes, respectively. Consequently, despite their promising performance, Transformers are challenging to deploy on resource-constrained devices, such as consumer-level GPUs. In this paper, we aim to design an efficiency-driven Transformer tailored for lightweight SR tasks, achieving both improved performance and memory overhead. We start by conducting a preliminary analysis of self-attention, observing that the similarity modeling performed by self-attention and resulting extracted features remain highly consistent across several layers, as shown in Figure 2 . This finding exhibits that self-attention may extract overlapping features, suggesting the possibility of reducing computational overhead without compromising representational capability by utilizing efficient alternatives. Building on this finding, we establish a design strategy that retains self-attention only in the first layer of each block while replacing the remaining layers with our proposed efficient alternative, the Convolutional Attention (ConvAttn) module. To effectively emulate self-attention’s long-range modeling and instance-dependent weighting, the ConvAttn module operates with a twofold mechanism. First, it simplifies self-attention’s long-range interaction by applying convolution with a shared 13 \times 13 large kernel throughout the entire network, targeting only a subset of channels. Second, dynamic kernels are generated to capture input-dependent weighting, mimicking the adaptive nature of self-attention. By combining these components, the ConvAttn module significantly reduces the reliance on memory-intensive self-attention while maintaining the representational capability of Transformers. With most self-attention layers replaced by ConvAttn, we leverage this efficiency to further enhance the remaining self-attention layers. Specifically, we enlarge the window size of self-attention, significantly improving performance with only a slight increase in FLOPs. However, increasing the window size leads to an enlarged score matrix, substantially raising peak memory usage. To address this, we introduce Flash Attention [ 10 , 14 ] into lightweight SR tasks to avoid materializing the score matrix. Our optimized implementation allows us to scale up the window size to 32 \times 32 while reducing latency and memory usage by 16 \times and 12.2 \times , respectively, as illustrated in Table 1 . Based on these approaches, we introduce the lightweight SR network called Emulating Self-attention with Convolution (ESC). The proposed ESC outperforms ATD-light [ 73 ] on Urban100 \times 4 with 0.1dB improvements on PSNR while being 8.9 \times faster, as shown in Figure 1 . Additionally, ESC-light surpasses ELAN-light [ 75 ] by a large margin of 0.29dB in PSNR on Urban100 \times 2 while also reducing latency by 22%. We further validate our ESC in scenarios where reducing FLOPs and parameter size is essential by introducing ESC-FP, which outperforms MambaIRV2-light [ 21 ] on Manga109 \times 4 with reductions of FLOPs and parameter size by 20% and 32%, respectively. Through our extensive experiments, we demonstrate that our ESC fully leverages the advantages of Transformers – including their large receptive fields, representational capacity, and scalability with respect to data volume – even though most self-attentions are replaced by the ConvAttn module. We support these results with in-depth experiments, as illustrated in Figure 6 , suggesting that the proposed ConvAttn module extracts similar features with self-attention. Our contributions are summarized as follows: • We demonstrate that well-designed convolution partially replaces self-attention, significantly improving efficiency without sacrificing the advantages of Transformers. • We mitigate the memory overhead of self-attention by integrating Flash Attention, thereby enlarging the window size to 32 \times 32 without excessive memory usage. To the best of our knowledge, this is the first successful application of Flash Attention in lightweight SR tasks. • We fully exploit the transformer’s benefits in lightweight SR tasks by making them more efficient and simpler. 2 Related Work Traditional CNNs Early deep learning‐based SR research [ 13 , 29 , 37 , 78 , 77 ] primarily employed convolution that extracts local features from images. These approaches typically relied on stacked convolutions, which often resulted in over‐parameterization. To address this issue, numerous studies [ 28 , 53 , 35 , 46 ] have shared their weights across multiple layers to reduce parameter size. However, many of them suffer from performance degradation since the input-independent weights are shared