CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution

Xin Liu, Jie LiuJie TangGangshan WuState Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China [email protected] {liujie,tangjie,gswu}@nju.edu.cn https://github.com/EquationWalker/CATANet · 2025

Read CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution by Xin Liu, Jie LiuJie TangGangshan WuState Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China [email protected] {liujie,tangjie,gswu}@nju.edu.cn https://github.com/EquationWalker/CATANet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06896.txt.

Introduction

Xin Liu, Jie Liu, Jie Tang, and Gangshan Wu present a 2025 network for lightweight image super resolution. The task is to rebuild a sharp image from a small blurry one. Transformer models do it well, but their cost grows fast with image size, and window based fixes can only see local patches.

Their network, CATANet, groups tokens by content across the whole image, not by place. Shared group centers are learned in training and then held fixed, which keeps the cost low. Attention inside each group links far apart but like regions, and a cross group step shares global context. Tests beat a top cluster based rival while the model stays light.

Catalog record

CREATORXin Liu, Jie LiuJie TangGangshan WuState Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China [email protected] {liujie,tangjie,gswu}@nju.edu.cn https://github.com/EquationWalker/CATANet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06896

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution Xin Liu, Jie LiuJie TangGangshan WuState Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China [email protected] {liujie,tangjie,gswu}@nju.edu.cn https://github.com/EquationWalker/CATANet Abstract Transformer-based methods have demonstrated impressive performance in low-level visual tasks such as Image Super-Resolution (SR). However, its computational complexity grows quadratically with the spatial resolution. A series of works attempt to alleviate this problem by dividing Low-Resolution images into local windows, axial stripes, or dilated windows. SR typically leverages the redundancy of images for reconstruction, and this redundancy appears not only in local regions but also in long-range regions. However, these methods limit attention computation to content-agnostic local regions, limiting directly the ability of attention to capture long-range dependency. To address these issues, we propose a lightweight Content-Aware Token Aggregation Network (CATANet). Specifically, we propose an efficient Content-Aware Token Aggregation module for aggregating long-range content-similar tokens, which shares token centers across all image tokens and updates them only during the training phase. Then we utilize intra-group self-attention to enable long-range information interaction. Moreover, we design an inter-group cross-attention to further enhance global information interaction. The experimental results show that, compared with the state-of-the-art cluster-based method SPIN, our method achieves superior performance, with a maximum PSNR improvement of 0.33dB and nearly double the inference speed. 0 0 footnotetext: ✉: Corresponding author ([email protected]). 1 Introduction Figure 1 : Performance and model complexity comparison on Urban100 dataset for upscaling factor ×2. Single Image Super-Resolution (SISR) is a classic task in computer vision and image processing. Its goal is to recover High-Resolution (HR) images from its Low-Resolution (LR) counterpart. SISR is widely applied in various fields, such as medical imaging, digital photography, and reducing server costs for streaming media transmission. Figure 2 : The overall architecture of CATANet and the structure of Token Aggregation Block and Local-Region Self-Attenion. Since the pioneering work of Convolutional Neural Networks (CNNs) in SISR [ 9 ] , numerous CNN-based methods [ 10 , 20 , 22 , 25 , 61 , 21 , 58 ] have been proposed to challenge the reconstruction of HR images from LR images. Due to the local mechanism of convolution, which limits the capture of global dependencies, some CNN-based methods [ 25 , 61 ] use very deep and complex network architectures to increase receptive fields and achieve better performance. However, these methods inevitably increase computational resources, which restricts their applicability. Recently, with the success of transformers in NLP [ 42 ] , transformers have been applied to multiple high-level computer vision tasks [ 12 , 8 , 31 , 46 , 3 ] , achieving remarkable results. The key success of transformer-based approaches is the Self-Attention mechanism, which can effectively capture long-range dependencies. Due to the powerful potential of transformers, they have also attracted attention in low-level computer vision tasks [ 24 , 53 , 50 , 5 , 45 , 59 , 4 ] , including image super-resolution. HAT [ 4 ] shows through experiments that utilizing more global information can effectively enhance the quality of reconstructed images. However, these methods divide larger images into smaller local regions for separate processing to alleviate the high complexity of global self-attention. Although this strategy can improve the efficiency of transformer-based models and provide more refined local information, it still has some limitations. SwinIR [ 24 ] divides the image into content-agnostic local windows, limiting the use of similar tokens over long ranges and resulting in undesirable results. Axial stripe attention [ 11 ] expands the receptive field in a cross-shaped pattern, but this approach remains content-agnostic and may introduce some irrelevant interfering information. To alleviate these issues, some methods have explored clustering-based solutions. SPIN [ 56 ] , for instance, employs the soft k-means-based token algorithm [ 19 ] for clustering, using the cluster centers as proxies between query and key in the attention mechanism, facilitating the propagation of long-range information. However, SPIN still encounters two primary limitations: (1) The cluster centers provide a sparse representation of the image tokens. However, relying exclusively on these centers for long-range information propagation results in a coarse approximation, which is insufficient for capturing and leveraging detailed long-range dependencies. (2) The inference speed of SPIN is limited due to the need for iterative processing over clustering centers during inference, which constrains the deployment of lightweight models. ATD [ 57 ] introduces an auxiliary dictionary to learn priors from the training data and uses the dictionary to classify tokens, leading to more accurate token grouping. However, ATD applies multiple attention mechanisms concurrently to boost performance, which significantly increases computational burden and makes ATD less suitable for lightweight scenarios, as shown in Fig. 1 . To address these challenges, we propose a novel efficient Content-Aware Token Aggregation Network (CATANet) with Token-Aggregation Block as its core component. Token-Aggregation Block mainly consists of efficient Content-Aware Token Aggregation module, Intra-Group Self-Attention, and Inter-Group Cross-Attention. In contrast to the clustering methods previously discussed, we designed an efficient token centers updating strategy within our Content-Aware Token Aggregation (CATA) module. It shares token centers across all image tokens and updates them only during the training phase, eliminating the impact of updating token centers on model inference speed. Unlike SPIN [ 56 ] , which uses clustering centers for long-range information propagation, our Intra-Group Self-Attention employs CATA module to efficiently aggregate content-similar tokens together, forming content-aware regions. Attention is performed within each group, allowing for finer-grained long-range interactions among token information. Moreover, we introduce Inter-Group Cross-Attention, which applies cross-attention between each group and token centers, further enhancing global information interaction. In summary, our main contributions are as follows: \bullet We propose a novel lightweight image SR network, Content-Aware Token Aggregation Network (CATANet). Our CATANet combines token aggregation with attention mechanisms to capture long-range dependencies while ensuring high inference efficiency. \bullet To mitigate the impact of token centers updates on inference speed, we designed efficient Content-Aware Token Aggregation (CATA) that only updates the token centers during the training phase. \bullet We propose Intra-Group Self-Attention and Inter-Group Cross-Attention, which operate between tokens and effectively capture long-range and global dependencies, effectively reduce interference from irrelevant information, and achieve computational complexity comparable to local-region-based methods. \bullet We conducted extensive experiments to demonstrate that our method surpasses the state-of-the-art cluster-based lightweight SR method SPIN, with a maximum PSNR improvement of 0.33 dB and nearly double the inference speed. 2 Related Work Figure 3 : (a) A simple illustration for obtaining the initial token centers. (b) Visualization of sub-grouping. The dashed boxes of the same color indicate the same group (left) or subgroup (right). (c) Each subgroup’s \mathbf{Q}_{j} to attend to the \mathbf{K}_{j} of two consecutive subgroups, where the same color denotes the same group \mathcal{G}_{j} Deep Networks for SR. Deep neural networks have become the mainstream solutions for Image Super-Resolution in recent years due to their powerful representation learning capabilities. Since SRCNN [ 9 ] first successfully applied CNNs to the SR field through a three-layer CNN network, a large number of CNN-based methods have achieved state-of-the-art performance using more complex or efficient structures, such as those with residual connections [ 25 , 61 , 47 , 27 ] and U-shaped architectures [ 7 , 34 , 29 ] . Compared

Read the complete text