Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency

Hao Yu, Haoyu Chen, Yan Jiang, Wei Peng, Zhaodong Sun, Samuel Kaski, Guoying Zhao · 2025

Read Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency by Hao Yu, Haoyu Chen, Yan Jiang, Wei Peng, Zhaodong Sun, Samuel Kaski, Guoying Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20092.txt.

Introduction

Hao Yu and six co-authors compare two core tools for image models: self-attention and convolution. Self-attention is flexible, but its cost grows very fast with image size. Convolution is cheaper and carries useful built in assumptions about images. Their 2025 paper asks what still gives self-attention its edge in modern vision systems.

The team names two sources of that edge. One is adaptive routing, where links between positions shift with image content. Fixed convolution filters lack that flexibility. The other is lateral inhibition, where scores compete and weak links fade. They build Attentive Convolution around both ideas using small three by three kernels. They then form a model family called AttNet. AttNet scores well on ImageNet with few parameters. The new operator also helps diffusion based image generation, with faster sampling.

Catalog record

CREATORHao Yu, Haoyu Chen, Yan Jiang, Wei Peng, Zhaodong Sun, Samuel Kaski, Guoying Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20092

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency Hao Yu, Haoyu Chen, Yan Jiang, Wei Peng, Zhaodong Sun, Samuel Kaski, Guoying Zhao Abstract Self-attention (SA) has become the cornerstone of modern vision backbones for its powerful expressivity over traditional Convolutions (Conv). However, its quadratic complexity remains a critical bottleneck for practical applications. Given that Conv offers linear complexity and strong visual priors, continuing efforts have been made to promote the renaissance of Conv. However, a persistent performance chasm remains, highlighting that these modernizations have not yet captured the intrinsic expressivity that defines SA. In this paper, we re-examine the design of the CNNs, directed by a key question: what principles give SA its edge over Conv? As a result, we reveal two fundamental insights that challenge the long-standing design intuitions in prior research (e.g., Receptive field). The two findings are: (1) Adaptive routing : SA dynamically regulates positional information flow according to semantic content, whereas Conv employs static kernels uniformly across all positions. (2) Lateral inhibition : SA induces score competition among token weighting, effectively suppressing redundancy and sharpening representations, whereas Conv filters lack such inhibitory dynamics and exhibit considerable redundancy. Based on this, we propose Attentive Convolution (ATConv), a principled reformulation of the convolutional operator that intrinsically injects these principles. Interestingly, with only 3 × 3 3\times 3 kernels, ATConv consistently outperforms various SA mechanisms in fundamental vision tasks. Building on ATConv, we introduce AttNet, a CNN family that can attain 84.4% ImageNet-1K Top-1 accuracy with only 27M parameters. In diffusion-based image generation, replacing all SA with the proposed 3 × 3 3\times 3 ATConv in SiT-XL/2 reduces ImageNet FID by 0.15 in 400k steps with faster sampling. Code is available at: github.com/price112/Attentive-Convolution. I Introduction Convolutional neural networks (CNNs) [ 1 , 2 , 3 , 4 , 5 , 6 ] have long dominated computer vision, achieving remarkable success across diverse tasks owing to their inherent visual inductive biases and computational efficiency on high-dimensional inputs. Recently, Vision Transformers (ViTs) [ 7 , 8 , 9 , 10 , 11 , 12 ] have emerged as a strong competitive alternative, leveraging the self-attention mechanism [ 13 ] to model global dependencies through content-based query–key interactions. Unlike convolutions, which encode fixed local patterns with limited receptive fields, self-attention enables flexible long-range modeling. However, vanilla self-attention suffers from quadratic computational complexity with respect to the input resolution, rendering it inefficient for visual data characterized by high dimensionality and substantial encoding redundancy. Despite that the ViT series [ 7 ] leverages large patch sizes (e.g., 16x16 or 32x32) to reduce the sequence length of input images, its computational complexity is still much higher than that of traditional CNNs. Furthermore, self-attention’s position-agnostic design treats all spatial locations equally during pixel dependency modeling, requiring extensive training resources to learn fundamental visual priors such as locality and object continuity from scratch. To address these limitations, modern ViTs have undergone substantial architectural evolution. An important revolution was the adoption of CNN-like pyramid designs [ 12 , 9 ] , which progressively downsample spatial dimensions, restricting attention computation to manageable resolutions. Based on this, recent state-of-the-art ViTs [ 14 , 15 , 16 , 17 , 18 ] further hybridize attention with depth-wise convolutions [ 19 , 20 ] , creating models that strategically leverage convolution’s efficiency for local processing while preserving attention’s capacity for global modeling. Notably, the persistence of convolution reveals a crucial insight: its visual efficiency and inherent visual inductive biases remain indispensable for vision systems. Recognizing this complementarity, researchers have sought to augment CNNs with ViT-inspired principles. The most common idea is to expand the receptive field that approximates the global modeling of attention. ConvNeXt [ 21 , 22 ] modernizes ConvNets with ViT design elements and adopts larger kernels instead of the traditional 3\times 3 ones, while reparameterization-based architectures [ 23 , 24 ] further extend kernel sizes up to 31 \times 31 to capture long-range dependencies. Although these strategies yield notable improvements over classical CNNs such as ResNet [ 1 ] , a substantial gap from modern ViTs persists. For example, with similar parameters, TransNeXt-Tiny [ 17 ] wins ConvNeXt-Tiny [ 21 ] by 2.0% Top-1 accuracy in ImageNet-1K classification. This disparity highlights that simply augmenting CNN from the receptive field perspective is insufficient to capture the fundamental advantages of ViTs. Fig. 1: Visualization of adaptive routing and lateral inhibition on 3 \times 3 ATConv, 7 \times 7 DWConv [ 21 ] , 3 \times 3 Conv [ 1 ] , vanilla global attention [ 7 ] , window [ 9 ] and pooling [ 12 ] attention. We analyze operator-intrinsic behavior by extracting influence maps G(h,w)=\sum_{c}\lvert\partial y_{c^{\ast},h^{\ast},w^{\ast}}/\partial x_{c,h,w}\rvert . Adaptive routing is visualized via distance-weighted centroids of high-influence regions, with \mathrm{FAR} at radial threshold r_{0} measuring long-range preference. Diverse arrows indicate different inputs produce different aggregation patterns, while uniform arrows reveal fixed routing. Lateral inhibition is quantified by the off-center suppression response to positive perturbations at anchors, where brighter regions indicate stronger surround suppression. ATConv with a compact 3 \times 3 kernel exhibits pronounced adaptive routing and lateral inhibition effects similar to the global self-attention . See our depository for detailed visualization code. In this paper, we dive deeper and present two new perspectives to understand the underlying visual expressivity of self-attention over Conv-based operators: the adaptive routing and lateral inhibition properties. Through a unified weighted aggregation framework, we demonstrate that vanilla convolution is a static weight aggregation operator with an identity representation basis, which leverages the fixed pixel aggregation rule across all spatial positions. In contrast, self-attention implements adaptive routing via query-key interactions with learnable basis space transformations. This adaptivity enables attention modeling to have selective information transfer based on semantic relevance rather than fixed rules. More importantly, the softmax normalization over attention scores induces crucial lateral inhibition dynamics [ 25 ] within attention calculation, a canonical mechanism of biological vision where neurons compete for visual selection. In the primary visual cortex (V1) [ 26 ] , neurons exhibit similar center-surround antagonism, with neighboring neurons competing to represent different features. This competitive dynamic prevents redundant encoding and sharpens feature selectivity—precisely the characteristics lacking in standard convolution. In self-attention mechanisms, the softmax on attention score creates competitive dynamics which implements mutual suppression: increasing the attention weight to one position necessarily decreases others. This sharpens the representations and avoids giving flat responses (e.g., low-rank prediction). The resulting competitive pressure amplifies discriminative connections while suppressing spurious or noisy patterns, leading to more selective and robust representations. In contrast, convolution’s static and independent kernels lack such inhibitory dynamics, fundamentally constraining their expressivity and leading to substantial representation redundancy. To validate our analysis, we propose Attentive Convolution (ATConv), a principled revolution of convolution that embeds the adaptive routing and lateral inhibition principles distilled from self-attention. Unlike vanilla convolutions that employ static, input-agnostic kernels, ATConv adaptively derives its kernels from the input through a context-to-kernel translation mechanism. The key insight is that although convolution operates locally, its kernels can be contextually conditioned to encode global dependencies. In this way, global semantic relevance is translated into input-dependent kernels, implicitly establishing adaptive routing correspondences through local operations, without incurring the quadratic cost of global pairwise

Read the complete text