Elastic Attention Cores for Scalable Vision Transformers
2026
Read Elastic Attention Cores for Scalable Vision Transformers (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12491.txt.
Introduction
Song and five colleagues ask whether vision transformers really need every image patch to talk directly to every other patch. Their VECA model routes global information only through a small set of learned cores, while all dense patches stay intact and update across layers. That change cuts attention cost from quadratic to linear in patch count for a fixed core budget. Unlike earlier latent-token designs, it avoids squeezing the spatial representation itself. The result supports both global recognition and dense prediction without direct patch-to-patch links.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80248 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Elastic Attention Cores for Scalable Vision Transformers Abstract Vision Transformers (ViTs) achieve strong data-driven scaling by leveraging all-to-all self-attention. However, this flexibility incurs a computational cost that scales quadratically with image resolution, limiting ViTs in high-resolution domains. Underlying this approach is the assumption that pairwise token interactions are necessary for learning rich visual-semantic representations. In this work, we challenge this assumption, demonstrating that effective visual representations can be learned without any direct patch-to-patch interaction. We propose VECA (Visual Elastic Core Attention), a vision transformer architecture that uses efficient linear-time core-periphery structured attention enabled by a small set of learned cores. In VECA, these cores act as a communication interface: patch tokens exchange information exclusively through the core tokens, which are initialized from scratch and propagated across layers. Because the N N image patches only directly interact with a resolution invariant set of C C learned “core” embeddings, this yields linear complexity O ( N ) O(N) for predetermined C C , which bypasses quadratic scaling. Compared to prior cross-attention architectures, VECA maintains and iteratively updates the full set of N N input tokens, avoiding a small C C -way bottleneck. Combined with nested training along the core axis, our model can elastically trade off compute and accuracy during inference. Across classification and dense tasks, VECA achieves performance competitive with the latest vision foundation models while reducing computational cost. Our results establish elastic core-periphery attention as a scalable alternative building block for Vision Transformers. Project repository here. Alan Z. Song †1 Yinjie Chen †2 Mu Nan ⋄2 Rui Zhang 2 Jiahang Cao 2 Weijian Mai 2 Muquan Yu 2 Hossein Adeli 3 Deva Ramanan 1 Michael J. Tarr 1 Andrew F. Luo ∗2 1 Carnegie Mellon University 2 University of Hong Kong 3 Columbia University † Co-first authors, equal contribution ⋄ Additional core contributor ∗ Corresponding author A.Z.S.: [email protected] Y.C.: [email protected] [email protected] [email protected] [email protected] 1 Introduction Recent advances in computer vision have been driven by increases in both computational resources and dataset scale. These developments have reshaped architectural design, driving a shift away from strong inductive biases and towards more flexible, data-driven models [ 139 , 140 , 83 , 74 , 116 ] . Whereas convolutional neural networks directly encode locality and translation equivariance into their structure [ 93 , 54 ] , these design choices constrain the representations that can be learned at scale. As datasets have grown, the value of these inductive biases has diminished. Vision Transformers (ViTs [ 33 ] ) replace convolutions with self-attention and impose only weak spatial constraints through patchification, enabling consistent scaling and global receptive fields at every layer [ 113 , 105 ] . Yet this flexibility comes at a substantial computational cost. ViTs rely on dense token-to-token interactions, where every patch attends to every other patch, leading to quadratic complexity that becomes prohibitive at high resolutions [ 79 , 101 ] . Prior work has sought to accelerate self-attention by approximating the softmax operator. While these methods reduce computational cost, they still largely seek to preserve the underlying self-attention mechanism. We illustrate our framework in Figure 1 . Figure 1: Outline of our core-periphery attention structure. (a) Self-attention utilizes a fully connected attention matrix with N^{2} comparisons given N input patches. VECA constructs a core-periphery matrix with C core tokens that form a clique, requiring only 2NC+C^{2} comparisons. (b) We visualize our dense features with UMAP. Our method produce stable embeddings across resolutions. (c) Top : We visualize our features and [CLS] -dense cosine similarity of our model, comparing such visuals to its teacher (DINOv3) as well as other baseline embeddings. Bottom : We visualize core attention weights with UMAP. Core attention maps start off isotropic (spherical), and become increasingly semantic. (d) Top : FLOPs of our attention block versus DINOv3 across resolutions. Bottom : We can elastically vary the number of tokens, trading off accuracy and speed. In this work, we revisit the necessity for dense patch-to-patch self-attention by introducing Visual Elastic Core Attention ( VECA ). VECA replaces standard self-attention with a block-sparse attention matrix mediated by a small, resolution invariant set of C learned core tokens. When viewed as an attention connectivity graph, this forms a core-periphery network [ 120 , 165 ] . The core tokens constitute a fully connected clique [ 92 ] that interacts with all peripheral image patches, eliminating the need for patches to directly attend to one another. By routing information through this set of core tokens, the computational cost of the attention operation becomes (2NC+C^{2}) , where N is the number of image patches. For a predetermined C , this yields linear complexity relative to image resolution. We leverage this architecture to enable adaptive computation by applying nested dropout [ 119 ] to the core tokens during training. This encourages the network to learn ordered representations, implicitly ranking the core tokens. At test time, this enables elastic adjustments to the inference cost. Building upon recent work that successfully distill representations from strong vision foundation models [ 147 , 114 , 127 , 124 ] , we supervise our model using a DINOv3 [ 132 ] teacher. We observe emergent behavior in core-to-patch spatial attention maps across layers, which evolve from isotropic (blob like) to semantically-aligned groups. Notably, this isotropic-to-semantic transform emerges without any explicit constraints encouraging such behavior. Elastically decreasing the number of cores during inference impacts early-network attention granularity, yielding noticeably coarser maps. Concretely, our contributions are as follows: (1) We propose VECA, a visual backbone that replaces quadratic self-attention with linear-time core-periphery attention in every layer, forcing token interactions through a resolution-invariant bottleneck. (2) We evaluate VECA on both classification and dense spatial tasks, where our model achieves competitive performance with the DINOv3 teacher. At ViT-B size, our model achieves less than 2% top-1 Acc. difference (81.93 versus 83.56) on Imagenet-1k classification, and less than 0.28 mIoU difference (57.46 versus 57.74 – despite 87.1\% fewer attention interactions per layer) on the challenging PASCAL context segmentation benchmark. (3) We explore the properties of the core tokens. We identify emergent behavior and empirically show they exhibit object centric representations. Our work suggests that elastic core-periphery attention is a promising and scalable building block of Vision Transformers. 2 Related work Elastic Models and Representations. Scalable inference has explored elastic representations and adjustable cost networks [ 170 ] . Matryoshka representation learning [ 71 ] produces truncatable embeddings, while elastic neural networks enable adjustable model capacity. Parameter hierarchies can be learned via nested dropout [ 119 , 133 , 129 , 48 ] , yielding ordered components [ 72 ] . This has been further extended to width-adaptive architectures [ 136 , 65 , 161 ] . Such structural elasticity supports sub-network extraction [ 19 , 142 , 77 , 166 ] with transformer specific designs [ 70 , 20 ] . Input-dependent computation can be achieved via early-exiting [ 138 , 149 , 85 ] or iterative processing [ 39 , 30 , 152 , 58 ] . At the data-level, elasticity can utilize token merging [ 12 ] , scratchpads [ 155 ] , or iterative expansion [ 49 ] . Sparse expert selection offers an additional axis of adjustment [