Simpler Fast Vision Transformers with a Jumbo CLS Token
Anthony Fuller Yousef Yassin Daniel G. Kyrollos, ⋆ \star Evan Shelhamer ⋆ \star James R. Green, Carleton University, University of British Columbia, Vector Institute ⋆ \star Co-advising author · 2025
Read Simpler Fast Vision Transformers with a Jumbo CLS Token by Anthony Fuller Yousef Yassin Daniel G. Kyrollos, ⋆ \star Evan Shelhamer ⋆ \star James R. Green, Carleton University, University of British Columbia, Vector Institute ⋆ \star Co-advising author (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15021.txt.
Introduction
Anthony Fuller and four co-authors speed up vision transformers, the image models behind many recent gains, without giving up their flexibility. Their 2025 paper notes that rival fast designs hard-wire one input shape or shave tokens and lose accuracy. Their fix works on the model's summary token, the one slot that gathers global information.
They slim the many patch tokens and widen that one summary token, calling the result a Jumbo token. A wider shared feed-forward block serves just that token, adding capacity at low cost. On standard image tests the design beat register-token baselines from tiny to large sizes while holding speed. Gains also showed in segmentation, self-supervised pretraining, and test-time adaptation. The plain shape of the design keeps it compatible with existing tools.
Catalog record
| CREATOR | Anthony Fuller Yousef Yassin Daniel G. Kyrollos, ⋆ \star Evan Shelhamer ⋆ \star James R. Green, Carleton University, University of British Columbia, Vector Institute ⋆ \star Co-advising author |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-15021
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Simpler Fast Vision Transformers with a Jumbo CLS Token Anthony Fuller Yousef Yassin Daniel G. Kyrollos, ⋆ \star Evan Shelhamer ⋆ \star James R. Green, Carleton University, University of British Columbia, Vector Institute ⋆ \star Co-advising author Abstract We introduce a simple enhancement of vision transformers (ViTs) to improve accuracy while maintaining throughput. Our approach, Jumbo, creates a wider CLS token, which is split to match the patch token width before attention, processed with self-attention, and reassembled. After attention, Jumbo applies a dedicated, wider FFN to this token. Since there is only one Jumbo token, its cost is minimal, and because we share this FFN across layers, its parameter count is controlled. Jumbo significantly improves over ViT+Registers [ 1 ] on ImageNet-1K and ImageNet-21K. These gains are largest at small sizes / high speeds, e.g., ViT-nano+Jumbo outperforms ViT-nano+Registers by 13 % 13\% . In fact, our Jumbo models are so efficient that they outperform specialized compute-efficient models while preserving the architectural advantages of plain ViTs, such as support for token dropping and other modalities. Accordingly, we demonstrate that Jumbo excels in these two settings via masked autoencoding and on a suite of time series benchmarks. Code and weights available: https://github.com/antofuller/jumbo 1 Introduction There has been tremendous progress in computer vision over the past decade. Innovations have resulted in models with greater absolute accuracy, greater speed for a given accuracy, and greater simplicity. Vision transformers (ViTs) [ 2 ] exemplify this progress. ViTs are simple . They split an image into patches, linearly project patches to form patch embeddings, add position embeddings to patch embeddings, prepend a learnable “ CLS ” token, and then process this set of vectors with a transformer; these operations can be implemented in a single file of {\sim}100 lines of PyTorch [ 3 ] . ViTs are flexible . They offer seamless sparse computation via token dropping [ 4 , 5 , 6 ] , patch resizing [ 7 , 8 ] , image size extrapolating [ 9 ] , model scaling [ 10 , 11 ] , and multimodal processing [ 12 , 13 ] . ViTs are effective . They achieve state-of-the-art (SOTA) performance in image classification [ 14 , 15 ] , object detection [ 16 , 17 ] , and semantic segmentation [ 18 ] ; ViTs are core components of vision-language models [ 19 , 20 ] . However, plain ViTs often underperform more specialized architectures w.r.t. compute efficiency, especially at high speeds. Many recent architectures improve the computational efficiency of ViTs [ 21 , 22 , 23 , 24 ] , where an improvement in efficiency typically refers to achieving a better accuracy-speed trade-off. These architectures leverage a hierarchical design, following classic architectures, such as ResNets [ 25 ] and VGGNets [ 26 ] — they progressively reduce spatial dimensions while increasing channel width. A larger width corresponds to a larger feedforward network (FFN), which increases model capacity by enabling non-linear processing over more dimensions. Yet, the cost of larger FFNs is controlled in these architectures by decreasing the number of patches/tokens. We highlight this added capacity at little computational cost as a common element of such models. These efficient architectures typically intersperse modules such as convolutions, pooling, sparse or linear attention, and other innovations with standard ViT modules (i.e., self-attention and point-wise FFNs). However, these innovations sacrifice attractive abilities of plain ViTs, such as token dropping and multimodal processing. Instead of leveraging asymmetry across the network depth (i.e., hierarchical models) to improve performance, our method leverages asymmetry across the sequence length to improve performance. 10 15 20 25 30 35 40 45 60 65 70 75 80 EfficientViT SHViT MobileNetV4 ViT+Registers BiXT ViT+Jumbo (ours) Throughput (K images/s) Top-1 Acc. ( \% ) (a) ImageNet-Val 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 41 42 43 44 45 46 47 ViT+Registers ViT+Jumbo (ours) 1.9\times faster Throughput (K images/s) Top-1 Acc. ( \% ) (b) ImageNet- 21K at 224^{2} px 10 15 20 25 30 35 40 45 50 55 60 65 70 Eff.ViT SHViT MobileNetV4 ViT+Reg. BiXT ViT+Jumbo (ours) Throughput (K images/s) Top-1 Acc. ( \% ) (c) ImageNet-v2 Legend Method Venue Attention- only Non- hierarchical EfficientViT [ 21 ] ICCV ’23 ✗ ✗ SHViT [ 24 ] CVPR ’24 ✗ ✗ MobileNetV4 [ 27 ] ECCV ’24 ✗ ✗ ViT+Registers [ 1 ] ICLR ’24 ✓ ✓ BiXT [ 28 ] NeurIPS ’24 ✓ ✓ ViT+Jumbo (ours) ✓ ✓ Our Jumbo is attention-only and non-hierarchical, which provides out-of-the-box support for SOTA self-supervised learning algorithms, multimodal data, and non- 2 D data types—while being much simpler than existing efficient architectures. Figure 1 : Plain ViTs (attention-only and non-hierarchical) are in red , and others are in blue . ViT+Jumbo outperforms SOTA compute-efficient architectures while preserving the advantages of plain ViTs. ViT+Jumbo equals or outperforms ViT+Registers on the standard ImageNet-1K and strictly improves on the challenging ImageNet-21K dataset. Throughput is measured on an RTX 4090 GPU using PyTorch 2.6.0, torch.compile , and a 512 batch size. ViT+Jumbo (ours) ViT+Registers [ 1 ] Figure 2 : (Left) Our ViT+Jumbo method creates a wider CLS token that gets split into several tokens, with width equal to the patch width, prior to multi-headed self-attention (MHSA). After attention, the split Jumbo CLS token is reassembled via concatenation, and is then processed by its own FFN . Patches are processed by their own, shared FFN . (Right) ViT+Registers creates many register tokens all equal to the patch width. All registers, patches, and the CLS token are processed by a shared FFN . ViT+Jumbo enhances global processing as the (split) global tokens can interact via an FFN, in addition to attention. Our method, called Jumbo , increases the width of the CLS token to J times the patch width. Before self-attention, our method splits this extra wide Jumbo token into J smaller tokens, includes them in self-attention, and then concatenates them to reassemble a single Jumbo token. Rather than sharing FFN parameters with all patches, Jumbo leverages a wider CLS -specific FFN to increase model capacity. The cost of this larger FFN is minimal: processing a single token is fast, and sharing Jumbo weights across layers is memory efficient. Crucially, Jumbo preserves all the advantages of plain ViTs previously discussed, while efficiently adding global capacity. This includes support for token dropping, which is critical for many self-supervised learning (SSL) algorithms [ 29 , 30 , 4 , 31 ] . Through a series of controlled experiments we show that ViT+Jumbo outperforms ViT+Registers [ 1 ] , which is our closest comparison. Specifically, Jumbo clearly improves in image classification on ImageNet-1K [ 32 ] , with gains up to 15\% at an equal speed. Jumbo scales well to more challenging tasks, outperforming ViT+Registers by 3\% at an equal speed on ImageNet-21K [ 33 ] . Additionally, ViT+Jumbo is the first attention-only and non-hierarchical architecture to outperform specialized compute-efficient architectures at high speeds, like EfficientViT [ 21 ] . In remaining attention-only and non-hierarchical, ViT+Jumbo is compatible with SOTA self-supervised learning and adaptable to other data modalities such as time series — we demonstrate both of these applications. We pretrain ViT+Jumbo using masked autoencoding (MAE [