Quick ViTs: Speeding up Vision Transformers through Equivariance

David Nordström, Johan Edstedt, Fredrik Kahl, Georg Bökman · 2025

Read Quick ViTs: Speeding up Vision Transformers through Equivariance by David Nordström, Johan Edstedt, Fredrik Kahl, Georg Bökman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15441.txt.

Introduction

David Nordström, Johan Edstedt, Fredrik Kahl, and Georg Bökman speed up vision transformers in this 2025 paper. These image models split pictures into square patches and process each patch. Natural images repeat the same edges and textures at many angles, a symmetry the standard design ignores.

The team's Quick ViTs build that symmetry into the model's layers, covering both flips and rotations. The change cuts computing work more than fivefold and memory use up to eightfold in the affected layers. The authors also study when those savings turn into real speed on modern graphics chips. Trained on a large image set, the models matched or beat standard accuracy while using far fewer resources.

Catalog record

CREATORDavid Nordström, Johan Edstedt, Fredrik Kahl, Georg Bökman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quick ViTs: Speeding up Vision Transformers through Equivariance David Nordström, Johan Edstedt, Fredrik Kahl, Georg Bökman Abstract Natural images exhibit strong geometric regularities: local structures, such as edges, corners, and textures, appear in many orientations and mirror configurations. Since Vision Transformers (ViTs) operate on square image patches, these transformations naturally correspond to the dihedral symmetry group D 8 \mathrm{D}_{8} , also known as the octic group. Recent work has shown that ViTs can be made reflection equivariant and more efficient than standard ViTs simultaneously by implementing the linear layers in the Fourier domain of the reflection group. In this work, we extend the equivariance to reflections and rotations and analyze the scalability of the resulting networks. Our Quick ViTs, based on octic equivariant linear layers, achieve 5.33x reductions in FLOPs and up to 8x reductions in memory compared to ordinary linear layers. By analyzing the arithmetic intensity of these layers, we identify theoretical limits on how much the FLOP savings translate into throughput improvements on modern GPUs. However, these limitations disappear as the embedding dimensions increase. Enabled by their computational efficiency, we conduct a broader empirical evaluation of equivariant ViTs than in previous work. Upon training supervised (DeiT-III) and self-supervised (DINOv2) on ImageNet-1K, we find that our Quick ViTs match or exceed baseline accuracy while at the same time providing substantial efficiency gains. 1 Introduction In the pursuit of flexible yet scalable models, Vision Transformers (ViTs) [ 22 ] have emerged as the dominant architecture in modern computer vision. While ViTs are sometimes heralded as having minimal inductive biases, one key to their success is the construction of visual tokens from image patches and the weight-sharing over the tokens implemented by transformer layers. This weight-sharing ensures permutation equivariance and yields better performing models than allowing arbitrary interactions over tokens [ 5 ] . In general, equivariance can provide a powerful inductive bias in neural networks by enforcing structured responses to transformations such as permutations, translations, rotations, or reflections. However, implementations of equivariant networks often lack in computational efficiency compared to non-equivariant counterparts [ 35 ] , limiting their popularity in large scale settings. The permutation equivariance in transformers is a clear exception to this rule, as it instead improves the computational efficiency of the model. Recent work has furthermore demonstrated how equivariance under reflections can be implemented efficiently in ViTs, obtaining equivariant ViTs that are more computationally efficient than standard ViTs [ 11 ] . In this work, we extend the equivariance to rotations and reflections and introduce Quick ViTs, a family of computationally efficient equivariant vision transformers. Our central observation is that implementing equivariance in the Fourier domain of the symmetry group allows the linear layers to be computed significantly more efficiently. While [ 11 ] only present proof-of-concept experiments, we provide a more extensive evaluation using a self-supervised DINOv2 pipeline and evaluating on both segmentation and classification. We find that rotation and reflection equivariant layers can effectively preserve or improve performance of baseline models while requiring significantly fewer FLOPs, see Figure 1 . The analysis of computational complexity in [ 11 ] is limited to counting FLOPs. We also consider the arithmetic intensity of the equivariant layers, which is crucial to obtain optimal GPU throughput. The analysis of arithmetic intensity shows how the combination of hardware and datatype determines how much of the reduction in FLOPs can be expected to translate into throughput improvements. Figure 1 : Computational savings . Using our equivariant layers in ViTs significantly reduces the computational complexity without sacrificing accuracy on ImageNet-1K, for both supervised and self-supervised training. Detailed results can be found in Section 5 . In summary, our contributions are as follows: (a) We introduce Quick ViTs, a family of vision transformers with octic equivariant layers ( Section 3 ). We propose two variants that are either fully equivariant ( \mathcal{I}_{8} ) or break equivariance in late layers of the network ( \mathcal{H}_{8} ). The equivariant linear layers use 5.33 times fewer FLOPs and 8 times less memory per feature dimension than ordinary linear layers. (b) We analyze the arithmetic intensity of the equivariant linear layers and relate this analysis to the observed throughput improvements ( Section 4 ). (c) We demonstrate empirically that our ViTs can be integrated with state-of-the-art training recipes (DeiT III and DINOv2) without re-tuning hyperparameters ( Section 5 ). We achieve a 40\% FLOP saving with our \mathcal{I}_{8}(\text{ViT-H}) and \mathcal{H}_{8}(\text{ViT-H}) models while matching baseline performance ( Figure 1 ). (d) We study the effects of different invariantization methods including symmetry breaking and varying number of equivariant layers ( Section 5.3 ). These ablations help guide future research on equivariant architectures at scale. Our approach seamlessly integrates into existing ViT architectures and leads to significant gains in throughput and memory efficiency, without sacrificing accuracy. In addition to introducing new state-of-the-art ViTs, from a broader perspective, our contributions give clear evidence that equivariance can matter at scale, which has been a subject of debate in recent literature [ 1 , 60 , 12 , 11 ] . 2 Related Work Vision Transformers. The ViT was introduced by [ 22 ] and has subsequently achieved state-of-the-art results in many domains of computer vision [ 14 , 46 , 34 , 58 , 23 , 40 ] . Significant efforts have been made to scale ViTs [ 64 , 19 ] , alongside strategies to do so efficiently [ 2 ] . Our approach to making ViTs faster via equivariance is orthogonal to other approaches such as pruning or quantizing. We view it as interesting future work to combine our method with such other efficiency improvements. Equivariant Networks. The equivariance of CNNs to (cyclic) image translations can be extended to incorporate larger symmetry groups such as rotations and reflections, as shown by [ 15 , 21 ] using Group Equivariant CNNs (G-CNNs). [ 16 , 61 ] generalized G-CNNs to steerable CNNs, where the features transform according to general group representations. Our ViTs can be seen as a more scalable ViT analogue of the octic steerable CNNs by [ 16 ] . There have also been prior efforts on attention- and transformer-based equivariant architectures, for both point clouds [ 26 , 30 , 3 , 38 ] and, more closely to ours, images [ 49 , 63 , 48 , 37 ] . Even when equivariance is not explicitly built into the architecture, neural networks may still learn approximately equivariant representations when trained on data with the corresponding symmetries [ 10 ] . However, these prior works do not obtain computational benefits over non-equivariant networks, in contrast to our ViTs. Our work is part of an ongoing research direction of studying and improving the scalability of equivariant networks [ 6 , 12 , 7 , 56 , 43 ] . Prior work in this direction mostly focuses on point cloud data, with the notable exception of [ 28 ] and [ 11 ] who consider images. [ 28 ] increase the computational efficiency of G-CNNs but in contrast to our work do not achieve benefits over standard networks. Directly inspiring our work, [ 11 ] demonstrate that incorporating reflection equivariance into modern image classifiers increases computational efficiency while maintaining classification performance.

Read the complete text