A Circular Argument: Does RoPE need to be Equivariant for Vision?

Chase van de Geijn, Timo Lüddecke, Polina Turishcheva, Alexander S. Ecker · 2025

Read A Circular Argument: Does RoPE need to be Equivariant for Vision? by Chase van de Geijn, Timo Lüddecke, Polina Turishcheva, Alexander S. Ecker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08368.txt.

Introduction

Chase van de Geijn and co-authors ask whether rotary positional codings for vision models really need to be equivariant, the property usually credited for their success. Rotary codings first proved themselves on one dimensional language sequences, and generalizations to images and video are now spreading. The team shows rotary coding is among the most general equivariant embeddings in one dimension, and that mixed rotary coding plays the same role in higher dimensions when generators commute.

They then propose spherical rotary coding, which drops the commutativity demand and uses non commuting generators instead. In tests it learns as well as or better than its equivariant relatives. Relative position may matter less than commonly believed in vision, freeing faster, better generalizing designs.

Catalog record

CREATORChase van de Geijn, Timo Lüddecke, Polina Turishcheva, Alexander S. Ecker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Circular Argument: Does RoPE need to be Equivariant for Vision? Chase van de Geijn, Timo Lüddecke, Polina Turishcheva, Alexander S. Ecker Abstract Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such as images and videos. The success of RoPE has been thought to be due to its positional equivariance, i.e. its status as a relative positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for M M -dimensional data, if we require commutative generators – a property necessary for RoPE’s equivariance. However, we question whether strict equivariance plays a large role in RoPE’s performance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but assumes non-commutative generators. Empirically, we find Spherical RoPE to have the equivalent or better learning behavior compared to its equivariant analogues. This suggests that relative positional embeddings are not as important as is commonly believed, at least within computer vision. We expect this discovery to facilitate future work in positional encodings for vision that can be faster and generalize better by removing the preconception that they must be relative. 1 Introduction Deep learning is in the age of transformers [ 73 ] . At their core, transformers are built on attention [ 4 , 61 ] , which is a permutation-invariant operation [ 74 ] , making them agnostic to word or token position within a corpus. To break this symmetry, tokens must be modified with position embeddings [ 25 , 83 ] . Recently, Rotational Positional Encodings (RoPE) [ 68 ] have gained popularity, touting an emphasis on the relative position between two tokens rather than their absolute positions [ 22 , 26 , 43 , 46 ] . However, some of the original claims of RoPE have been called into question leading to confusion as to why it works: 68 claimed the attention scores to decay with distance between tokens. This was found to be true only for attention with the same query and key [ 5 ] . Moreover, transformers with causal masking have been shown to require no positional encodings to be capable of recovering absolute position [ 24 ] , making RoPE’s relative (shift-equivariant) claim questionable. However, many new methods continue to be motivated by RoPE’s benefit from shift-equivariance [ 27 , 60 , 85 ] . To guide future research in positional encodings, it is important to discover whether shift-equivariance truly makes RoPE successful and needs to be preserved when extending it. Both transformer and RoPE were originally designed for one-dimensional sequences such as language. RoPE encodes position by pairing dimensions within the query and key vectors within a transformer and rotating the paired dimensions. Transformers have become the current staple across all AI fields [ 17 , 29 , 46 , 47 , 63 ] . Naturally, RoPE’s recent popularity in NLP has also spread to Vision Transformers (ViT), where the data is two- or three-dimensional, corresponding to images and videos. How to extend RoPE to other modalities is nontrivial and assumptions must be made to maintain equivariance [ 44 , 60 ] . The most commonly used approach for extending RoPE to ViTs is through Axial RoPE [ 14 , 26 , 77 ] , partitioning the embedding dimensions into dimensions rotated independently either by the horizontal or the vertical position of the tokens. However, this approach does not allow for diagonal attention patterns where horizontal and vertical information “mix", which have been hypothesized to enhance generalization; consequently, learned Mixed RoPE was proposed [ 26 ] . Even more recently, LieRE [ 52 ] generalized Mixed RoPE from pair-wise rotations to higher dimensional rotations using learned skew-symmetric Lie algebras. If one defines rotations to be special orthogonal transformation, LieRE is the most general form of rotation encoding. However, while general, LieRE does not guarantee equivariance. In this paper, we investigate the relationships across these different forms of positional encoding. In Section 3 , we mathematically show RoPE with parameterized rotation speeds to be equivalent to LieRE for one-dimensional data. When the number of positional dimensions is higher dimensional, LieRE is not guaranteed to be equivariant unless constraints are placed on the Lie algebras. Using this insight, we derive Axial RoPE by imposing a “mutual exclusivity" constraint on the eigenvalues of LieRE’s generators. Further, we will show that if one loosens this constraint – requiring the Lie algebra to be commutative between the generator – then one arrives at Mixed RoPE. To be a relative encoding, this commutativity property is necessary [ 44 , 60 ] , thus making Mixed RoPE the most general form of LieRE which maintains equivariance. However, it has been noted that requiring the positional embedding to be relative is an inductive bias whose necessity to RoPE’s success is unclear [ 1 , 5 , 24 ] . The perceived necessity of equivariance has led to a circular argument where positional embeddings are assumed to perform well because they are relative, and all new embeddings must be relative because relative embeddings perform well. To break this cycle, we believe that it is imperative to establish the importance of equivariance embeddings for multi-dimensional RoPE. In Section 4 , we propose alternative methods to establish a cause-effect experiment to evaluate whether equivariances is a predominant contributor to RoPE’s faster training dynamics and generalization. To this end, we propose Spherical RoPE which takes a non-commutative assumption, thus breaking equivariance, and Uniform RoPE, which maintains equivariance, but has only a single shared rotation speed. In Section 5 , we find that Spherical RoPE has the same training behaviors as its equivariant analogues and we find that Uniform RoPE outperforms the standard learned encodings, while performing worse than other RoPE methods. We conclude that our evidence suggests that the performance of RoPE over traditional embeddings is not explained by equivariance. 2 Background In this section, we review concepts and notation from previous work on rotary positional embeddings. We introduce the methods in both historical and progressively general order which we will use to prove in Section 3 that Mixed RoPE is the most general M -D rotary embedding with equivariance. For a broader literature review on positional embeddings see Appendix C . For a compact overview of symbols, see Appendix D . 2.1 Attention We use the standard attention mechanism from 73 , given by \mathbf{Z}=\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{V}. (1) We consider only single-headed attention to simplify notation, so here \mathbf{Q},\mathbf{K} , and \mathbf{V} are elements of \mathbb{R}^{T\times N} , where T is the number of tokens and N is the network’s latent dimension. We will primarily use index notation, where the above equation is expressed as, \mathbf{z}_{i}=\sum_{j=1}^{\top}a(\mathbf{q}_{i},\mathbf{k}_{j})\mathbf{v}_{j}. We define the attention mechanism, a(\mathbf{q}_{i},\mathbf{k}_{j}) , as a(\mathbf{q}_{i},\mathbf{k}_{j})=\frac{e^{\alpha(\mathbf{q}_{i},\mathbf{k}_{j})}}{\sum_{j=1}^{T}e^{\alpha(\mathbf{q}_{i},\mathbf{k}_{j}^{\prime})}}, (2) where what we refer to as the attention score is given by \alpha(\mathbf{q},\mathbf{k})=\mathbf{q}^{\top}\mathbf{k}. (3) This formulation of attention is equivariant to permutations of the token order. To break this symmetry, the position of the tokens must be “encoded" into the attention scores. Thus, we re-express the attention score

Read the complete text