ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices
Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, Chun Yuan, [.6em] Tsinghua University · 2025
Read ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices by Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, Chun Yuan, [.6em] Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03737.txt.
Introduction
Hao Yu and nine co-authors present a new position encoding method for transformer models in a 2025 paper. Transformers need a way to mark where each token sits in a sequence. A popular method rotates token vectors by fixed angles, but hand set rotations limit what the model can express.
Their method makes the rotation angles trainable, as long as the angle matrices commute with each other. The authors prove that this commuting property is the key condition for scaling and for stable behavior when positions shift. Two trainable designs are given that meet the condition. On a major image benchmark, the method beat the prior best at both standard and higher resolutions.
Catalog record
| CREATOR | Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, Chun Yuan, [.6em] Tsinghua University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03737
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, Chun Yuan, [.6em] Tsinghua University Abstract The Transformer architecture has revolutionized various fields since it was proposed, where positional encoding plays an essential role in effectively capturing sequential order and context. Therefore, Rotary Positional Encoding (RoPE) was proposed to alleviate these issues, which integrates positional information by rotating the embeddings in the attention mechanism. However, RoPE utilizes manually defined rotation matrices, a design choice that favors computational efficiency but limits the model’s flexibility and adaptability. In this work, we propose ComRoPE, which generalizes RoPE by defining it in terms of trainable com muting angle matrices. Specifically, we demonstrate that pairwise commutativity of these matrices is essential for RoPE to achieve scalability and positional robustness. We formally define the RoPE Equation, which is an essential condition that ensures consistent performance with position offsets. Based on the theoretical analysis, we present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation, which significantly improve performance, surpassing the current state-of-the-art method by 1.6% at training resolution and 2.9% at higher resolution on the ImageNet-1K dataset. Furthermore, our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research. To ensure reproducibility, the source code and instructions are available at https://github.com/Longin-Yu/ComRoPE . † † † Corresponding author. This work was done when Shuning Jia was an intern at Tsinghua University. 1 Introduction The Transformer architecture ( Vaswani 2017 ) has been widely adopted across various fields, including Natural Language Processing (NLP) ( Devlin et al. 2019 ; Radford 2018 ; Radford et al. 2019 ; Brown 2020 ; GLM et al. 2024 ) and Computer Vision (CV) ( Dosovitskiy 2020 ) . Moreover, an increasing number of Transformer-based applications ( Zhang et al. 2024 ; Li et al. 2025 ; Chen et al. 2024 ; Zhang et al. 2025b ; Liu et al. 2024 ; Zhang et al. 2025a ; Liu et al. 2023 ; Zhang and Qi 2024 ; Lai et al. 2024 ) continue to demonstrate their effectiveness across a wide range of domains. At the core of the Transformer model lies the attention mechanism, which enables the model to selectively focus on different parts of the input based on relevance, thereby effectively capturing long-range dependencies and contextual relationships. However, since the attention mechanism is insensitive to the fundamental position information, it cannot inherently capture the order of elements in the data. In order to alleviate this issue, positional embeddings are added to the input representation, providing the model with the necessary positional information to process the sequence order and structure effectively. A variety of positional encoding methods have been proposed, which can be divided into two categories: Absolute Positional Encoding (APE) and Relative Positional Encoding (RPE). APE explicitly encodes the absolute position of each token by generating fixed positional embeddings, which are directly added to the input embeddings at the beginning of training Vaswani 2017 . However, APE struggles with handling long sequences and exhibits high sensitivity to positional shifts, limiting the scalability and robustness of models Kazemnejad et al. 2023 . In contrast, RPE does not directly modify input embeddings; instead, it incorporates relative positional information within the attention mechanism, enabling the model to effectively capture positional relationships between tokens. Among all the existing works, Rotary Position Embedding (RoPE) Su et al. 2024 has gained significant attention due to the advantage of applying a rotational transformation to token embeddings. It encodes relative positions by treating each pair of features as coordinates and rotating them by an angle proportional to their position, enabling all tokens to interact within the attention mechanism regardless of distance. However, existing RoPE methods face several key challenges: i) The essential components (i.e., the RoPE matrices) of previous RoPE approaches rely on 2D rotation groups, which simplify computations but consequently restrict their feature projection capabilities, especially in high-dimensional spaces ( Su et al. 2024 ) . ii) Moreover, the majority of the rotation matrices of RoPE require to be manually designed, leading to insufficient capability and suboptimal performance ( Press et al. 2021 ; Raffel et al. 2020b ) . iii) Finally, previous attempts ( Ostmeier et al. 2024 ) to extend the rotation group often prioritize design simplicity, making it difficult to consistently satisfy relative position dependency—a critical property of RoPE that ensures positional robustness against offsets. Our objectives. This work aims to develop a novel RoPE method for Transformers that is both scalable and robust. Specifically, we seek to extend the rotation group from the existing 2D representation to a larger subgroup of the special orthogonal group, allowing for higher degrees of freedom while preserving consistent behavior with respect to position offsets. Unlike existing methods that rely on manually designed non-trainable rotation matrices, which suffer from limited expressiveness and reduced robustness, our approach is designed to offer richer feature representation capabilities. This framework addresses the scalability limitations of current approaches and enhances their robustness against positional transformations. Our contributions. This work introduces ComRoPE, a novel framework that significantly enhances positional encoding in Transformers. ComRoPE leverages trainable angle matrices, extending the RoPE mechanism with higher scalability and robustness. We identify the pairwise commutativity of these matrices as a necessary and sufficient condition for effective positional encoding, thereby unifying various existing RoPE formulations under a single theoretical framework. The contributions are summarized as follows: • We formally define the RoPE function parameterized by angle matrices and prove that pairwise commutativity is a necessary and sufficient condition, offering a unified theory that encompasses several existing RoPE variants. • We introduce ComRoPE, a scalable and robust solution that leverages two types of trainable commuting angle matrix sets as sufficient solutions to the RoPE equation, capturing richer positional representations without the need for manual design. • Our extensive experiments show that ComRoPE surpasses the current state-of-the-art LieRE, achieving a performance increase of 1.6\% at training resolution and 2.9\% at higher resolutions on the ImageNet-1K classification task while delivering strong results across other benchmarks. • We explore further applications of ComRoPE, providing valuable insights for advancing position encoding techniques in future Transformer-based models. 2 Related work 2.1 Position information in attention Transformers ( Vaswani 2017 ) utilize the attention mechanism to capture similarities within sequences. However, they lack inherent sequential information and cannot capture the positional information of each token. To address this limitation, positional encoding ( Devlin et al. 2019 ; Raffel et al. 2020a ; Shaw et al. 2018 ) was introduced. As research progressed, positional encoding generally evolved into two types: APE ( Vaswani 2017 ; Sukhbaatar et al. 2015 ; Devlin et al. 2019 ) and RPE ( Shaw et al. 2018 ; Su et al. 2024 ; Horn et al. 2021 ; Liutkus et al. 2021 ) . ( Vaswani 2017 ) first proposed using APE in the form of sine and cosine functions, effectively representing the positional relationships within the input sequence. This positional encoding method achieved remarkable results in natural language processing, becoming a foundational component for many NLP tasks. However, the