Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE
Sameeksha Sriram, Ayush Paliwal, Alexander S. Ecker, Chase van de Geijn · 2025
Read Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE by Sameeksha Sriram, Ayush Paliwal, Alexander S. Ecker, Chase van de Geijn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11665.txt.
Introduction
Sameeksha Sriram and three co-authors revisit rotary positional embeddings. This widely used method tells transformer models where each token sits. It works by rotating embedding vectors. Extensions to higher dimensions rotate vectors on spheres instead of circles. But sphere rotations do not commute, so rotation order becomes unclear.
Their method, QuatRo, uses quaternions to set the rotation axes. Quaternions are the classic algebra for three-dimensional rotation. The goal is to keep the shift property that makes rotary embeddings work. At the same time, richer spatial inputs should be handled well. For models that reason about three-dimensional scenes, a cleaner rotary code could improve how position is represented.
Catalog record
| CREATOR | Sameeksha Sriram, Ayush Paliwal, Alexander S. Ecker, Chase van de Geijn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE Sameeksha Sriram, Ayush Paliwal, Alexander S. Ecker, Chase van de Geijn Abstract Rotary Positional Embeddings (RoPE) have demonstrated exceptional performance as a positional encoding method, consistently outperforming their baselines. While recent work has sought to extend RoPE to higher-dimensional inputs, many such extensions are non-commutative, thereby forfeiting RoPE’s shift-equivariance property. Spherical RoPE is one such non-commutative variant, motivated by the idea of rotating embedding vectors on spheres rather than circles. However, spherical rotations are inherently non-commutative, making the choice of rotation sequence ambiguous. In this work, we explore a quaternion-based approach—Quaternion Rotary Embeddings (QuatRo)—in place of Euler angles, leveraging quaternions’ ability to represent 3D rotations to parameterize the axes of rotation. We show Mixed RoPE and Spherical RoPE to be special cases of QuatRo. Further, we propose a generalization of QuatRo to Clifford Algebraic Rotary Embeddings (CARE) using geometric algebra. Viewing quaternions as the even subalgebra of C l ( 3 , 0 , 0 ) Cl(3,0,0) , we extend the notion of rotary embeddings from quaternions to Clifford rotors acting on multivectors. This formulation enables two key generalizations: (1) extending rotary embeddings to arbitrary dimensions, and (2) encoding positional information in multivectors of multiple grades, not just vectors. We present preliminary experiments comparing spherical, quaternion, and Clifford-based rotary embeddings. \workshoptitle UniReps 2025 1 Introduction Rotary positional embeddings (RoPE) have proven remarkably effective in language modeling, prompting interest in adapting their success to higher-dimensional domains such as vision and multimodal learning ( 6 ) . Extending RoPE beyond 1D sequences requires careful consideration to preserve properties such as relative positional dependence (shift-equivariance) and reversibility ( 2 ; 8 ) . While early extensions sought to preserve strict equivariance ( 10 ; 5 ) , recent findings suggest this property may not be essential for strong performance, opening the door to non-commutative generalizations ( 9 ) . Spherical RoPE ( 9 ) exemplifies this approach: the rotations are performed on a sphere — a space where rotations fail to commute — thus breaking strict equivariance. Its implementation uses Euler angles (yaw and roll matrices), which explicitly constrain the rotations to be around the principal axes. Rather utilizing the Euler angles, one could parameterize rotations with quaternions, an idea mused in 8 , but discarded due to their non-commutativity. This allows for simple parameterization of the axes of rotation. In this work, we revisit Quaternion Rotary Embeddings (QuatRo). Quaternions offer a compact, stable representation of 3D rotations allowing us to parameterize the axes of rotation rather than the assumed principal axes of Spherical RoPE. We further generalize QuatRo to Clifford Algebraic Rotary Embeddings (CARE), leveraging the geometric algebra framework. By interpreting quaternion rotors as grade-2 blades of Cl(3,0,0) acting on grade-1 vectors, we derive a principled method for: 1. Generalizing rotary embeddings to arbitrary dimensions. 2. Allowing embeddings to inhabit multivector spaces, enabling richer positional transformations across grades. This generalization not only subsumes quaternion-based methods but also creates new possibilities for encoding positional structure in higher-dimensional and multimodal settings. While CARE generalizes QuatRo to higher-dimensional data such as video or point clouds, this work is still in progress, and experiments are currently restricted to 2D images. 2 Related Work Several recent efforts have sought to extend Rotary Positional Embeddings (RoPE) to higher-dimensional data ( 7 ) . The most general formulation to date is LieRE ( 3 ) , which models RoPE as a rotation of D -dimensional query sub-vectors via the exponential of a linear combination of skew-symmetric matrices. Under the standard proof of RoPE’s relative positional property, these generators must commute, as can be seen through the Baker–Campbell–Hausdorff formula. This constraint has led prior work to impose commutativity requirements on the rotation generators to preserve strict shift-equivariance ( 10 ; 5 ; 2 ) . However, recent studies have questioned the necessity of these constraints. In particular, 9 propose Spherical RoPE , which applies rotary encodings on the sphere—a setting where rotations do not commute—showing that performance can remain competitive despite breaking equivariance. Their approach parameterizes rotations using Euler angles, introducing potential issues such as gimbal lock and unintuitive composition behavior. Our method, Quaternion Rotary Embeddings (QuatRo), builds on this line of work by replacing Euler angles with quaternion rotations. Quaternions can be viewed as a compact and numerically stable representation of 3D rotations, corresponding to the even subalgebra of Cl(3,0,0) . While QuatRo can be interpreted as a special case of LieRE with fixed 3\times 3 skew-symmetric generators, our generalization— Clifford Algebraic Rotary Embeddings (CARE)—extends beyond LieRE’s scope. CARE treats rotary embeddings as Clifford rotors acting on multivectors, enabling both higher-dimensional generalization and graded (multi-grade) positional encoding. The relationship between LieRE and CARE is nuanced: in one view, LieRE can be seen as a restricted subclass of CARE where the generators are limited to certain skew-symmetric matrices; in another view, certain CARE configurations reduce to LieRE. 3 Background Due to space constraints, we focus on the specific algebraic tools and notation relevant to our method, and refer the reader to 4 for comprehensive dives into geometric algebra and rotors and 9 for N -D positional encodings. 3.1 Quaternions and Rotors Quaternions form a four-dimensional non-commutative algebra over the real numbers, with basis \{1,\mathbf{i},\mathbf{j},\mathbf{k}\} and multiplication rules: \displaystyle\mathbf{i}^{2}=\mathbf{j}^{2}=\mathbf{k}^{2}=\mathbf{i}\mathbf{j}\mathbf{k}=-1, \displaystyle\mathbf{i}\mathbf{j}=\mathbf{k},\quad\mathbf{j}\mathbf{k}=\mathbf{i},\quad\mathbf{k}\mathbf{i}=\mathbf{j}, with anti-commutativity for distinct basis elements, e.g., \mathbf{j}\mathbf{i}=-\mathbf{i}\mathbf{j} . A quaternion q=a_{0}+a_{\mathbf{i}}\mathbf{i}+a_{\mathbf{j}}\mathbf{j}+a_{\mathbf{k}}\mathbf{k} can be split into a scalar part a_{0} and a vector part \mathbf{v}=a_{\mathbf{i}}\mathbf{i}+a_{\mathbf{j}}\mathbf{j}+a_{\mathbf{k}}\mathbf{k} . Pure quaternions (zero scalar part) can represent 3D vectors, while unit quaternions represent 3D rotations. Given a unit quaternion rotor r=e^{\frac{\theta}{2}\mathbf{u}}=\cos(\theta/2)+\mathbf{u}\sin(\theta/2), where \mathbf{u} is a unit pure quaternion indicating the rotation axis, a vector \mathbf{a} is rotated via \mathbf{a}^{\prime}=r\,\mathbf{a}\,r^{-1}. This provides smooth composition of rotations as they can be composed with the quaternion product r=r_{1}r_{2} . 3.2 Geometric Algebra and Clifford Rotors Quaternions are isomorphic to the even subalgebra of Cl(3,0,0) , making them a special case of \begin{array}[]{c@{\hskip.25cm}|@{\hskip.25cm}c}\textbf{Quaternions}\hfil\hskip 7.11317pt\vrule\hskip 7.11317pt&\textbf{Even-Grade CA}\\ \hline\cr\begin{array}[]{c|rrrr}&1&\boldsymbol{i}&\boldsymbol{j}&\boldsymbol{k}\\ \hline\cr 1&1&\boldsymbol{i}&\boldsymbol{j}&\boldsymbol{k}\\ \boldsymbol{i}&\boldsymbol{i}&-1&\boldsymbol{k}&-\boldsymbol{j}\\ \boldsymbol{j}&\boldsymbol{j}&-\boldsymbol{k}&-1&\boldsymbol{i}\\ \boldsymbol{k}&\boldsymbol{k}&\boldsymbol{j}&-\boldsymbol{i}&-1\end{array}\hfil\hskip 7.11317pt\vrule\hskip 7.11317pt&\begin{array}[]{c|rrrr}&1&e_{12}&e_{23}&e_{13}\\ \hline\cr 1&1&e_{12}&e_{23}&e_{13}\\ e_{12}&e_{12}&-1&e_{13}&-e_{23}\\ e_{23}&e_{23}&-e_{13}&-1&e_{12}\\ e_{13}&e_{13}&e_{23}&-e_{12}&-1\end{array}\end{array} Figure 1: Multiplication tables of quaternions and the even subalgebra of \mathrm{Cl}(3,0,0) , illustrating their isomorphism. Clifford rotors. This perspective allows generalization to higher dimensions by replacing quaternion rotors with the bivectors of Cl(n,0,0) . These generalized rotors perform rotations with the operation, \mathbf{a}^{\prime}=e^{\frac{\theta}{2}\mathbf{B}}~\mathbf{a}~e^{-\frac{\theta}{2}\mathbf{B}}. where e^{\frac{\theta}{2}\mathbf{B}}=\cos(\theta/2)+\mathbf{B}\sin(\theta/2). \mathbf{B} is a unit bivector which defines the plane of rotation, analogous to the unit quaternion. However, in contrast to quaternions, Clifford rotors can act not only on vectors (grade-1) but also on higher-grade elements, enabling a potentially richer positional encoding schemes. 3.3 Rotary Positional Embeddings (RoPE) RoPE encodes absolute positions by applying a position-dependent rotation to query and key vectors in the attention mechanism. For the i th 2D sub-vector of the query and key (x_{i},y_{i}) , RoPE applies a rotation by angle \theta_{p} determined by the position p : \mathbf{q}^{\prime}_{i}=\begin{bmatrix}x_{i}^{\prime}\\ y_{i}^{\prime}\end{bmatrix}=\begin{bmatrix}\cos\theta_{i}p&-\sin\theta_{i}p\\ \sin\theta_{i}p&\cos\theta_{i}p\end{bmatrix}\begin{bmatrix}x_{i}\\ y_{i}\end{bmatrix}. In this formulation, p is one dimensional, however it can be extended to 2 D inputs. One method of extending RoPE to 2 D is to perform two spherical rotations to 3D sub-vectors of the query and key. \mathbf{q}^{\prime}_{i}=\begin{bmatrix}x_{i}^{\prime}\\ y_{i}^{\prime}\\ z^{\prime}_{i}\end{bmatrix}=\begin{bmatrix}1&0&0\\ 0&\cos\theta_{i}p_{1}&-\sin\theta_{i}p_{1}\\ 0&\sin\theta_{i}p_{1}&\cos\theta_{i}p_{1}\\ \end{bmatrix}\begin{bmatrix}\cos\theta_{i}p_{2}&-\sin\theta_{i}p_{2}&0\\ \sin\theta_{i}p_{2}&\cos\theta_{i}p_{2}&0\\ 0&0&1\end{bmatrix}\begin{bmatrix}x_{i}\\ y_{i}\\ z_{i}\end{bmatrix}. 4 QuatRo While Spherical RoPE rotates around two