Rethinking RoPE: A Mathematical Blueprint for N-dimensional Rotary Positional Embedding

Haiping Liu, Lijing Lin, Jingyuan Sun, Zhegong Shangguan, Hongpeng Zhou * · 2025

Read Rethinking RoPE: A Mathematical Blueprint for N-dimensional Rotary Positional Embedding by Haiping Liu, Lijing Lin, Jingyuan Sun, Zhegong Shangguan, Hongpeng Zhou * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06308.txt.

Description

Rotary position embedding, or RoPE, lets large language models sense word order by rotating parts of their internal vectors. Haiping Liu and co-authors ask what RoPE must look like beyond one dimension, for inputs like images where designs so far have been improvised. Using Lie group theory, they derive the exact conditions any such scheme must meet. Two demands drive them: only relative positions should matter, and the encoding should be reversible.

Within that frame, valid schemes correspond to a special commutative corner of the rotation algebra. The common block-diagonal design used today turns out to be its maximal case. Mixing between axes becomes a change of basis that can be learned, and experiments favor balancing that mixing against local structure.

Catalog record

CREATORHaiping Liu, Lijing Lin, Jingyuan Sun, Zhegong Shangguan, Hongpeng Zhou *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06308

Browse the full library · About the Tower of Babel