Context-aware Rotary Position Embedding

Ali Veisi, Delaram Fartoot, Hamidreza Amirzadeh · 2025

Read Context-aware Rotary Position Embedding by Ali Veisi, Delaram Fartoot, Hamidreza Amirzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23083.txt.

Introduction

Ali Veisi, Delaram Fartoot, and Hamidreza Amirzadeh upgrade how word order is fed to Transformer models in a 2025 arXiv paper. The widely used rotary method spins word codes by fixed wave rates. Those rates ignore what the words say, which can blunt links that hinge on context. Their method, CARoPE, sets a fresh rate set per attention head from the word codes at hand.

Bounded shifts keep the spin stable while rates adapt to each text. Tests on a large web text set used GPT-2 class models. The new method beat the rotary method and other common picks on guess scores, even at long spans. Training also ran faster with no loss of stability.

Catalog record

CREATORAli Veisi, Delaram Fartoot, Hamidreza Amirzadeh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context-aware Rotary Position Embedding Ali Veisi, Delaram Fartoot, Hamidreza Amirzadeh Abstract Positional encoding is a vital component of Transformer architectures, enabling models to incorporate sequence order into self-attention mechanisms. Rotary Positional Embeddings (RoPE) have become a widely adopted solution due to their compatibility with relative position encoding and computational efficiency. However, RoPE relies on static, input-independent sinusoidal frequency patterns, limiting its ability to model context-sensitive relationships. In this work, we propose CARoPE (Context-Aware Rotary Positional Embedding), a novel generalization of RoPE that dynamically generates head-specific frequency patterns conditioned on token embeddings. This design introduces token- and context-sensitive positional representations while preserving RoPE’s efficiency and architectural simplicity. CARoPE computes input-dependent phase shifts using a bounded transformation of token embeddings and integrates them into the rotary mechanism across attention heads. We evaluate CARoPE on the FineWeb-Edu-10B dataset using GPT-2 variants trained on next-token prediction tasks. Experimental results show that CARoPE consistently outperforms RoPE and other common positional encoding baselines, achieving significantly lower perplexity, even at longer context lengths. Additionally, CARoPE enables faster training throughput without sacrificing model stability. These findings demonstrate that CARoPE offers a scalable, expressive, and efficient upgrade to existing positional encoding strategies in Transformer models. 1 Introduction Transformer architectures have revolutionized the field of deep learning Vaswani 2017 , achieving state-of-the-art performance across a wide range of tasks in natural language processing Devlin et al. 2019 ; Liu 2019 ; Chowdhery et al. 2023 ; Team et al. 2023 ; Touvron et al. 2023 ; Achiam et al. 2023 . A key component of their success is the self-attention mechanism, which enables the model to dynamically capture relationships between elements in a sequence, regardless of their distance. However, unlike traditional sequence models such as Recurrent Neural Networks (RNNs) Sherstinsky 2020 or Convolutional Neural Networks (CNNs) Gehring et al. 2017 , transformers lack an inherent sense of order or position Yun et al. 2019 . This makes positional encoding a crucial component, as it injects position-related information into the model to enable sequence-aware processing. Over the years, several strategies for positional encoding have been proposed. These include fixed sinusoidal embeddings Vaswani 2017 , learnable absolute position embeddings Devlin et al. 2019 , relative position encodings Press et al. 2021 ; Raffel et al. 2020 , and rotary positional embeddings (RoPE) Su et al. 2024 . Among these, RoPE has become one of the most widely adopted approaches due to its compatibility with self-attention and ability to encode relative positions through rotation-based transformations. RoPE works by rotating the query and key vectors within the multi-head attention mechanism using fixed sinusoidal frequencies. Although effective, RoPE still relies on predefined static frequency patterns that are uniform across different inputs and attention heads. As a result, it remains position-dependent but not token- or context-dependent, limiting its expressiveness in modeling more nuanced sequence structures. In this work, we propose CARoPE ( Context-Aware Rotary Positional Embedding ), a novel enhancement of RoPE that introduces dynamic, input-dependent frequency values for each attention head. By making frequency generation sensitive to the input content, CARoPE enables the model to adaptively encode positional information in a way that reflects both the position and the underlying context. This results in more expressive and flexible positional representations that are conditioned on the input context and vary across attention heads. Unlike RoPE’s fixed sinusoidal formulation, CARoPE learns a nonlinear transformation of the input embeddings to generate head-specific frequency patterns, which are then integrated into the rotary positional mechanism. This context-aware extension enables richer, token-sensitive position encoding without sacrificing the efficiency and compatibility of the original RoPE framework. We assess the effectiveness of our approach across multiple benchmark datasets, employing GPT-2 variants for the standard next-token prediction task. CARoPE consistently outperforms existing positional encoding methods, including RoPE, and achieves lower perplexity in generated sequences. 2 Proposed Method We formulate CARoPE as a generalization of Rotary Positional Embedding (RoPE), designed to introduce context-dependent positional modulation within the attention mechanism. While RoPE encodes relative position through fixed sinusoidal rotations, CARoPE replaces these static frequencies with dynamic, token- and head-specific alternatives. To motivate our method, we first reinterpret standard RoPE through the lens of phase accumulation. In RoPE, the position-dependent rotation applied to each embedding pair is defined as: \phi_{i}(m)=m\cdot\theta_{i}, where m is the sequence position and \theta_{i}=10000^{-2i/d} is the fixed frequency assigned to the i -th embedding pair in a d -dimensional space. This can be reformulated as a cumulative sum: \phi_{i}(m)=\sum_{t=1}^{m}\theta_{i}. Noting that \theta_{i} follows a geometric progression, the phase term becomes: \phi_{i}(m)=\sum_{t=1}^{m}\theta_{1}^{i}=m\cdot\theta_{1}^{i}, revealing that each rotational component increases exponentially with dimension. CARoPE generalizes this formulation by replacing the fixed base frequency \theta_{1} with a learned, input-dependent function f(x_{t}) , where x_{t}\in\mathbb{R}^{d} is the embedding of the token at position t . The generalized phase term becomes: \phi_{i}^{(h)}(m)=\sum_{t=1}^{m}f(x_{t})_{h}^{i}, where h indexes the attention head, and f(x_{t})_{h}\in(0,1) is a learned, bounded scalar frequency specific to head h and token x_{t} . This formulation maintains the exponential dimension-wise progression of RoPE but allows the frequency to vary across both tokens and heads, yielding context-aware phase accumulation. The frequency modulation function f is implemented as: f(x_{t})=\frac{1}{\text{softplus}(x_{t}W)+1}, where W\in\mathbb{R}^{d\times h} projects the token embedding to h scalar values, one per head. The softplus activation ensures positivity, while the inverse squashing maps outputs to the interval (0,1) , promoting stability when raised to higher powers. After computing \phi_{i}^{(h)}(m) for each position, head, and dimension, we construct sinusoidal components: \cos\big(\phi_{i}^{(h)}(m)\big),\quad\sin\big(\phi_{i}^{(h)}(m)\big), which are then applied to the query and key vectors using the standard RoPE formulation. To preserve stability and enable efficient training, we initialize CARoPE using the standard RoPE formulation. Since RoPE corresponds to a special case of CARoPE. This initialization ensures the model begins with a valid and expressive positional prior. 3 Experiment Setup 3.1 Datasets For training, we use the FineWeb dataset Penedo et al. 2024 , a large-scale dataset (15 trillion tokens) for LLM pretraining, derived from 96 CommonCrawl snapshots. FineWeb has been shown to produce better-performing LLMs than other open pretraining datasets Penedo et al. 2024 . More specifically, we use a 10B sample of the FineWeb-Edu dataset, which consists of 1.3T tokens from educational web pages filtered from the FineWeb dataset. We allocate 9.9B tokens for training and 0.1B for evaluation. For evaluation, we use the test set of FineWeb-Edu. GPT-Small models Sequence Length RoPE CARoPE Learnable Sinusoidal 512 21.31 21.23 21.90 22.14 1024 56.61 21.39 - 166.18 GPT-Tiny models Sequence Length RoPE CARoPE Learnable Sinusoidal 512 29.33 28.99 30.48 30.62 1024 81.27 36.74 - 223.28 Table 1: Perplexity comparison on the FineWeb-Edu-10B evaluation set. The first row reports results from GPT-Small models, and the second row shows results from GPT-Tiny models. All models were trained for 19k steps on the FineWeb-Edu-10B training set with a context length of 512. 3.2 Settings For all next-token prediction tasks, we use the GPT-2 variants Brown et al. 2020 . For the FineWeb-Edu-10B dataset, we use its small version (12 layers, 10 heads, and a hidden dimension of 768) with 124M parameters, and a tiny version of GPT-2 (44M parameters) with 6 layers, 8 heads, and a hidden dimension of 512. The evaluation metric is perplexity (PPL), and we train the models with sequence length of 512. All the

Read the complete text