Why Do Accumulated Transformations Extrapolate?
Mahesh Godavarti · 2026
Read Why Do Accumulated Transformations Extrapolate? by Mahesh Godavarti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24975.txt.
Introduction
Mahesh Godavarti explains why one kind of attention extrapolates to longer contexts than it was trained on. PaTH Attention replaced the fixed rotations of rotary position embeddings with accumulated, data dependent Householder reflections. It extrapolated strongly, then degraded at extreme lengths. Godavarti asks whether that pattern needs Householder structure. A variant keeps the rotary blocks but lets the angles accumulate from the tokens. It behaves the same way: better extrapolation, then degradation.
He proves the effect for accumulated orthogonal maps under mild conditions. Their products turn incoherent after finitely many steps, muting attention to tokens. A stylised model makes the mechanism concrete. The accumulated turns create a mixing window that does not grow with context length. The muting learned in training carries unchanged to any test length. A lower bound shows the degradation cannot be fixed. As context grows, no choice of turns can guarantee the target signal survives.
Catalog record
| CREATOR | Mahesh Godavarti |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Do Accumulated Transformations Extrapolate? Mahesh Godavarti Abstract PaTH Attention showed that replacing RoPE’s position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance eventually degrades at extreme context lengths. This paper asks whether that behavior depends on Householder-specific structure or reflects a more general property of accumulated transformations along source-to-query paths. We study a simpler variant that keeps RoPE’s block-diagonal SO ( 2 ) \mathrm{SO}(2) rotations but replaces position-indexed angles with accumulated token-dependent angles. This simpler mechanism exhibits the same qualitative pattern: improved extrapolation followed by degradation at sufficiently long contexts. We prove that the result extends to accumulated orthogonal transformations satisfying certain regularity conditions: their products become incoherent after a finite number of steps, suppressing attention to distant tokens. We analyze the mechanism through a stylized model of attention that explains both behaviors. Accumulated rotations of queries and keys create a finite mixing window independent of context length; the per-token suppression learned during training transfers unchanged to any evaluation length, and high-dimensional concentration produces a score gap that suppresses far-token attention while near-route transport preserves the target signal. This suggests a concrete mechanism linking accumulated transport to length extrapolation. On the other hand, a lower bound shows that accumulated rotations must eventually degrade: as the far set grows with context length, no choice of rotations can guarantee preservation of the near target signal without explicit far-mass control. Furthermore, we show that for SO ( 2 ) \mathrm{SO}(2) rotations, also rotating values makes residual far contributions combine incoherently, extending the extrapolation range. Controlled transformer experiments support these predictions. Random accumulated rotations substantially improve extrapolation over RoPE, learned token-dependent rotations maintain near-training-length perplexity far beyond the training context, and rotating queries, keys, and values improves over rotating queries and keys alone. Rotation-only models still degrade at extreme lengths, while ALiBi remains approximately length-stable, consistent with the need for explicit far-mass control. 1 Introduction PaTH Attention ( 16 ) replaces RoPE’s ( 14 ) position-indexed rotations with products of data-dependent Householder reflections and demonstrates strong extrapolation at 760M parameters: each token contributes a transformation, and the source-query relationship is determined by the product of intervening steps rather than by absolute position. The mechanism behind this established extrapolation phenomenon is not well understood. This paper seeks to shed light on why accumulated transformations help length extrapolation. We also ask whether the mechanism requires Householder’s full generality. We find that even RoPE’s commuting block-diagonal \mathrm{SO}(2) rotations, with accumulated token-dependent angles instead of position-indexed ones, extrapolate and then degrade in the same qualitative pattern. This suggests the mechanism is not specific to any particular orthogonal structure, and indeed we prove that accumulated products of orthogonal transformations satisfying certain regularity conditions become incoherent, suppressing attention to distant tokens. We verify that both \mathrm{SO}(2) rotations and Householder reflections satisfy these conditions. We call the ordered source-to-query path from token j to query i a route . The intervening tokens on this route generate per-token rotations whose product gives the source-query rotation. In RoPE, this rotation is determined entirely by the distance i{-}j ; in accumulated transport, it depends on which tokens lie along the route. Distant routes pass through many independent token-dependent steps and become incoherent, while nearby routes can remain approximately aligned. The boundary between these two regimes is independent of context length, so the model sees the same near/far structure at any evaluation length. This train/test distributional match is a necessary condition for extrapolation, but it is not sufficient by itself. NoPE ( 8 ) provides an instructive counterexample: identity transport is length-stable, but it does not create a score gap that suppresses far tokens. Distributional match alone does not quantify how strongly far tokens are suppressed, whether that suppression can be overwhelmed as context grows, or what happens to far values that survive score selection. The analysis below addresses these questions. We develop a stylized model of attention that explains both the extrapolation and the eventual degradation: 1. Score-side decoherence (mechanism for extrapolation). Accumulated rotations of queries and keys create a finite mixing window whose boundary is independent of context length. Once training length covers this window, the near/far route regime is the same at training and evaluation within the stylized model. Within this regime, high-dimensional concentration produces a score gap that suppresses far-token attention mass, while near-route transport preserves the target signal. We prove that the result holds for any accumulated orthogonal transformation with a spectral gap (Appendix B.5 ). 2. Far-mass lower bound (mechanism for eventual degradation). Per-token suppression is stable across lengths, but the far set grows with context. A lower bound proves that this growth is fundamental: without explicit far-mass control, no choice of rotations can guarantee that the target signal is preserved at unbounded lengths. This bound applies to any orthogonal transport. This is consistent with the flat extrapolation of distance-bias methods such as ALiBi ( 12 ) , which directly control the total far mass that rotations alone cannot bound. In addition, we show that for \mathrm{SO}(2) rotations, rotating values as well as queries and keys extends the extrapolation range, since far values that still receive attention mass combine incoherently, bounding the far contribution’s covariance. We do not aim to reproduce PaTH at scale. Instead, we train small decoder-only transformers with accumulated \mathrm{SO}(2) rotations to isolate the mechanism; the results support each prediction. Random accumulated rotations instantiate the independence and spectral-gap assumptions most directly among our experimental variants and strongly improve extrapolation over RoPE. Rotation-only models degrade gradually at extreme lengths while ALiBi remains flat, consistent with the far-mass requirement. Adding value rotation further reduces long-context degradation, consistent with the theoretical prediction. Learned token-dependent rotations match RoPE at training length and maintain near-training-length perplexity through 16\times the training context. The remainder of the paper follows this progression. Section 2 defines the stylized attention model and the route notation used throughout. Section 3 shows that accumulated rotations produce a content-dependent mixing window and a score gap that forces small total far attention mass, establishing an upper bound on far-token interference. Section 4 proves a lower bound showing that even stable per-token suppression cannot eliminate far-mass leakage as the far set grows; explicit far-mass control (e.g., distance bias) is structurally necessary. Section 5 shows that, for the \mathrm{SO}(2) variant, rotating values as well as queries and keys tightens the upper bound on far-token interference by making the surviving far contributions combine incoherently. Section 6 establishes near-signal preservation and summarizes the combined picture. Section 7 tests the resulting predictions in controlled \mathrm{SO}(2) experiments. Section 8 discusses scope and relation to other positional methods; Section 9 concludes. 2 Preliminaries and Stylized Model 2.1 Transformer attention and the aggregation model For a single attention head, let x_{j}\in\mathbb{R}^{d_{\rm model}} be the residual representation at position j . Standard scaled dot-product attention forms q_{i}=W_{Q}x_{i},\qquad k_{j}=W_{K}x_{j},\qquad v_{j}=W_{V}x_{j}, (1) then computes scores and softmax weights. With an explicit logit scale \lambda>0 , s_{ij}=\frac{q_{i}^{\top}k_{j}}{\sqrt{d_{k}}}+b_{ij},\qquad\alpha_{ij}=\frac{\exp(\lambda s_{ij})}{\sum_{\ell}\exp(\lambda s_{i\ell})}, (2) where b_{ij} may include a causal mask or positional bias. The usual normalization is recovered by taking \lambda=1 ; larger \lambda makes the softmax more selective. The attention-head output before the output projection is o_{i}=\sum_{j}\alpha_{ij}\,v_{j}. (3) Thus attention has a score side, which produces the