On the Geometry of Positional Encodings in Transformers
Giansalvo Cirrincione · 2026
Read On the Geometry of Positional Encodings in Transformers by Giansalvo Cirrincione (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05217.txt.
Introduction
Cirrincione builds a theory of what place codes ought to do, swapping design. Any model without a place signal cannot solve any task tuned to word order. Training assigns distinct vectors to distinct spots at every best point, under mild rules. The best code match uses multi-D scaling on Hellinger gaps between place spreads, with quality judged by one number, the stress. The code has rank r, at most n minus 1, needing r(n+d) knobs instead of nd.
Catalog record
| CREATOR | Giansalvo Cirrincione |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66249 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05217
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Geometry of Positional Encodings in Transformers Giansalvo Cirrincione Abstract Neural language models process sequences of words, but the mathematical operations inside them — matrix multiplications and attention mechanisms — are insensitive to the order in which words appear. Positional encodings are the component added to remedy this: they inject information about the position of each word into its vector representation. Despite their importance, positional encodings have been designed largely by trial and error, without a mathematical theory of what they ought to do. This paper develops such a theory. Three questions are addressed. First, is positional information strictly necessary? It is proved that any Transformer without a positional signal treats every permutation of the input as equivalent to the original, and therefore cannot solve any task sensitive to word order (Theorem 1 ). Second, what structure does a learned positional encoding acquire? The Positional Separation Theorem (Theorem 4 ) establishes that, under mild and verifiable conditions, training assigns distinct vector representations to distinct sequence positions at every global minimiser. Third, what would an optimal positional encoding look like? Each position in a corpus has a characteristic distribution of words that tend to appear there; the natural criterion for an encoding is to reproduce the statistical distances between these distributions. An exact reproduction is shown to be impossible in general (the relevant geometry is curved), and the best achievable approximation is constructed via classical multidimensional scaling (MDS) on the Hellinger distance between positional distributions (Proposition 6 , Algorithm 1 ). The quality of any encoding — sinusoidal, learned, or relative — is measured by a single number, the stress , which quantifies how faithfully it reproduces the corpus geometry. As a byproduct, a theoretical justification for the widely-used sinusoidal encoding is obtained: it is approximately optimal for corpora whose positional statistics vary smoothly with position. A fourth result identifies the minimal parametrisation of the positional matrix: the information-optimal encoding has effective rank r = rank ( B ) ≤ n − 1 r=\mathrm{rank}(B)\leq n-1 , where B B is the doubly-centred Gram matrix of the Hellinger distances, and can be represented with r ( n + d ) r(n+d) parameters instead of n d nd . On the synthetic corpus of the experiments ( n = 32 n=32 , d = 128 d=128 ), rank r = 3 r=3 suffices to reduce stress by 99.8 % 99.8\% relative to the sinusoidal encoding, using 88 % 88\% fewer parameters than a free positional matrix. Appendix A develops a proof of the Monotonicity Conjecture within the Neural Tangent Kernel (NTK) regime, through five lemmas covering masked language modelling (MLM) losses, sequence classification losses, and general losses satisfying a positional sufficiency condition. Experiments on SST-2 and IMDB with bert base confirm the theoretical predictions, and reveal that Attention with Linear Biases (ALiBi) achieves much lower stress than the sinusoidal encoding and Rotary Position Embedding (RoPE) on both corpora — a finding consistent with a rank- 1 1 interpretation of the MDS encoding under approximate shift-equivariance of the corpus. Submitted to Transactions on Machine Learning Research (TMLR) Keywords: positional encoding, Transformer, Hellinger distance, multidimensional scaling, permutation equivariance, information geometry, Neural Tangent Kernel. 1 Introduction Among the components of the Transformer architecture ( Vaswani et al. 2017 ) , positional encodings occupy a peculiar position. Every other design choice — the query-key-value structure, the softmax normalisation, the residual connections, the layer normalisation — has attracted substantial theoretical scrutiny in recent years. Positional encodings have not. The original paper proposes two variants — a fixed sinusoidal scheme and a learned alternative — notes that they perform comparably, and moves on. No theoretical argument is given for why either should work, what properties a good encoding ought to have, or what the learning algorithm discovers when the encoding is treated as a free parameter. This absence of theory has practical consequences. The field has since produced a proliferation of positional encoding schemes — RoPE ( Su et al. 2024 ) , ALiBi ( Press et al. 2022 ) , and others — each motivated by empirical performance or architectural convenience, without a common theoretical framework. The question of what a positional encoding ought to do, stated precisely enough to admit a proof, has not been addressed. This paper addresses it. A mathematical theory is developed, organised around three questions. Is positional information necessary? The answer is yes. A Transformer without any positional signal computes a function equivariant to permutations of the input: reordering the tokens produces a correspondingly reordered output, with no ability to distinguish the original from any permuted sequence. Any task requiring sensitivity to word order is beyond the reach of such a model (Theorem 1 ). What does training learn? When the positional encoding is a learnable matrix P\in\mathbb{R}^{n\times d} ( n positions, d dimensions) optimised by gradient descent, the Positional Separation Theorem (Theorem 4 ) states that every minimiser assigns distinct embedding vectors to distinct positions, under three conditions that are generically satisfied in practice. This is complemented by the Monotonicity Conjecture (Conjecture 5 ), which posits that the geometry of P^{*} reflects the statistical distances between positions in the corpus. What would be optimal? The question is whether a positional encoding can be constructed, independently of training, that faithfully represents the statistical structure of the corpus. An exact isometry is shown to be unattainable in general — the relevant statistical manifold is curved — and the best approximation is constructed via classical multidimensional scaling on the Hellinger metric (Proposition 6 , Algorithm 1 ). The stress criterion measures how well any encoding reproduces the corpus geometry. As a byproduct: the sinusoidal encoding is the MDS optimum for corpora with smooth positional statistics, providing the theoretical justification the original paper lacked. An important clarification: the stress criterion measures geometric faithfulness — how well an encoding reproduces the statistical distances between positions — not predictive superiority . A lower-stress encoding is not guaranteed to yield better downstream accuracy; two encodings can have very different stress values while performing comparably on a given task. The stress criterion is a principled geometric diagnostic, not a performance predictor. Experiments. The theory is validated on a synthetic corpus with controlled positional structure and on two real-world sentiment datasets (SST-2 and IMDB) with bert base . Five encoding types are compared (sinusoidal, RoPE, ALiBi, MDS, random); stress is measured as a function of embedding dimension; the Positional Separation Theorem is verified for both scratch-trained and pre-trained models; and the Monotonicity Conjecture is tested via direct violation counting. Relationship to companion work. This paper is part of a broader mathematical programme whose goal is to derive the Transformer from first principles. The connection between positional encodings and the symmetric-antisymmetric decomposition M=M_{s}+M_{a} of the attention weight matrix, which gives a complementary algebraic perspective, is developed in Bonino et al. 2025 . Hierarchy of contributions. The four results of this paper form a hierarchy. Theorem 1 is a necessary baseline: without positional signal, no order-sensitive task is solvable. Theorem 4 characterises what training cannot do: it cannot collapse two positions to the same embedding at a global minimiser. Proposition 6 is the core constructive contribution: it identifies the