Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy · 2025
Read Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings by Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00073.txt.
Introduction
Chunsheng Zuo, Pavel Guerzhoy, and Michael Guerzhoy explain how transformer models can know word order without any built-in position tags, in a paper published in 2024. Many assumed the position signal had to be stamped onto each token. The authors instead point at the geometry of the embeddings themselves.
Nearby tokens end up with more similar vectors than distant ones, in trained and even in freshly initialized causal transformers. That gradient of similarity gives the model a ruler from which positions can be rebuilt. The pattern held across a common range of settings in their tests. The authors offer it as the likely channel for order in tag-free designs.
Catalog record
| CREATOR | Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00073
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy Abstract Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters. 1 Introduction Recent results by Haviv et al. 2022 , Kazemnejad et al. 2024 , and Chi et al. 2023 suggest that positional encodings are not necessary when training decoder-only Transformer language models. These results motivate our investigation of how Transformers might represent positional information without positional encodings. As shown in ( Tsai et al. 2019 ; Zuo and Guerzhoy 2024 ) , the non-causal attention mechanism is equivariant to the permutation of the input tokens — the prediction for input token n+1 is invariant to permutations of tokens 1,2,...,n-1 . Therefore, without positional encodings, the causal attention mechanism is required for the Transformer to consider the order of the input tokens. Chi et al. 2023 hypothesize that causal attention allows positional information to be stored using the variance (taken across the indices of the embedding vector — essentially the norm) of the embeddings, which generally decreases for tokens at later positions. They argue that the variance will tend to decrease because, when using causal attention, embedding n is computed using embeddings 1,2,...,n-1 in the previous layer, whereas embedding n+k will be computed using k more input embeddings, leading to variance shrinkage for embedding n+k . We identify a different possible way of representing positional information that also arises from the fact that embeddings at earlier positions are computed using fewer embeddings from the previous layer compared to those at later positions. Specifically, we observe that embeddings at nearby indices will tend to be more similar to each other (in the sense of cosine similarity). This property could, in principle, enable the reconstruction of a token’s position. The rest of the paper is organized as follows. We briefly review the literature on causal attention’s connection to storing position information in Section 2.1 . We then describe the pattern of nearby embeddings’ being more similar to each other that we refer to as the adjacency pattern , which we later link to the storing of position information in the network. We then present theoretical observations that explain how and why the adjacency pattern arises across many contexts in Section 3 . We confirm through experiments on synthetic data that the pattern we report appears in a variety of configurations, both in trained and untrained architectures that use causal attention, in Section 4 . We demonstrate a range of synthetic tasks where the position is important in Section 4.2 . In Sections 5.1 , 5.2 , and 5.3 , we demonstrate that the pattern of large cosine similarity between nearby embeddings shows up in a variety of settings, for both trained and untrained models. In Section 5.4 , we point out that Chi et al. 2023 ’s theory of the position information’s being stored in the embedding variance is insufficient to explain what we observe in our experiments. In Section 5.5 , we explore the extent to which position information is stored in different Transformer layers, and to what extent that information can be thought of as being stored in the variance and in the adjacency pattern . We discuss our results in Section 6 and discuss some limitations in Section 7 . Figure 1: Self-cosine-similarity matrices of randomly initialized (first row) and trained (second row) 6-layer Transformers with causal attention and no positional encodings on the task of Reversal (22). The matrices are produced using a testing sample of 22 tokens, "rev(8502251258017069)=", as input, showing results from the embeddings to the output of layer 6 left to right for the initialized and trained models. The number in the bracket represents the adjacency probability score. 2 Background 2.1 Transformers with causal attention store position information without positional encodings Mechanisms analogous to modern attention in Transformers have long been used in recurrent neural networks Bahdanau et al. 2014 ; Schmidhuber 1992 . An attention mechanism is central to the Transformer architecture Vaswani et al. 2017 . In a Transformer with “non-causal" attention, an output at the k -th position is agnostic to permutations in the positions of the inputs from other positions, a property known as permutation invariance (the logits above positions 1...k are permutation- equivariant in the input). Without positional encodings, permutation equivariance prevents the output of each layer from taking into account the position of input tokens. In contrast, Tsai et al. 2019 show that Transformers with causal attention are not permutation-equivariant to the input sequence. This implies the possibility of the success of Haviv et al. 2022 in training causal Transformers without positional encodings — “non-causal" attention could not accomplish that. 2.2 The self-cosine-similarity matrix and the adjacency pattern The self-cosine-similarity matrix is a method to visualize the similarity (in the sense of a small angle) between all pairs of vectors within a sequence of embeddings. To create a self-cosine-similarity matrix {\bm{C}} for a sequence of n token embeddings {\bm{X}}\in\mathbf{R^{n\times d}} of dimension d , we define each entry {\bm{D}}_{ij} as the cosine similarity between the i^{th} and j^{th} token embeddings, namely, {\bm{D}}_{ij}=\text{similarity}({\bm{X}}_{i},{\bm{X}}_{j})=\cos\theta_{({\bm{X}}_{i},{\bm{X}}_{j})} . Since the cosine similarity operation is commutative, {\bm{D}}_{ij}={\bm{D}}_{ji} , resulting in the self-cosine-similarity matrix’s being diagonally symmetrical. We use the term adjacency pattern to describe a special type of self-cosine-similarity matrix that we observe. An example of this pattern can be found in Figure 1 , where the matrix is darker (higher values) closer to the diagonal and brighter (lower values) further away, indicating that each embedding vector is more similar to vectors closer to it and less similar to vectors further away from it. The key idea in this paper is that embeddings exhibit an adjacency pattern , meaning position information may, in principle, be partially recoverable from them, as embeddings corresponding to spatially nearby positions tend to be more similar. The self-cosine-similarity matrix is used in ( Wang and Chen 2020 ) to visualize various positional encodings, some of which, such as the sinusoidal embeddings, demonstrate the adjacency pattern . In our work, the self-cosine-similarity matrix is applied to the causal attention’s output embeddings directly in order to examine their adjacency pattern. 3 How the adjacency pattern arises Chi et al. 2023 demonstrate that, in the first hidden layer of the causal Transformer, the variance of the individual coordinate values within embeddings goes down with token index k . They infer that information about the position is related the the variance of the embedding. Chi et al. 2023 explain the decrease in variance by observing that embedding k is computed using a larger and larger context as k grows. In this section, we use the observation to show that we should expect