Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
Kazuki Irie · 2025
Read Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph by Kazuki Irie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00659.txt.
Introduction
Kazuki Irie wrote a paper, published in 2024, revisiting an old and half-forgotten result about how language models know word order. Transformers are often given explicit positional encodings, tags that mark each word's place. Irie shows that deep models which generate text one word at a time do not need them: depth alone lets the network tell apart shuffled word sequences.
The reason is that stacked layers of set-style processing can jointly act in an order-sensitive way during generation. The fact was known around the first GPT models but faded as the field exploded, leading to recent rediscoveries. The paper restates the explanation plainly, with its history, so the record is clear.
Catalog record
| CREATOR | Kazuki Irie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00659
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph Kazuki Irie Abstract Do autoregressive Transformer language models require explicit positional encodings (PEs)? The answer is ‘no’ provided they have more than one layer—they can distinguish sequences with permuted tokens without the need for explicit PEs. This follows from the fact that a cascade of (permutation invariant) set processors can collectively exhibit sequence-sensitive behavior in the autoregressive setting. This property has been known since early efforts (contemporary with GPT-2 ) adopting the Transformer for language modeling ( Irie et al. 2019 ) . However, this result does not appear to have been well disseminated, leading to recent rediscoveries. This may be partially due to a sudden growth of the language modeling community after the advent of GPT-2/3, but perhaps also due to the lack of a clear explanation in prior work, despite being commonly understood by practitioners in the past. Here we review the long-forgotten explanation why explicit PEs are nonessential for multi-layer autoregressive Transformers (in contrast, one-layer models require PEs to discern order information of their inputs), as well as the origin of this result, and hope to re-establish it as a common knowledge. 1 Introduction The field of language modeling has seen new waves of interest after the promising results of GPT-2 ( Radford et al. 2019 ) , impressive capabilities of GPT-3 ( Brown et al. 2020 ) , and unprecedented versatility of ChatGPT and GPT-4 ( Bubeck et al. 2023 ; Achiam et al. 2023 ) , manipulating human languages in a way no machine has ever before. Figure 1: A cascade of set processors can behave as a sequence processor . An illustration of autoregressive Transformers without explicit positional encodings for two input sequences: (a, b, c) and its permutation (b, a, c). The color in the right diagram highlights the differences in terms of context seen by each layer at each position (expressed as a set). With two (or more) layers, as soon as an input is different at one position (here at the first position), autoregressive Transformers see different contexts for all later positions at the top layer. In contrast, one-layer models can not distinguish between these two input sequences (in this example, as soon as the second token is fed) even though they are strictly speaking not permutation invariant (since the context at the first position is different). Here we only swap the two first tokens; with a more “complex” permutation, one-layer models may see different contexts at more positions but in all cases, they systematically fail to do so in the last step. About a decade before the current ‘‘Large Language Model era’’ or LLM-era 1 1 1 Here by “LLM-era” we roughly refer to the time after GPT-3. The term “petroplyph” in the title is a hyperbole with a double meaning: first, it highlights that results from the pre-LLM era are now largely regarded as “prehistoric” and they are overlooked; second, more specifically to the positional encoding result discussed here, figures similar to Figure 1 were frequently sketched in old notes and during whiteboard discussions from that time, but such a figure was not included in the 4-page Interspeech paper ( Irie et al. 2019 ) . , neural language modeling research had also seen a smaller but significant growth after Tomáš Mikolov’s breakthrough results with recurrent neural network language models ( Mikolov et al. 2010 ; Mikolov et al. 2011 ) . This had made neural language modeling ( Nakamura and Shikano 1989 ; Elman 1989 ; Schmidhuber and Heil 1994 ; Bengio et al. 2000 ) a popular research topic, particularly among speech recognition and machine translation researchers as these two fields used to be the major application areas at the time when language models were not yet a standalone system— they were merely a component in a larger system with a specialized application ( Jelinek et al. 1975 ; Brown et al. 1988 ) , except in certain visionary work ( Sutskever et al. 2011 ) . When the Transformer encoder-decoder architecture was shown to be successful for machine translation ( Vaswani et al. 2017 ) , several works investigated its application to build conventional (i.e., autoregressive) language models using the decoder, e.g., Liu et al. 2018 ; Radford et al. 2018 ; Al-Rfou et al. 2019 ; Dai et al. 2019 ; Baevski and Auli 2019 , or non-autoregressive models of language using the encoder, e.g., Devlin et al. 2019 ; producing many methods and practical knowledge for optimizing Transformers to language modeling, concurrently to GPT-2 ( Radford et al. 2019 ) . While the recent surge of interest in language modeling has been very exciting for the field, it has also led to some discontinuities, e.g., certain common knowledge and results from pre-LLM studies appear to have been lost amid this rapid growth. This short review focuses on one of such results, namely the property that multi-layer autoregressive Transformer language models can process sequences without explicit positional encodings ( Irie et al. 2019 ) . In fact, it is often argued that positional encodings are necessary for Transformers, because the self-attention operation ( Vaswani et al. 2017 ; Cheng et al. 2016 ; Parikh et al. 2016 ; Lin et al. 2017 ) is permutation invariant. There is a flaw in this deduction: it overlooks the behavior of multi-layer (i.e., cascaded) self-attention in the autoregressive setting. Here we provide a simple explanation of this result (Figure 1 and Sec. 3 ), which, although known to language modeling practitioners of the pre-LLM era, was never published (to the best of our knowledge). We also refer back to early work on this property (Sec. 5 ). 2 Background: Self-Attention Following the original definition ( Vaswani et al. 2017 ) , one Transformer “layer” consists of two sub-layers: a self-attention layer and a feedforward block. Given that a typical feedforward block processes information at each position/time step exclusively, the self-attention layer is the only sequence processing component of the Transformer layer. Autoregressive Self-Attention. Let d and T denote positive integers. An autoregressive self-attention layer transforms an input sequence \{{\bm{x}}_{t}\}_{t=1}^{T},{\bm{x}}_{t}\in\mathbb{R}^{d} to an output sequence \{{\bm{y}}_{t}\}_{t=1}^{T},{\bm{y}}_{t}\in\mathbb{R}^{d} as follows: \displaystyle{\bm{q}}_{t},{\bm{k}}_{t},{\bm{v}}_{t} \displaystyle= \displaystyle{\bm{W}}_{q}{\bm{x}}_{t},{\bm{W}}_{k}{\bm{x}}_{t},{\bm{W}}_{v}{\bm{x}}_{t} (1) \displaystyle{\bm{K}}_{t} \displaystyle= \displaystyle\big[{\bm{K}}_{t-1},{\bm{k}}_{t}]\in\mathbb{R}^{d\times t} (2) \displaystyle{\bm{V}}_{t} \displaystyle= \displaystyle\big[{\bm{V}}_{t-1},{\bm{v}}_{t}]\in\mathbb{R}^{d\times t} (3) \displaystyle{\bm{y}}_{t} \displaystyle= \displaystyle{\bm{V}}_{t}\mathrm{softmax}({\bm{K}}_{t}^{\top}{\bm{q}}_{t}) (4) where {\bm{W}}_{q},{\bm{W}}_{k},{\bm{W}}_{v}\in\mathbb{R}^{d\times d} are trainable weight matrices, [{\bm{A}},{\bm{a}}] denotes the concatenation of vector {\bm{a}} to matrix {\bm{A}} which increments the time dimension ( {\bm{K}}_{0} and {\bm{V}}_{0} are initially empty), and \mathrm{softmax} is along the time dimension. We omit the 1/\sqrt{d} scaling inside \mathrm{softmax} , as well as the output projection, which are irrelevant for our discussion. While the equations above accurately describe the model conceptually, self-attention is also often expressed in the following matrix form that better reflects the possibility to parallelize computation over the time axis during training. By denoting the input as {\bm{X}}=[x_{1},...,x_{T}]\in\mathbb{R}^{d\times T} ( {\bm{X}}_{i}=x_{i} ) and the output as {\bm{Y}}=[y_{1},...,y_{T}]\in\mathbb{R}^{d\times T} , it yields: \displaystyle{\bm{Q}},{\bm{K}},{\bm{V}} \displaystyle={\bm{W}}_{q}{\bm{X}},{\bm{W}}_{k}{\bm{X}},{\bm{W}}_{v}{\bm{X}} (5) \displaystyle{\bm{Y}} \displaystyle={\bm{V}}\mathrm{softmax}({\bm{M}}\odot({\bm{K}}^{\top}{\bm{Q}})) (6) where {\bm{M}}\in\mathbb{R}^{T\times T} is the so-called attention mask . For these equations to be equivalent to Eqs. 1 - 4 above, i.e., for autoregressive self-attention, {\bm{M}} is set to be the upper triangular matrix, i.e., {\bm{M}}_{i,j}=1 if i\leq