Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

Yin Li · 2025

Read Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization by Yin Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06398.txt.

Introduction

Yin Li, writing alone, analyzes how position codes shape transformer power in this 2025 paper. Transformers need position signals because they lack built-in sequence order. Schools of coding include sine waves, learned vectors, relative offsets, and bias tricks.

Li builds a theory frame that scores each style on three axes. Those axes are what functions the model can express, how well it generalizes, and how far it stretches to longer inputs. New codes based on wavelets and Legendre polynomials are proposed in the same frame. Tests on synthetic sequence tasks showed the new codes generalize and stretch better than classic sine codes. Design guidance for many fields follows.

Catalog record

CREATORYin Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization Yin Li Abstract Positional encodings are a core component of transformer-based architectures, enabling such models to process sequential data without recurrence. Despite their critical role, the theoretical properties of various positional encoding schemes—including sinusoidal, learned, relative, and recent bias-based methods such as Attention with Linear Biases (ALiBi)—remain poorly understood. In this paper, we present a comprehensive theoretical framework to analyze how different positional encodings affect a transformer’s expressiveness, generalization ability, and extrapolation to sequences longer than those seen during training. We derive formal definitions of expressiveness in terms of function approximation classes, obtain generalization bounds under different encoding schemes using Rademacher complexity analyses, and propose several novel positional encoding methods based on orthogonal function families (e.g., wavelets, Legendre polynomials) and information-theoretic criteria. We also characterize the extrapolation capacity of existing and proposed encodings, extending ALiBi’s biasing approach to a more unified theoretical setting. Our lightweight experimental evaluation on synthetic sequence-to-sequence tasks validates key theoretical predictions, showing that encoding schemes grounded in orthogonal transforms can outperform standard sinusoidal encodings in both generalization and extrapolation. This work fills an important gap in transformer theory, offering new insights that can guide design choices in natural language processing, computer vision, and other domains where transformers dominate. 1 Introduction Transformer architectures ( Vaswani et al. 2017 ) have become foundational in numerous areas of machine learning, including natural language processing (NLP), computer vision, and time-series modeling. By eschewing recurrence and convolution in favor of a self-attention mechanism, transformers achieve superior parallelism and scaling properties. However, the lack of inherent sequential processing necessitates the incorporation of positional encodings (PEs) to inject information about the order of tokens. The original transformer paper introduced sinusoidal positional encodings, and subsequent work explored learned absolute encodings ( Vaswani et al. 2017 ) , relative encodings ( Shaw et al. 2018 ) , and bias-based schemes such as Attention with Linear Biases (ALiBi) ( Press et al. 2021 ) . Despite widespread empirical adoption, the theoretical understanding of how PEs influence a transformer’s expressiveness, generalization, and extrapolation capabilities is limited. This paper proposes a thorough theoretical analysis of positional encodings, addressing the following overarching questions: • Expressiveness: What classes of sequence-to-sequence functions can a transformer approximate under different PE schemes? Are there inherent limitations imposed by specific encodings? • Generalization: How do PEs affect a transformer’s ability to generalize from training to unseen data, especially when sequence lengths vary? Can we derive generalization bounds that capture the influence of different PEs? • Extrapolation: Why do certain encoding methods (e.g., ALiBi) facilitate extrapolation to longer sequences? Can we formalize this phenomenon and propose new encodings that further enhance extrapolation? • Novel Encodings: Are there theoretically motivated PE schemes—based on wavelet transforms, Legendre polynomials, or information-theoretic principles—that surpass existing methods in expressiveness, generalization, or extrapolation? By developing a unified mathematical framework to answer these questions, we aim to provide deep insights into why PEs work, how they can be improved, and what limitations current methods entail. We also propose new PE schemes grounded in orthogonal function families (e.g., wavelets, Legendre polynomials) and analyze their theoretical properties. Lightweight experiments on small-scale synthetic tasks corroborate our theoretical findings, demonstrating that wavelet-based encodings can yield superior extrapolation performance compared to standard sinusoidal PEs. Contributions. Our main contributions are: 1. Expressiveness Characterization: We formally define the expressiveness of transformer models under different PE schemes, showing how absolute, relative, and bias-based encodings impact the set of sequence-to-sequence functions that can be approximated. 2. Generalization Bounds: Using tools from statistical learning theory (Rademacher complexity, covering numbers), we derive generalization bounds that explicate the role of PEs in controlling model capacity and overfitting for varying sequence lengths. 3. Extrapolation Analysis: We extend the theoretical understanding of ALiBi’s biasing mechanism, provide a unified extrapolation framework for bias-based PEs, and identify the limits of extrapolation for alternative encoding schemes. 4. Novel PE Schemes: We propose several novel positional encodings based on orthogonal functions (e.g., wavelets, Legendre polynomials) and information-theoretic criteria (maximizing mutual information between positions). We analyze their expressiveness, generalization, and extrapolation properties. 5. Lightweight Validation: We implement the proposed PE schemes in pure NumPy to run small-scale experiments on synthetic tasks designed to test extrapolation and generalization. These confirm our theoretical predictions without requiring GPUs. The rest of this paper is structured as follows. Section 2 reviews existing PE methods. Section 3 defines and analyzes expressiveness under different PEs. Section 4 derives generalization bounds. Section 5 provides a theoretical framework for extrapolation. Section 6 introduces novel PE schemes and analyzes their properties. Section 7 details lightweight experimental validation. Finally, Section 8 discusses implications, limitations, and future work. 2 Background and Related Work This section reviews transformer architectures and existing positional encoding methods, emphasizing their empirical performance and the lack of deep theoretical analysis. 2.1 Transformer Architecture The transformer model ( Vaswani et al. 2017 ) processes an input sequence of length N by first mapping each token to a d_{\text{model}} -dimensional embedding. Denote the input embeddings by \mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{N}]^{\top}\in\mathbb{R}^{N\times d_{\text{model}}} . Because the transformer is permutation-invariant, positional encodings \mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{N}]^{\top}\in\mathbb{R}^{N\times d_{\text{model}}} are added to \mathbf{X} so that the model can utilize order information: \mathbf{Z}^{(0)}=\mathbf{X}+\mathbf{P},\quad\mathbf{Z}^{(0)}\in\mathbb{R}^{N\times d_{\text{model}}}. The transformer then applies L layers of multi-head self-attention and position-wise feed-forward networks to compute contextualized representations \mathbf{Z}^{(L)}\in\mathbb{R}^{N\times d_{\text{model}}} . We omit further architectural details; see ( Vaswani et al. 2017 ) for a complete description. 2.2 Sinusoidal and Learned Positional Encodings In the original transformer, ( Vaswani et al. 2017 ) introduced sinusoidal absolute positional encodings defined elementwise as \displaystyle\mathrm{PE}(pos,2i) \displaystyle=\sin\left(\frac{pos}{10000^{\frac{2i}{d_{\mathrm{model}}}}}\right), (1) \displaystyle\mathrm{PE}(pos,2i+1) \displaystyle=\cos\left(\frac{pos}{10000^{\frac{2i}{d_{\mathrm{model}}}}}\right), (2) where pos\in\{0,1,\ldots,N-1\} and i\in\{0,1,\ldots,\frac{d_{\mathrm{model}}}{2}-1\} . These encodings allow the transformer to learn to attend to relative positions because any linear combination of two sinusoidal vectors encodes relative shifts. The paper also considered learned positional encodings, where a trainable embedding matrix \mathbf{P}\in\mathbb{R}^{N\times d_{\mathrm{model}}} is optimized jointly with the model. Empirically, learned and sinusoidal encodings perform similarly on standard benchmarks. 2.3 Relative Positional Encodings Relative positional encodings incorporate the distance between pairs of tokens into the attention mechanism, rather than injecting absolute position vectors. ( Shaw et al. 2018 ) add a learnable embedding \mathbf{R}_{i-j} to the attention logits whenever computing attention between tokens at positions i and j . Such encodings can reduce the reliance on absolute position signals and improve performance on tasks with long-range dependencies. Alternative relative schemes include ( Huang et al. 2020 ) and ( Raffel et al. 2020 ) . 2.4 Attention with Linear Biases (ALiBi) ( Press et al. 2021 ) introduced ALiBi, which adds a bias term to attention logits proportional to the negative distance between tokens: b(i,j)=-\alpha\,|i-j|, (3) where \alpha>0 is a slope hyperparameter. This bias encourages the model to pay more attention to nearby tokens while still preserving a form of positional information. Crucially, ALiBi enables extrapolation to sequences longer than those used during training because the bias formula applies uniformly to any token distance. Empirical results show improved performance on long-context tasks without modifying the model’s architecture or retraining. 2.5 Other Positional Encoding Variants Several other PE schemes have been proposed: • Rotary Positional Embeddings (RoPE) ( Su et al. 2021 ) : Applies a rotation in embedding space to

Read the complete text