How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models

Mohamed El Amine Seddik · 2026

Read How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models by Mohamed El Amine Seddik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06826.txt.

Introduction

Mohamed El Amine Seddik studies how attention pooling affects signal recovery in sequence models, using random matrix theory. Token vectors are drawn from a two-class Gaussian mixture and pooled with fixed attention weights, then summed up by a sample covariance matrix. As sizes grow he derives the limiting spread of eigenvalues, the outlier ones, and how eigenvectors align with the hidden signal.

Signal recovery passes through two sharp phase shifts governed by dimension ratios and attention-dependent numbers. The optimal attention weights turn out to be the top eigenvector of the position correlation matrix. As a special case, untuned causal attention beats plain mean pooling when early tokens carry more signal. Tests match the theory closely.

Catalog record

CREATORMohamed El Amine Seddik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06826

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models Mohamed El Amine Seddik Abstract We study the spectral properties of sample covariance matrices constructed from pooled sequence representations, where token embeddings are drawn from a fixed two-class Gaussian mixture table and pooled via (fixed) attention weights. Working in the high-dimensional regime d , V , N → ∞ d,V,N\to\infty with d / V → δ d/V\to\delta and d / N → γ d/N\to\gamma , we derive exact characterizations of the limiting eigenvalue distribution, outlier eigenvalues, and eigenvector alignment with the hidden signal. The bulk spectrum follows a non-Marchenko–Pastur law given by the free multiplicative convolution κ ⁡ ( MP δ ⊠ MP γ ) \kappa(\mathrm{MP}_{\delta}\boxtimes\mathrm{MP}_{\gamma}) , reflecting the finite vocabulary structure. Signal recovery undergoes two successive BBP-type phase transitions characterized by the scalars: δ , γ , α = 𝒘 ⊤ ​ 𝑹 ​ 𝒘 \delta,\gamma,\alpha={\bm{w}}^{\top}{\bm{R}}{\bm{w}} and κ = ‖ 𝒘 ‖ 2 \kappa=\|{\bm{w}}\|^{2} , where 𝒘 {\bm{w}} denotes the attention pooling weights and 𝑹 {\bm{R}} the positional correlation matrix. An aftermath of our analysis demonstrates that the optimal attention weights maximizing the signal-to-noise ratio α / κ \alpha/\kappa are given by the (normalized) top eigenvector of 𝑹 {\bm{R}} , and we show (as a particular case of our analysis) that parameter-free causal self-attention with τ / d \tau/d score scaling yields deterministic harmonic weights that improve signal recovery over mean pooling whenever early tokens carry more signal. Extensive simulations confirm sharp agreement between theory and finite-dimensional experiments. 1 Introduction The attention mechanism ( Vaswani et al. 2017 ) is the central building block of modern transformer architectures that have achieved remarkable success across language modeling ( Devlin et al. 2019 ; Brown et al. 2020 ) , vision ( Dosovitskiy et al. 2020 ) , and many current AI applications. Despite this empirical success, a precise theoretical understanding of why and how attention helps remains limited. Most existing theoretical analyses focus on expressivity ( Yun et al. 2019 ) , optimization landscapes ( Sahiner et al. 2022 ) , or rank collapse phenomena ( Dong et al. 2021 ) . A complementary and largely unexplored direction is to study attention through the lens of statistical signal recovery : given a structured data model, how does the choice of attention weights affect the ability to recover a latent signal from high-dimensional observations? Random matrix theory (RMT) provides a natural framework for this question. The well known BBP phase transition within the random matrix community ( Baik et al. 2005 ; Baik & Silverstein 2006 ) characterizes when a hidden signal becomes detectable from the spectrum of a sample covariance matrix, and the associated eigenvector overlap formulas ( Paul 2007 ; Benaych-Georges & Nadakuditi 2011 ) quantify the alignment between the estimated and true signal directions. These tools have been successfully applied to high-dimensional statistics ( Johnstone 2001 ; Johnstone & Lu 2009 ) , wireless communications ( Couillet & Debbah 2011 ) , and more recently to understanding the high-dimensional behaviour of machine learning algorithms and deep learning representations ( Pennington & Worah 2017 ; Seddik et al. 2020 ; Loureiro et al. 2021 ; Couillet & Liao 2022 ) . In this work, we propose a tractable statistical model for sequence data with a fixed two-class Gaussian mixture embedding table and positional correlations, and we study how attention-based pooling affects signal recovery. Our main contributions are: (1) the exact limiting spectral distribution of the sample covariance of the pooled representations in the high-dimensional regime, characterized by a non-Marchenko–Pastur law (Theorem 1 ); (2) two successive BBP-type phase transitions for signal detectability, at the population and sample levels, with explicit outlier eigenvalue and eigenvector overlap formulas (Theorems 2 – 4 ); (3) a general optimality result showing that the attention weights maximizing signal recovery are the (normalized) top eigenvector of {\bm{R}} (Theorem 5 ); (4) as a particular case of our general framework, a concrete analysis of parameter-free causal self-attention, yielding deterministic harmonic weights that provably improve \alpha/\kappa over mean pooling when the signal is carried on early sequence positions (Section 5 ); and (5) extensive simulations validating all theoretical predictions (Section 6 ). Related work. The Baik-Ben Arous-Péché (BBP) phase transition was established by Baik et al. 2005 for complex spiked Wishart matrices and extended by Baik & Silverstein 2006 . Eigenvector overlap formulas were derived by Paul 2007 and generalized by Benaych-Georges & Nadakuditi 2011 ; Benaych-Georges & Nadakuditi 2012 . The Marchenko–Pastur law ( Marčenko & Pastur 1967 ) and free probability ( Nica & Speicher 2006 ; Mingo & Speicher 2017 ) underpin our bulk analysis, with comprehensive treatments in Bai et al. 2010 ; Couillet & Liao 2022 . On the ML side, Pennington & Worah 2017 initiated RMT for deep learning. Theoretical analyses of attention span expressivity ( Yun et al. 2019 ) , rank collapse ( Dong et al. 2021 ) , inductive biases ( Edelman et al. 2022 ) , and kernel limits ( Hron et al. 2020 ) . Closer to our focus on the statistical role of softmax attention, Duranthon et al. 2025 prove a statistical advantage of softmax attention over linear alternatives in a single-location regression setting, while Boncoraglio et al. 2026 study the generalization, weights spectra and scaling laws of a single-head attention layer in high dimensions using tools closely related to ours. A complementary line of work ( Jha & Reagen 2025 ) analyzes the interaction between trained attention and statistical structure in a high-dimensional spiked-data setup. The latent-variable perspective on embeddings by Arora et al. 2016 further motivates our spiked embedding model. To the best of our knowledge, our work is the first to provide an exact RMT analysis of how attention weights modulate signal recovery under the considered setup of a fixed embedding table. 2 Sequence Statistical Model 2.1 Two-class embedding table Let {\bm{\mu}}\in{\mathbb{R}}^{d} be a deterministic vector with direction {\bm{u}}:={\bm{\mu}}/\|{\bm{\mu}}\| . We consider a two-class fixed embedding table {\bm{E}}=[{\bm{e}}_{1},\ldots,{\bm{e}}_{V}]\in{\mathbb{R}}^{d\times V} with \displaystyle{\bm{e}}_{v}=s_{v}{\bm{\mu}}+{\bm{z}}_{v},\quad v\in[V], (1) where (s_{v})_{v=1}^{V}\in\{\pm 1\}^{V} are balanced signs ( V/2 entries +1 and V/2 entries -1 ), {\bm{z}}_{v}\sim{\mathcal{N}}({\bm{0}},{\bm{I}}_{d}) are i.i.d. noise vectors, and {\bm{Z}}=[{\bm{z}}_{1},\ldots,{\bm{z}}_{V}]\in{\mathbb{R}}^{d\times V} . The embedding table is a two-class Gaussian mixture of means \pm{\bm{\mu}} with \|{\bm{\mu}}\| controlling class separability. Remark 1 (Relaxing Gaussianity) . The Gaussian assumption on the noise vectors {\bm{z}}_{v} is made for convenience and can be substantially relaxed. Standard universality results in random matrix theory ( Bai et al. 2010 ; Couillet & Liao 2022 ) imply that the bulk spectrum of {\bm{S}} and the location of its outlier eigenvalue are insensitive to the precise distribution of the {\bm{z}}_{v} ’s, provided each coordinate has zero mean, unit variance, and a bounded fourth-order moment. All our main results therefore extend straightforwardly to this broader class of embedding-noise distributions. 2.2 Sequence data We are given N i.i.d. sequences of length T . For each sample n\in[N] and position t\in[T] , draw \xi_{n,t}\in\{\pm 1\} such that \mathbb{E}\left[\xi_{n}\xi_{n}^{\top}\right]={\bm{R}} for some positional correlation matrix {\bm{R}}\in{\mathbb{R}}^{T\times T} , and draw tokens x_{n,t} uniformly from \{v:s_{v}=\xi_{n,t}\} . The embedded sequence is {\bm{X}}^{(n)}=[X_{1}^{(n)},\ldots,X_{T}^{(n)}] with X_{t}^{(n)}:={\bm{e}}_{x_{n,t}} , and we study the sample covariance {\bm{S}}:=\frac{1}{N}{\bm{C}}{\bm{C}}^{\top}\in{\mathbb{R}}^{d\times d} , where {\bm{C}}=[{\bm{c}}_{1},\ldots,{\bm{c}}_{N}]\in{\mathbb{R}}^{d\times N} filled

Read the complete text