A Random Matrix Analysis of In-context Memorization for Nonlinear Attention Zhenyu Liao, Jiaqing Liu, TianQi Hou, Difan Zou, Zenan Ling Abstract Attention mechanisms have revolutionized machine learning (ML) by enabling efficient modeling of global dependencies across inputs. Their inherently parallelizable structures allow for efficient scaling with the exponentially increasing size of both pretrained data and model parameters. Yet, despite their central role as the computational backbone of modern large language models (LLMs), the theoretical understanding of Attentions, especially in the nonlinear setting, remains limited. In this paper, we provide a precise characterization of the in-context memorization error of nonlinear Attention , in the high-dimensional proportional regime where the number of input tokens n n and their embedding dimension p p are both large and comparable. Leveraging recent advances in the theory of large kernel random matrices, we show that nonlinear Attention typically incurs higher memorization error than linear ridge regression on random inputs. However, this gap vanishes—and can even be reversed—when the input exhibits statistical structure, particularly when the Attention weights align with the input signal direction. Our results reveal how nonlinearity and input structure interact with each other to govern the memorization performance of nonlinear Attention. The theoretical insights are supported by numerical experiments. 1 Introduction $*$ $*$ footnotetext: Equal contribution. $\dagger$ $\dagger$ footnotetext: Author to whom any correspondence should be addressed. Since its introduction, the Transformer architecture has become a cornerstone of modern machine learning (ML) and artificial intelligence (AI) vaswani2017Attentiona , powering large language models (LLMs) such as BERT devlin2019BERT , LLaMA touvron2023LLaMA , and the GPT series openai2024GPT4 . Originally developed for sequence modeling tasks such as machine translation and language modeling, Transformers have demonstrated remarkable versatility and now achieve state-of-the-art performance across a wide range of applications, including those that are not inherently sequential dosovitskiy2020Image . At the heart of this success lies the Attention mechanism, which enables flexible integration of information across positions and scales efficiently with both data and model size. Despite its empirical success, our theoretical understanding of Attention, especially in its nonlinear form, remains limited, particularly in terms of how it learns statistical patterns from high-dimensional input tokens. Recent years have seen increasing use of high-dimensional statistics vershynin2018high , statistical physics carleo2019Machine , and random matrix theory (RMT) couillet2022RMT4ML to derive insights into the design and optimization of large-scale ML models. Rather than producing generalization bounds that can sometimes be loose, e.g., in the over-parametrized regime, high-dimensional analysis offers precise characterizations that explain phenomena such as the neural tangent kernel jacot2018neural , double descent in generalization mei2021generalization ; liao2021random ; nakkiran2020deep ; hastie2022Surprises , and benign overfitting bartlett2020benign ; bartlett2021Deep , which now inform core ML design principles. A brief review of related work is provided in Section 1.2 . Yet, a precise theoretical characterization of nonlinear Attention , particularly on structural inputs , remains largely elusive. The main technical challenges stem from the nonlinearity of the Attention operator and the complex interactions between input tokens and Attention weights via queries, keys, and values. Prior theoretical efforts often rely on restrictive assumptions: focusing on in-context learning by reducing Attention to gradient descent on (generalized) linear model, which only holds under particular weight configuration bai2023Transformers ; lu2025Asymptotic ; assuming simplified Attention matrices (e.g., all-ones noci2022Signal or random Markov matrices naderi2024Mind ), or adopting stylized models from Bayesian learning tiberi2024Dissecting , sequence multi-index models troiani2025Fundamental , or generalized Potts model rende2024Mapping in statistical mechanics. This paper fills this gap by presenting a precise analysis of the in-context memorization of nonlinear Attention applied to structural inputs, in the high-dimensional regime where the input length n and embedding dimension p are both large and comparable. Building upon recent advances in the eigenspectral analysis of nonlinear random kernel matrices, we derive precise expressions for the in-context memorization error (defined in 3 ) of nonlinear Attention under a model where the Attention weights admit a full-plus-low-rank decomposition ( 1 ) and the input tokens follow a signal-plus-noise statistical structure ( 2 ). Our result shows that the memorization error of nonlinear Attention is determined by a system of nonlinear equations involving the dimension ratio p/n , the alignment between input signal and Attention weights, and the nonlinearity (via its two Hermite coefficients). By focusing on this canonical setting, our analysis takes a step forward to unveil the theoretical origin of many visually striking features emerging in modern large-scale ML. 1.1 Our Contribution The main contribution of this paper are summarized as follows. 1. In 1 of Section 3 , we derive a precise characterization of the in-context memorization error ( 3 ) for nonlinear Attention, under a signal-plus-noise model ( 2 ) for the input tokens, in the high-dimensional regime where the sequence n and embedding dimension p are both large and comparable. We show that the Attention memorization error is governed by a system of equations involving the dimension ratio p/n , the interaction between input signal and the Attention weights, and the nonlinearity via two of its Hermite coefficients. 2. In Section 4 , we compare the memorization error of nonlinear Attention to that of ridge regression (given in 2 ). While nonlinear Attention generally incurs slightly higher error than ridge regression for random inputs, this disadvantage disappears—and can even be reversed—for structured inputs, particularly when the Attentions weights are well-aligned to the input signal. We further show that Attention lacking a linear component (i.e., with a vanishing first-order Hermite coefficient) are unable to effectively reduce memorization error as the embedding dimension p and/or the signal-to-noise ratio (SNR) increase, shedding novel light on the critical role of “linear component” in effective memorization. 3. From a technical perspective, we establish in 1 a novel Deterministic Equivalent (see 4 for a formal definition) for the resolvent of a generalized sample covariance matrix (SCM) of the form \mathbf{C}\mathbf{X}\mathbf{X}^{\top}\mathbf{C}^{\top} . This extends classical SCM that has been extensively studied in the literature by considering a population covariance \mathbf{C}=\mathbf{C}(\mathbf{X}) that depends on the input \mathbf{X} , and may be of independent interest beyond the scope of this work. 1.2 Related Work Here, we briefly review related work. A more detailed discussion is provided in Appendix A . Theoretical understanding of Transformer and Attention. Theoretical studies of Transformers have sought to characterize their expressive power and in-context learning (ICL) capabilities. It has been established that Transformers are universal sequence-to-sequence function approximators yun2019Are . A growing body of work has focused on understanding the ICL behavior of Transformers and Attention, that is, their ability to adapt to new downstream tasks from a few example dong2024Survey . Various theoretical perspectives have been proposed, including connections between ICL and implicit Bayesian inference xie2021Explanation , algorithm selection grag2022what ; li2023Transformers ; bai2023Transformers , and gradient-based optimization oswald2023Transformers ; wumany ; zhang2024context ; chen2024transformers ; li2025robustness . However, these analyses often rely on restrictive assumptions or idealized weight configurations. In contrast, our work provides a random matrix analysis of nonlinear Attention that explicitly captures the generic interaction between structured input signals and Attention weights, offering a more flexible and data-dependent understanding of in-context memorization. Memorization of neural networks. Classical results have characterized the memorization capacity of shallow neural networks under various settings baum1988capabilities ; bubeck2020Network , with recent extensions to deep nets park2021Provable ; vardi2021Optimal as well as single-layer Attention mahdavi2023Memorization ; chen2024can . These studies typically focus on worst-case or combinatorial bounds, e.g., on the number of distinct training samples that can be memorized by a network. In contrast, here we focus on the statistical (so average-case) in-context memorization behavior of nonlinear Attention, by considering a structured signal-plus-noise model for the inputs. In particular, our analysis quantifies how the memorization performance depends on the alignment between the Attention weights and the input signal. Random matrix analyses of ML methods. Random matrix theory (RMT) has emerged as a powerful and flexible tool to understand the dynamics and generalization properties of large-scale ML models. It has been successfully applied to shallow pennington2017nonlinear ; liao2018spectrum ; liao2018dynamics ; louart2018random and deep neural networks benigni2019eigenvalue ; fan2020spectra ; pastur2020gauss , and more recently to linear Attention lu2025Asymptotic . These analyses encompass both homogeneous (e.g., standard normal) pennington2017nonlinear ; mei2019generalization and structured (e.g., mixture-type) input data liao2018spectrum ; ali2022random ; mai2025Breakdown . To the best of our knowledge, the present work provides the first precise characterization of the statistical memorization error of nonlinear Attention on structured input, extending RMT analysis to a broader and more realistic class of Attention-based models. 1.3 Notations and Organization of the Paper Notations. Scalars are denoted by lowercase letters, vectors by bold lowercase, and matrices by bold uppercase. For a matrix \mathbf{X}\in{\mathbb{R}}^{p\times n} , we write \mathbf{X}^{\top} for its transpose, \mathbf{x}_{i}\in{\mathbb{R}}^{p} for i th column, and \|\mathbf{X}\| for its spectral norm. We use \mathbf{I}_{p} for the identity matrix of size p . For a vector \mathbf{x}\in{\mathbb{R}}^{p} , its Euclidean norm is given by \|\mathbf{x}\|=\sqrt{\mathbf{x}^{\top}\mathbf{x}} . For a random variable x , we denote its expectation by {\mathbb{E}}[x] . Paper organization. The remainder of the paper is organized as follows. In Section 2 , we introduce the problem setting and define nonlinear Attention along with its in-context memorization error. Section 3 presents our main theoretical results on the high-dimensional characterization of in-context memorization error for nonlinear Attention. In Section 4 , we discuss the implications of our results by contrasting the memorization performance of nonlinear Attention to that of linear ridge regression. Numerical experiments are also provided to support our theoretical findings. Section 5 concludes the paper and outlines future directions. Technical lemmas, proofs, and supplementary materials are deferred to the appendices. 2 Problem Setting and Preliminaries We consider a general class of nonlinear (self-)Attention, which is not necessarily restricted to the standard Softmax formulation. Definition 1 (Nonlinear Attention) . Let \mathbf{X}=[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]\in{\mathbb{R}}^{p\times n} be the embedding of an input sequence of tokens \mathbf{x}_{1},\ldots,\mathbf{x}_{n}\in{\mathbb{R}}^{p} of length n . A (single-head) nonlinear Attention output \mathbf{A}_{\mathbf{X}}\in{\mathbb{R}}^{p\times n} with key, query, and value matrices \mathbf{W}_{K}\in{\mathbb{R}}^{d\times p},\mathbf{W}_{Q}\in{\mathbb{R}}^{d\times p},\mathbf{W}_{V}\in{\mathbb{R}}^{p\times p} and entry-wise nonlinearity f\colon{\mathbb{R}}\to{\mathbb{R}} , is defined as: \mathbf{A}_{\mathbf{X}}=\mathbf{W}_{V}\mathbf{X}f(\mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p})/\sqrt{p}\equiv\mathbf{W}_{V}\mathbf{X}\mathbf{K}_{\mathbf{X}}. (1) 1 generalizes classical Attention ( kim2021Lipschitz ; likhosherstov2023Expressive ) by allowing f to be any nonlinear function, and includes the classical Softmax Attention for f being the (truncated) exponential function as a special case 1 1 1 For Softmax Attention, the Softmax function applies column-wise on the matrix, and 1 corresponds to entry-wise Attention ( wortsman2023Replacing ; ramapuram2024Theory ) . Remarkably, under 1 and for input tokens drawn from the signal-plus-noise model in 2 , taking f to be truncated exponential function leads to approximately the same output \mathbf{A}_{\mathbf{X}} as that using Softmax nonlinearity; see 4 in Appendix B for a detailed discussion. , up to a normalization factor \sqrt{p} instead of \sqrt{d} . Intuitively, the matrix \mathbf{K}_{\mathbf{X}}\equiv f(\mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p})/\sqrt{p}\in{\mathbb{R}}^{n\times n} defines an asymmetric kernel parameterized by \mathbf{W}_{Q},\mathbf{W}_{K} , and captures the pairwise similarly of input tokens. The output \mathbf{A}_{\mathbf{X}} is then obtained by “mixing” the values \mathbf{W}_{V}\mathbf{X} according to the obtained similarities in \mathbf{K}_{\mathbf{X}} . We consider that the product of key and query matrices \mathbf{W}_{K}^{\top}\mathbf{W}_{Q} in 1 writes as the sum of a full rank identity matrix and an asymmetric low-rank (in fact rank-one) matrix as follow. Assumption 1 (Full-plus-low-rank decomposition of Attention weights) . The key and query matrices \mathbf{W}_{K},\mathbf{W}_{Q}\in{\mathbb{R}}^{d\times p} in 1 satisfy, for some given \mathbf{w}_{Q},\mathbf{w}_{K}\in{\mathbb{R}}^{p} , \mathbf{W}_{K}^{\top}\mathbf{W}_{Q}=\mathbf{I}_{p}+\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\in{\mathbb{R}}^{p\times p}. (2) The full-plus-low-rank decomposition for \mathbf{W}_{K}^{\top}\mathbf{W}_{Q} in 1 is largely inspired by the empirical success of Low-Rank Adaption (LoRA) in fine-tuning Transformer-based LLMs ( hu2021LoRA ) . Note that 1 implies that d\geq p , though this condition is not essential and can be relaxed by considering block decomposition of \mathbf{W}_{K}^{\top}\mathbf{W}_{Q} with one full-rank sub-block. Also, while here we focus on the rank-one setting in 1 for clarity, our analysis extends to arbitrary but fixed (compared to n,p,d ) rank structure; see 5 in Appendix B for further discussion on this point. For the sake of our theoretical analysis, we assume the following for the nonlinearity f in 1 . Assumption 2 (Nonlinear function f ) . The function f\colon{\mathbb{R}}\to{\mathbb{R}} in 1 satisfies: 1. \lim_{t\to\infty}f(t)<\infty and |f(x)|\leq C_{1}\exp(C_{2}|x|) for some constants C_{1},C_{2}>0 ; and 2. f is centered with respect to standard Gaussian measure, that is, {\mathbb{E}}[f(\xi)]=0 for \xi\sim{\mathcal{N}}(0,1) , a_{1}\equiv{\mathbb{E}}[\xi f(\xi)]={\mathbb{E}}[f^{\prime}(\xi)]\neq 0 , \sqrt{2}a_{2}\equiv{\mathbb{E}}[\xi^{2}f(\xi)]=0 , and \nu\equiv{\mathbb{E}}[f^{2}(\xi)] . The first item of 2 hold for bounded nonlinearity such as sigmoid, truncated exponential, or ReLU variants. For the second item, note that under 1 and for tokens \mathbf{x}_{i} drawn from the signal-plus-noise model in 2 below, it follows from the Central Limit Theorem that the non-diagonal entry of [\mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}]_{ij}/\sqrt{p}\to{\mathcal{N}}(0,1) in law as p\to\infty and for i\neq j , so that f is applied on a random matrix with asymptotically Gaussian (but strongly correlated) entries, justifying the Gaussian-centric Hermite expansion. We consider in 2 that the zeroth-order Hermite coefficient {\mathbb{E}}[f(\xi)] of f is zero: This can be achieved by subtracting the same constant from all non-diagonal entries of \mathbf{K}_{\mathbf{X}} in ( 1 ) and should not alter the Attention memorization behavior. We consider input tokens independently drawn from the following signal-plus-noise model, in the high-dimensional regime where n,p,d are all large and comparable. Definition 2 (Signal-plus-noise model) . Each token-target pair (\mathbf{x}_{i},y_{i})\in{\mathbb{R}}^{p}\times\{\pm 1\},i\in\{1,\ldots,n\} is independently drawn from the following binary Gaussian signal-plus-noise model: \mathbf{x}_{i}=y_{i}\boldsymbol{\mu}+\mathbf{z}_{i}\in{\mathbb{R}}^{p},\quad y_{i}\in\{\pm 1\},\quad\mathbf{z}_{i}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{p}), (3) where \boldsymbol{\mu}\in{\mathbb{R}}^{p} is a deterministic signal vector. This model is widely used in the study of statistical learning under structured inputs. It induces an embedded signal \pm\boldsymbol{\mu} perturbed by standard Gaussian random noise. Assumption 3 (High-dimensional asymptotics) . As n\to\infty , the following hold: 1. p/n\to c\in(0,\infty) , d/n\in(0,\infty) ; and 2. the mean vector \boldsymbol{\mu}\in{\mathbb{R}}^{p} in 2 and the weight vectors \mathbf{w}_{Q},\mathbf{w}_{K}\in{\mathbb{R}}^{p} in 1 satisfy \limsup_{n}\max\{\|\boldsymbol{\mu}\|,\|\mathbf{w}_{Q}\|,\|\mathbf{w}_{K}\|\}<\infty . Under 2 and 3 , the matrix of inputs writes \mathbf{X}=\boldsymbol{\mu}\mathbf{y}^{\top}+\mathbf{Z} , for \mathbf{y}=[y_{1},\ldots,y_{n}]^{\top}\in{\mathbb{R}}^{n} , and random noise matrix \mathbf{Z}\in{\mathbb{R}}^{p\times n} having i.i.d. standard Gaussian entries. Note that both the rank-one signal \boldsymbol{\mu}\mathbf{y}^{\top} and the noise matrix \mathbf{Z} have spectral norms of order O(\sqrt{n}) . so that they are set on even ground in the high-dimensional regime as n,p\to\infty under 3 . While we focus here on the binary Gaussian signal-plus-noise model in 2 , our analysis can be extended, with additional technical efforts, to more general settings. In particular, this includes sub-exponential mixture model for \mathbf{z}_{i}\in{\mathbb{R}}^{p} having i.i.d. sub-gaussian entries of zero mean, unit variance, and symmetric in law, and multi-class settings where the number of classes is larger than two but remains finite as n,p\to\infty . See 6 in Appendix B for further discussion. In this paper, we aim to quantify the statistical memorization error of the nonlinear Attention defined in 1 , under the signal-plus-noise input model in 2 . To this end, we evaluate the performance of Attention as a feature extractor in downstream tasks via linear probing . Let \mathbf{A}_{\mathbf{X}}\in{\mathbb{R}}^{p\times n} be the nonlinear Attention output defined in ( 1 ) of 1 for input matrix \mathbf{X}=[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]\in{\mathbb{R}}^{p\times n} , and let \mathbf{y}=[y_{1},\ldots,y_{n}]^{\top}\in{\mathbb{R}}^{n} denote the associated labels. We define a ridge-regularized linear probing vector \mathbf{w}\in{\mathbb{R}}^{p} that minimizes the following mean squared error (MSE) on the pair (\mathbf{X},\mathbf{y}) : \displaystyle L(\mathbf{w}) \displaystyle=\frac{1}{n}\left\|\mathbf{y}^{\top}-\mathbf{w}^{\top}\mathbf{A}_{\mathbf{X}}\right\|^{2}+\gamma\|\mathbf{W}_{V}^{\top}\mathbf{w}\|^{2}=\frac{1}{n}\left\|\mathbf{y}^{\top}-\mathbf{w}_{V}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}\right\|^{2}+\gamma\|\mathbf{w}_{V}\|^{2}\equiv L(\mathbf{w}_{V}), (4) where \mathbf{w}_{V}=\mathbf{W}_{V}^{\top}\mathbf{w}\in{\mathbb{R}}^{p} , and \gamma\geq 0 is the regularization penalty. For any \gamma>0 , the unique minimizer of ( 4 ) admits the following closed-form expression: \displaystyle\mathbf{w}_{V}^{*} \displaystyle=\left(\mathbf{X}\mathbf{K}_{\mathbf{X}}\mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}+n\gamma\mathbf{I}_{p}\right)^{-1}\mathbf{X}\mathbf{K}_{\mathbf{X}}\mathbf{y}=\mathbf{X}\mathbf{K}_{\mathbf{X}}\left(\mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}+n\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y}. (5) With the explicit linear probing solution \mathbf{w}_{V}^{*} given in ( 5 ), we now define the in-context memorization error of the nonlinear Attention in 1 as follow. Definition 3 (In-context memorization error of nonlinear Attention) . For (\mathbf{X},\mathbf{y})\in{\mathbb{R}}^{p\times n}\times\{\pm 1\}^{n} drawn from the signal-plus-noise model in 2 , the in-context memorization error of nonlinear Attention in 1 is defined as the mean squared error of the optimal linear probe \mathbf{w}_{V}^{*} in ( 5 ): \displaystyle E \displaystyle=\frac{1}{n}\left\|\mathbf{y}^{\top}-(\mathbf{w}_{V}^{*})^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}\right\|^{2}=-\frac{\gamma^{2}}{n}\frac{\partial\mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y}}{\partial\gamma}, (6) where we denote the nonlinear resolvent matrix \mathbf{Q}(\gamma)=\left(\mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}/n+\gamma\mathbf{I}_{n}\right)^{-1}. (7) By Equation 6 , assessing the in-context memorization error of nonlinear Attention reduces to the analysis of the quadratic form \mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y} of the random nonlinear resolvent \mathbf{Q}(\gamma) defined in ( 7 ). When the random input tokens \mathbf{X} are drawn from the signal-plus-noise model in 2 , this analysis presents the following technical challenges. 1. The resolvent matrix \mathbf{Q} depends on the input \mathbf{X} in a highly nonlinear fashion: both through the entry-wise nonlinearity f (see 1 ) and through the matrix inverse in ( 7 ). 2. The structure of \mathbf{Q} is more complex than classical random matrix models (e.g., Wigner or Wishart matrices) studied in RMT ( bai2010spectral ) or high-dimensional statistics ( vershynin2018high ) . Specially, the matrix \mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}/n can be viewed as a nonlinear extension of the standard sample covariance (or Gram) matrix marvcenko1967distribution ; baik2006eigenvalues , but with the key distinction that the population covariance taking the form of an Attention kernel matrix that is itself dependent of \mathbf{X} . These challenges must be addressed to characterize the memorization error of nonlinear Attention. To this end, we introduce the notion of Deterministic Equivalent , which provides a tractable surrogate for analyzing the high-dimensional behavior of the random resolvent \mathbf{Q}(\gamma) defined in ( 7 ). Definition 4 (Deterministic Equivalent, ( couillet2022RMT4ML , Definition 4) ) . Let \mathbf{Q}\in{\mathbb{R}}^{n\times n} be a sequence of random matrices. A sequence of deterministic matrices \bar{\mathbf{Q}} (of the same size) is called a Deterministic Equivalent for \mathbf{Q} , denoted \mathbf{Q}\leftrightarrow\bar{\mathbf{Q}} , if for all (sequences of) deterministic matrices \mathbf{A}\in{\mathbb{R}}^{n\times n} and vectors \mathbf{a},\mathbf{b}\in{\mathbb{R}}^{n} of unit spectral and Euclidean norm, we have, \mathbf{Q}\leftrightarrow\bar{\mathbf{Q}}:\frac{1}{n}\tr\left(\mathbf{A}(\mathbf{Q}-\bar{\mathbf{Q}})\right)\to 0,\quad\mathbf{a}^{\top}(\mathbf{Q}-\bar{\mathbf{Q}})\mathbf{b}\to 0, (8) in probability as n\to\infty . We aim to deriving a Deterministic Equivalent for the nonlinear resolvent \mathbf{Q}(\gamma) defined in ( 7 ), which in turn enables high-dimensional characterization of the quadratic form \mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y}/n and the in-context memorization error E in ( 6 ) of 3 . This is the focus of the next section. 3 Main Technical Results This section presents our main technical contributions. We begin with 1 , which establishes a high-dimensional linearization of the Attention kernel matrix \mathbf{K}_{\mathbf{X}} defined in ( 1 ). Next, 1 provides a Deterministic Equivalent for the noise-only nonlinear Attention resolvent. Together, these results enables a precise characterization of the in-context memorization error E defined in 3 , which we present in 1 at the end of this section. To start with, note that under the full-plus-low-rank decomposition of the Attention weights in 1 , the Attention kernel matrix \mathbf{K}_{\mathbf{X}} in ( 1 ) admits a more tractable approximation via a Hermite polynomial expansion in the high-dimensional regime of 3 . This is given in the following result and proven in Section C.2 . Lemma 1 (High-dimensional linearization of Attention kernel matrix) . Let Assumptions 1 – 3 hold. Then, the Attention kernel matrix \mathbf{K}_{\mathbf{X}}=f(\mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p})/\sqrt{p} defined in ( 1 ) satisfies \|\mathbf{K}_{\mathbf{X}}-\tilde{\mathbf{K}}_{\mathbf{X}}\|=O(n^{-1/2})~~\text{with}~~\tilde{\mathbf{K}}_{\mathbf{X}}=\mathbf{K}_{N}+\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{K}}\mathbf{V}_{Q}^{\top},~~\boldsymbol{\Sigma}_{\mathbf{K}}=a_{1}\left[\begin{smallmatrix}\|\boldsymbol{\mu}\|^{2}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}&1&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 1&0&0\\ \boldsymbol{\mu}^{\top}\mathbf{w}_{Q}&0&1\end{smallmatrix}\right]\in{\mathbb{R}}^{3\times 3}, (9) with probability approaching one as n,p\to\infty . Here, a_{1} is the first Hermite coefficient of f (see 2 ), \mathbf{K}_{N}\equiv f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})/\sqrt{p}-\diag(\cdot) is a symmetric noise-only kernel matrix and \mathbf{U}_{K}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{K}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3},\quad\mathbf{V}_{Q}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{Q}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3}. (10) Moreover, we have that \max\{\|\mathbf{K}_{N}\|,\|\mathbf{U}_{K}\|,\|\boldsymbol{\Sigma}_{\mathbf{K}}\|,\|\mathbf{V}_{Q}\|\}=O(1) with high probability as n,p\to\infty . 1 shows that the nonlinear kernel matrix \mathbf{K}_{\mathbf{X}} can be decomposed as the sum of: 1. a symmetric noisy-only random kernel matrix \mathbf{K}_{N} ; 2 2 2 The noise-only kernel matrix \mathbf{K}_{N} is known in the literature as a random inner-product kernel matrix ( cheng2013random ; fan2019spectral ; kammounCovarianceDiscriminativePower2023 ) , with connections to single-hidden-layer (random) neural networks ( pennington2017nonlinear ; benigni2019eigenvalue ) . and 2. a low-rank, asymmetric informative matrix (rank at most three), whose structure depends on the interaction between the signal \boldsymbol{\mu} and Attention weights \mathbf{w}_{K},\mathbf{w}_{Q} , and on the nonlinearity f only via its first Hermite coefficient a_{1}={\mathbb{E}}[\xi f(\xi)],\xi\sim{\mathcal{N}}(0,1) . Note that under 2 , the input matrix \mathbf{X}=\boldsymbol{\mu}\mathbf{y}^{\top}+\mathbf{Z} also admits a rank-one signal-plus-noise decomposition. As such, the matrix of interest \mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}/n can be approximated, per 1 , as the sum of some full-rank and low-rank matrices. In the following result, we focus on the full-rank (and noise-only) part of the Attention matrix and derive a Deterministic Equivalent for its resolvent, the proof of which is given in Section C.3 . Proposition 1 (Deterministic Equivalent for noise-only nonlinear Attention resolvent) . Let \mathbf{Z}\in{\mathbb{R}}^{p\times n} be a random matrix having i.i.d. standard Gaussian entries, and define the symmetric noise-only kernel matrix \mathbf{K}_{N}=f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})/\sqrt{p}-\diag(\cdot) as in 1 . Then, as n,p\to\infty with p/n\to c\in(0,\infty) and \gamma>0 , the following Deterministic Equivalent (see 4 ) holds \left(\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}/n+\gamma\mathbf{I}_{n}\right)^{-1}\leftrightarrow m(\gamma)/c\cdot\mathbf{I}_{n}, where m(\gamma) is the unique Stieltjes transform solution to the fixed-point equation \displaystyle m(\gamma) \displaystyle=\left(\gamma/c+\nu/c+a_{1}^{2}/c^{2}-\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)^{-1}, with \mathbf{v}=\begin{bmatrix}\frac{a_{1}^{2}}{c^{2}}(1+c)&\frac{a_{1}}{c}&\frac{a_{1}}{c}&0&0&1\end{bmatrix}^{\top}\in{\mathbb{R}}^{6} and \mathbf{T}(\gamma)\in{\mathbb{R}}^{6\times 6} is a symmetric matrix whose entries are polynomial involving m(\gamma),\delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma),\delta_{4}(\gamma) defined in ( 28 ) of Section C.3 . Notably, the system of equations depends on the regularization penalty \gamma , the dimension ratio c , and the nonlinearity f via its Hermite coefficients a_{1} and \nu in 2 . By combining 1 and 1 , we obtain the following high-dimensional characterization of the in-context memorization error E defined in ( 6 ) for the nonlinear Attention in 1 . The proof is given in Section C.4 . Theorem 1 (High-dimensional characterization of in-context memorization error) . Let Assumptions 1 – 3 hold. Then, the in-context memorization error E defined in ( 6 ) satisfies E-\bar{E}\to 0 in probability as n,p\to\infty with p/n\to c\in(0,\infty) , where \bar{E}=-\gamma^{2}c^{2}\cdot\mathbf{e}_{7}^{\top}\left(c\mathbf{I}_{9}+\boldsymbol{\Delta}(\gamma)\boldsymbol{\Lambda}\right)^{-1}\boldsymbol{\Delta}^{\prime}(\gamma)\left(c\mathbf{I}_{9}+\boldsymbol{\Lambda}\boldsymbol{\Delta}(\gamma)\right)^{-1}\mathbf{e}_{7}. (11) Here, \mathbf{e}_{7}\in{\mathbb{R}}^{9} is the canonical basis vector with [\mathbf{e}_{i}]_{j}=\delta_{ij} , \boldsymbol{\Lambda},\boldsymbol{\Delta}(\gamma)\in{\mathbb{R}}^{9\times 9} are symmetric matrices defined in 8 of Section C.4 , and \boldsymbol{\Delta}^{\prime}(\gamma) is the derivative of \boldsymbol{\Delta}(\gamma) with respect to \gamma . 4 In-context Memorization of Nonlinear Attention versus Linear Regression In this section, we discuss the implications of our technical results in 1 , by contrasting the in-context memorization behavior of nonlinear Attention with that of linear ridge regression. Numerical experiments are also provided for moderately large n,p only in a few thousands, demonstrating the effectiveness of our proposed asymptotic analysis in practical settings. 4.1 In-context Memorization of Ridge Regression We begin by considering a classical baseline where the input embedding matrix \mathbf{X} is directly used for linear probing, instead of the nonlinear Attention output \mathbf{A}_{\mathbf{X}} defined in ( 1 ) of 1 . In this case, the probing vector \mathbf{w}_{\rm RR}\in{\mathbb{R}}^{p} is obtained by minimizing the following ridge-regularized MSE: L_{\rm RR}(\mathbf{w})=\frac{1}{n}\left\|\mathbf{y}^{\top}-\mathbf{w}^{\top}\mathbf{X}\right\|^{2}+\gamma\|\mathbf{w}\|^{2}. (12) This leads to the ridge regression solution defined below. Definition 5 (Ridge regression and its in-context memorization error) . For (\mathbf{X},\mathbf{y})\in{\mathbb{R}}^{p\times n}\times\{\pm 1\}^{n} drawn from the signal-plus-noise model in 2 , the ridge regression solution \mathbf{w}_{\rm RR} is given by \mathbf{w}_{\rm RR}=\left(\mathbf{X}\mathbf{X}^{\top}+n\gamma\mathbf{I}_{p}\right)^{-1}\mathbf{X}\mathbf{y}=\mathbf{X}\left(\mathbf{X}^{\top}\mathbf{X}+n\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y},\quad\gamma>0. (13) Its associated in-context memorization error is given by \displaystyle E_{\rm RR} \displaystyle=\frac{1}{n}\left\|\mathbf{y}^{\top}-\mathbf{w}_{\rm RR}^{\top}\mathbf{X}\right\|^{2}=-\frac{\gamma^{2}}{n}\frac{\partial\mathbf{y}^{\top}\left(\mathbf{X}^{\top}\mathbf{X}/n+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y}}{\partial\gamma}, (14) which is also the derivative of the quadratic form of the linear resolvent \left(\mathbf{X}^{\top}\mathbf{X}/n+\gamma\mathbf{I}_{n}\right)^{-1} . We now characterize the ridge regression memorization error E_{\rm RR} in ( 14 ), in the high-dimensional regime of 3 . The proof is standard and included in Section C.5 for completeness. Proposition 2 (High-dimensional characterization of in-context memorization for ridge regression) . Let 3 hold. Then, the in-context memorization error E_{\rm RR} defined in ( 14 ) of the ridge regression model in 5 satisfies E_{\rm RR}-\bar{E}_{\rm RR}\to 0 in probability as n,p\to\infty , with \bar{E}_{\rm RR}=-\frac{c\gamma^{2}m^{\prime}(\gamma)+c-1+\|\boldsymbol{\mu}\|^{2}\left(\gamma^{2}m^{\prime}(\gamma)+(1-c-\gamma)(\gamma m(\gamma)-1)\right)}{\left(1+\|\boldsymbol{\mu}\|^{2}-\|\boldsymbol{\mu}\|^{2}\gamma m_{\rm RR}(\gamma)\right)^{2}}, (15) where m_{\rm RR}(\gamma) is the Stieltjes transform solution to the following Marc̆enko-Pastur equation ( marvcenko1967distribution ; couillet2022RMT4ML ) : c\gamma m_{\rm RR}^{2}(\gamma)+\left(1-c+\gamma\right)m_{\rm RR}(\gamma)-1=0, (16) and m_{\rm RR}^{\prime}(\gamma)=-\frac{cm_{\rm RR}^{2}(\gamma)+m_{\rm RR}(\gamma)}{2c\gamma m_{\rm RR}(\gamma)+1-c+\gamma} is its derivative with respect to \gamma . In what follows, we leverage the high-dimensional characterization in 2 to assess how the in-context memorization error E_{\rm RR} of ridge regression in 5 is influenced by: the regularization strength \gamma , the dimension ratio c=\lim p/n , and the signal-to-noise ratio (SNR) \|\boldsymbol{\mu}\|^{2} . Remark 1 (Effect of regularization strength for ridge regression) . Under the settings and notations of 2 , the in-context memorization error E_{\rm RR} is an increasing function of the regularization strength \gamma . In the “ridgeless” limit \gamma\to 0 , the memorization error vanishes E_{\rm RR}\to 0 for p>n ; whereas in the strongly regularized limit \gamma\to\infty we have E_{\rm RR}\to 1 . Interestingly, when \gamma\to 0 and c=\lim p/n\to 0 , the Stieltjes transform m_{\rm RR}(\gamma) becomes singular, which is connected to the now well-known “double descent” phenomenon in test error curves ( bartlett2020benign ; mei2021generalization ; liao2020random ; hastie2022Surprises ) . Remark 2 (Effect of embedding dimension for ridge regression) . The in-context memorization error E_{\rm RR} of ridge regression is a decreasing function of the dimension ratio c=\lim p/n . For fixed n , increasing the embedding dimension p thus improves memorization. In the limit c\to 0 and for \gamma=0 , the memorization error converges to E_{\rm RR}\to 1/(1+\|\boldsymbol{\mu}\|^{2}) . Moreover, in the under-parametrized setting with p0 , f_{r}(t)=\max\left(-5,\min(5,r{\rm He}_{1}(t)+\sqrt{1-r^{2}}{\rm He}_{3}(t))\right) , where {\rm He}_{1}(t)=t and {\rm He}_{3}(t)=(t^{3}-3t)/\sqrt{6} is the first and third normalized Hermite polynomial, respectively. Fixing the “total energy” of f to \nu={\mathbb{E}}_{\xi\sim\mathcal{N}(0,1)}[f^{2}(\xi)]\approx 1 , we observe that memorization error decreases with increasing a_{1} , highlighting the crucial role of the linear component in f . To further support this, we compare, in Figure 3(b) and Figure 3(c) respectively, the trends of in-context memorization errors as a function of the embedding dimension p and SNR, for three nonlinearities f(t)=\tanh(t) (with a_{1}=0.6057 ), bounded linear f(t)=\max(-5,\min(5,t)) (with a_{1}\approx 1 ), and f(t)=\cos(t) (with a_{1}\approx 0 ). As shown in Figure 3(b) , when p increases, only Attentions having a linear component ( a_{1}\neq 0 ) exhibit a meaningful gain in memorization performance. Similarly, in Figure 3(c) , cosine-based Attention shows almost no improvement as SNR increases, whereas Attentions having a linear component consistently improve. These findings suggest that retaining a sufficient linear component in the Attention nonlinearity is not merely beneficial but essential for efficient information integration and memorization in Transformer-based architectures. Further experiments are provided in Appendix D to show XXX. 5 Conclusion and Perspectives In this paper, we provide a precise high-dimensional characterization of the in-context memorization error of nonlinear Attention applied on structured input drawn from a signal-plus-noise model. We show that, although nonlinear Attention typically incurs slighter higher memorization error than linear ridge regression for random inputs, this disadvantage vanishes—and can even be reversed—when the input possesses structure, particularly when the Attentions weights are aligned with the underlying input signal. Our analysis further reveal the critical role of the linear component in nonlinear Attention: Attentions with a vanishing first Hermite coefficient a_{1}=0 are fundamentally limited in their ability to reduce the memorization error (e.g., as the embedding dimension p or SNR increase). This insight underscores the importance of preserving some degree of linearity in Attention design for efficient memorization and representation learning. A natural extension of this work is to incorporate more realistic architectural components used in practical Transformers, such as skip connections or multi-head Attention. Another important direction is to move beyond the i.i.d. signal-plus-noise model in 2 considered here. In real-world scenarios such as natural language processing or time series analysis, the input (tokenized) sequences typically exhibit strong temporal correlations. For instance, the case of linear temporal correlation has been recently studied in moniri2024Asymptotics , though limited to linear regression model. It would be valuable to extend our nonlinear random matrix analysis to such structured input settings to further bridge the gap between theoretical understanding and practical deployment of Transformer architectures. Acknowledgments and Disclosure of Funding Z. Liao would like to acknowledge the National Natural Science Foundation of China (via fund NSFC-62206101) and the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001) for providing partial support. Difan Zou acknowledges the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A1515012444, and the central fund from HKU. Z. Ling is supported by the National Natural Science Foundation of China (via NSFC-62406119) and the Natural Science Foundation of Hubei Province (2024AFB074). Supplementary Material of A Random Matrix Analysis of In-context Memorization for Nonlinear Attention The technical appendices of this paper are organized as follows. An extended discussion of related work is given in Appendix A . Some auxiliary results and discussions are placed in Appendix B . The detailed proofs of our technical results are given in Appendix C . Additional numerical results are provided in Appendix D . Appendix A Further Discussions of Prior Efforts Transformers and empirical scaling laws. A growing body of work has established empirical scaling laws for Transformer models with respect to data size, model size, and computational budget. Early studies demonstrated power-law curves between generalization performance and model size for Transformer-based LLMs [ 41 , 34 ] , with subsequent extensions to transfer and multitask learning [ 35 , 87 ] . Notably, it has been shown in [ 36 ] that smaller models trained on more data can outperform larger “undertrained” models under fixed compute budget. Other studies have explored the effects of overparameterization, initialization, and depth-width trade-offs in the scaling laws of Transformer-based models [ 7 , 94 , 90 ] . Emergent phenomena and scaling transitions such as double descent [ 64 ] , in-context induction [ 67 ] , and phase shifts in predictability [ 30 ] have also been empirically observed. Investigations on Vision Transformers and instruction-tuned models [ 26 , 20 ] further support the universality of scaling behaviors across different modalities. Our work complements these empirical findings by providing a precise theoretical characterization on the scaling law of in-context memorization error of nonlinear Attention as a function of the sample-to-dimension ratio ( n/p ) and the SNR of the input data. Efficient Transformer variants and low-rank adaptation. The quadratic complexity of vanilla Attention with respect to sequence length has motivated a wide range of approximation-based methods to improve computational efficiency. Performer has replaced the Softmax nonlinearity with kernel-based random projections to achieve near-linear complexity [ 44 ] ; Linformer has projected keys and values into a low-dimensional subspace [ 85 ] ; Nyströmformer approximates the Attention matrix using the Nyström method [ 92 ] ; and Reformer has combined locality-sensitive hashing with reversible layers for memory savings [ 43 ] . In parallel, a series of works have proposed low-rank adaptation techniques for efficient fine-tuning of Transformer-based LLMs. LoRA has introduced trainable low-rank updates to frozen weights [ 38 ] ; QLoRA has extended this idea to quantized models with minimal performance degradation [ 23 ] ; LoRA-FA has improved memory efficiency via factorized updates [ 95 ] ; and UniPELT has unified multiple parameter-efficient tuning strategies into a single framework [ 58 ] . Motivated by these low-rank structures in computing and/or fine-tuning Transformer-based models, we assume in 1 a full-plus-low-rank decomposition of the Attention weights, and characterizes how such structure affects the memorization capacity of nonlinear Attention. Theoretical understanding of DNN, LLMs, and in-context learning. Recent theoretical advances in the optimization and generalization of over-parameterized deep neural networks (DNNs) have laid the groundwork for understanding the training behavior of modern large language models (LLMs). Despite the fact that LLMs typically operate in a regime where the number of model parameters far exceeds the number of training samples, first-order methods such as stochastic gradient descent have been shown to converge reliably and generalize effectively under specific conditions [ 47 , 3 ] for DNNs. Notably, the “edge of stability” phenomenon has emerged as a key concept, capturing the peculiar yet effective optimization dynamics commonly observed during the training of DNNs and LLMs [ 21 , 4 , 86 ] . Building on these insights, a growing body of work has investigated the mechanisms underlying in-context learning (ICL). Transformers have been shown to approximate gradient descent steps via Attention blocks [ 84 , 55 ] , and even to implement general-purpose learning algorithms directly from contextual input [ 1 , 32 ] . Alternative viewpoints interpret ICL as a form of implicit Bayesian inference [ 91 , 27 ] , offering probabilistic frameworks to explain generalization from prompts. At the mechanistic level, recent work has identified “induction heads” within Transformer architectures than enable token-level pattern recognition and generalization [ 66 ] . Appendix B Auxiliary Results and Further Discussions In this section, we provide further discussions on possible extensions of our theoretical results. We discuss in 4 the connection between the entry-wise Attention in 1 to the standard Softmax Attention, in 5 the extension of 1 beyond the rank-one setting, and in 6 the possibility to relax the binary Gaussian mixture model in 2 to, e.g., multi-class sub-gaussian mixture model. Remark 4 (On Softmax Attention) . As already mentioned in Footnote 1 , while 1 corresponds to entry-wise Attention (such the sigmoid or ReLU Attention [ 88 , 74 ] ) instead of commonly used Softmax Attention, under the setting of Assumptions 1 and 3 and for input token drawn from the signal-plus-noise model in 2 taking a truncated exponential function f(t)=\min(\exp(t),C) for some C>0 large, leads to approximately the same Attention matrix \mathbf{A}_{\mathbf{X}} as that of Softmax nonlinearity, up to a scaling factor. Precisely, note from the proof of 1 below in Section C.2 that the (i,j) entry of \mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X} is given, for i\neq j , by \displaystyle\mathbf{x}_{i}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{j}=\mathbf{x}_{i}^{\top}\mathbf{x}_{j}+\mathbf{x}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{x}_{j} \displaystyle=\underbrace{\mathbf{z}_{i}^{\top}\mathbf{z}_{j}}_{O(\sqrt{p})}+\underbrace{y_{i}y_{j}\|\boldsymbol{\mu}\|^{2}+(y_{j}\mathbf{z}_{i}+y_{i}\mathbf{z}_{j})^{\top}\boldsymbol{\mu}+y_{j}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j}+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j}+y_{i}y_{j}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}}_{O(1)}, and for i=j , by \displaystyle\mathbf{x}_{i}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i} \displaystyle=\|\mathbf{x}_{i}\|^{2}+\mathbf{x}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{x}_{i} \displaystyle=\underbrace{\|\mathbf{z}_{i}\|^{2}}_{O(p)}+\underbrace{\|\boldsymbol{\mu}\|^{2}+2y_{i}\mathbf{z}_{i}^{\top}\boldsymbol{\mu}+y_{i}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}}_{O(1)}. As such, the i th column of \mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p} (on which the Softmax function is applied) writes \mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i}/\sqrt{p}=\frac{1}{\sqrt{p}}\begin{bmatrix}\mathbf{z}_{1}^{\top}\mathbf{z}_{i}\\ \vdots\\ \|\mathbf{z}_{i}\|^{2}=p+O(\sqrt{p})\\ \vdots\\ \mathbf{z}_{n}^{\top}\mathbf{z}_{i}\end{bmatrix}+O_{\|\cdot\|_{\infty}}(p^{-1/2}), (17) for \mathbf{z}_{i}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{p}) , where we denote O_{\|\cdot\|_{\infty}}(p^{-1/2}) for random vector of infinity norm O(p^{-1/2}) with high probability. As such, for j\neq i , we have, conditioned on \mathbf{z}_{i} that \mathbf{z}_{j}^{\top}\mathbf{z}_{j}/\sqrt{p}\sim{\mathcal{N}}(0,\|\mathbf{z}_{i}\|^{2}/p) which is approximately {\mathcal{N}}(0,1) for p large. Thus, by Taylor-expansion of f(t)=\min\{\exp(t),C\} , we have, for large enough C that with high probability \min\left\{\exp\left(\mathbf{x}_{j}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i}/\sqrt{p}\right),C\right\}=\exp\left(\mathbf{z}_{j}^{\top}\mathbf{z}_{i}/\sqrt{p}\right)+O(p^{-1/2}), (18) for j\neq i and similarly \min\left\{\exp\left(\mathbf{x}_{i}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i}/\sqrt{p}\right),C\right\}=C, (19) as a consequence of the fact that \|\mathbf{z}_{i}\|^{2}/\sqrt{p}\simeq{\mathbb{E}}[\|\mathbf{z}_{i}\|^{2}]/\sqrt{p}=\sqrt{p} . Also, we have \sum_{j=1}^{n}\exp\left(\mathbf{x}_{j}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i}/\sqrt{p}\right)=\sum_{j\neq i}\exp\left(\mathbf{z}_{j}^{\top}\mathbf{z}_{i}/\sqrt{p}\right)+O(\sqrt{p})+C=n{\mathbb{E}}[\exp({\mathcal{N}}(0,1))]+O(\sqrt{p}), (20) so that for truncated Softmax function {\rm Softmax}\colon{\mathbb{R}}^{n}\to{\mathbb{R}}^{n} defined as {\rm Softmax}(\mathbf{z})=\frac{1}{\sum_{j=1}^{n}\min\{\exp(z_{j}),C\}}\begin{bmatrix}\min\{\exp(z_{1}),C\}\\ \vdots\\ \min\{\exp(z_{n}),C\}\end{bmatrix}, (21) we have that the normalization factor for each column of \mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p} is asymptotically the same, and that Assumptions 1 and 3 , the entry-wise truncated exponential function f(t)=\min(\exp(t),C) leads to approximately the same Attention matrix \mathbf{A}_{\mathbf{X}} as the Softmax Attention, up to a scaling factor. Nonetheless, it remains unclear how this entry-wise approximation of Attention matrix could translate into, e.g., the approximation of Softmax using truncated exponential function in terms of the in-context memorization error in 3 . See Figure 7 below for numerical results showing such examples and counterexamples. Remark 5 (Extension beyond rank-one setting) . While we consider in 1 that the Attention weights admits a full-plus-low-rank decomposition of the form \mathbf{W}_{K}^{\top}\mathbf{W}_{Q}=\mathbf{I}_{p}+\mathbf{w}_{K}\mathbf{w}_{Q}^{\top} , with \mathbf{w}_{K}\mathbf{w}_{Q}^{\top} being of rank one, it is possible to extend the analysis beyond the rank-one setting and consider a low-rank part of rank K (with K fixed as n,p\to\infty ). Notably, in that setting, the linearization result in 1 must be modified so that the term \mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{K}}\mathbf{V}_{Q}^{\top} takes account of the rank- K structure in the product \mathbf{W}_{K}^{\top}\mathbf{W}_{Q} . Remark 6 (Extension beyond binary Gaussian signal-plus-noise model) . The Gaussian signal-plus-noise model in 2 can be extended (at least) in the following two ways. 1. By considering more sophisticated structure in the statistical signal, for instance with \mathbf{X}=\mathbf{M}\mathbf{J}^{\top} where \mathbf{M}=[\boldsymbol{\mu}_{1},~\ldots,~\boldsymbol{\mu}_{K}]\in{\mathbb{R}}^{p\times K} is the matrix containing the means of the K>2 classes, and \mathbf{J}=[\mathbf{j}_{1},~\ldots,~\mathbf{j}_{K}]\in{\mathbb{R}}^{n\times K} is the canonical vector of class \mathcal{C}_{k}\in\{1,\ldots K\} , with [\mathbf{j}_{k}]_{i}=1 if \mathbf{x}_{i} belongs to class \mathcal{C}_{k} and zero otherwise. 2. By considering not necessarily Gaussian noise in the input tokens. An example is the sub-gaussian distribution that is symmetric in law. It has been long known in RMT that eigenspectra of large random matrices enjoy universal properties for Gaussian and non-Gaussian entries [ 77 , 72 ] , and we expect that most of our technical results also hold for sub-gaussian distributions, see for example 2 below. Appendix C Mathematical Proofs In this section, we present the proofs of the technical results in this paper. We first recall in Section C.1 a few lemmas that will be used in the proofs. The proof of 1 is given in Section C.2 , the proof of 1 is given in Section C.3 , the proof of 1 is given in Section C.4 , and finally the proof of 2 in Section C.5 . C.1 Useful Lemmas Lemma 2 (Spectral norm of kernel random matrix, [ 28 ] ) . For a random matrix \mathbf{Z}\in{\mathbb{R}}^{p\times n} having i.i.d. sub-gaussian entries that are symmetric in law, of zero mean and unit variance, and function f\colon{\mathbb{R}}\to{\mathbb{R}} such that |f(x)|\leq C_{1}\exp(C_{2}|x|) for some constants C_{1},C_{2}>0 , the random kernel matrix \mathbf{K}=f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})/\sqrt{p}-\diag(\cdot)\in{\mathbb{R}}^{n\times n}, (22) satisfies, with high probability as n,p\to\infty at the same pace, that 1. \|\mathbf{K}\|=O(1) if {\mathbb{E}}_{\xi\sim{\mathcal{N}}(0,1)}[f(\xi)]=0 ; and 2. \|\mathbf{K}\|=O(\sqrt{p}) with \|\mathbf{K}-{\mathbb{E}}[f(\xi)]\mathbf{1}_{n}\mathbf{1}_{n}^{\top}/\sqrt{p}\|=O(1) otherwise. Lemma 3 (Matrix norm controls) . We have the following facts on the operator norm of matrices and Hadamard product between matrices. 1. For \mathbf{A}\in{\mathbb{R}}^{n\times n} , we have \|\mathbf{A}\|_{\max}\leq\|\mathbf{A}\|\leq n\|\mathbf{A}\|_{\max} with \|\mathbf{A}\|_{\max}\equiv\max_{i,j}|A_{ij}| . 2. For \mathbf{A},\mathbf{B}\in{\mathbb{R}}^{N\times n} , we have \|\mathbf{A}\odot\mathbf{B}\|\leq\max(\sqrt{n},\sqrt{N})\|\mathbf{A}\|_{\max}\cdot\|\mathbf{B}\| . 3. If \mathbf{A}\in{\mathbb{R}}^{N\times n} is of rank one with \mathbf{A}=\mathbf{u}\mathbf{v}^{\top} , \mathbf{u}\in{\mathbb{R}}^{N},\mathbf{v}\in{\mathbb{R}}^{n} , we have \mathbf{A}\odot\mathbf{B}=\diag(\mathbf{u})\mathbf{B}\diag(\mathbf{v}) so that \|\mathbf{A}\odot\mathbf{B}\|\leq\|\mathbf{u}\|_{\infty}\cdot\|\mathbf{v}\|_{\infty}\cdot\|\mathbf{B}\|, (23) see [ 5 , Fact 13] . More generally, if \mathbf{A} is of rank K with \mathbf{A}=\sum_{k=1}^{K}\mathbf{u}_{k}\mathbf{w}_{K}^{\top} , we similarly have \displaystyle\|\mathbf{A}\odot\mathbf{B}\| \displaystyle=\|(\sum_{k=1}^{K}\mathbf{u}_{k}\mathbf{w}_{k}^{\top})\odot\mathbf{B}\|=\|\sum_{k=1}^{K}(\mathbf{u}_{k}\mathbf{w}_{k}^{\top})\odot\mathbf{B}\| \displaystyle\leq\sum_{k=1}^{K}\|(\mathbf{u}_{k}\mathbf{w}_{k}^{\top})\odot\mathbf{B}\|\leq\sum_{k=1}^{K}\|\mathbf{u}_{k}\|_{\infty}\cdot\|\mathbf{w}_{k}\|_{\infty}\cdot\|\mathbf{B}\|. C.2 Proof of 1 Here, we present the proof of 1 by “linearizing” the nonlinear kernel matrix \mathbf{K}_{\mathbf{X}}=f(\mathbf{X}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{X}/\sqrt{p})/\sqrt{p}\in{\mathbb{R}}^{n\times n}, (24) defined in ( 1 ) of 1 . To start, note that for the binary mixture model in 2 and under Assumptions 1 and 3 , we have \mathbf{x}_{i}=y_{i}\boldsymbol{\mu}+\mathbf{z}_{i} and \mathbf{W}_{K}^{\top}\mathbf{W}_{Q}=\mathbf{I}_{p}+\mathbf{w}_{K}\mathbf{w}_{Q}^{\top} , so that for i\neq j , \displaystyle\mathbf{x}_{i}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{j}=\mathbf{x}_{i}^{\top}\mathbf{x}_{j}+\mathbf{x}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{x}_{j} \displaystyle=\underbrace{\mathbf{z}_{i}^{\top}\mathbf{z}_{j}}_{O(\sqrt{p})}+\underbrace{y_{i}y_{j}\|\boldsymbol{\mu}\|^{2}+(y_{j}\mathbf{z}_{i}+y_{i}\mathbf{z}_{j})^{\top}\boldsymbol{\mu}+y_{j}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j}+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j}+y_{i}y_{j}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}}_{O(1)}, for y_{i},y_{j}\in\{\pm 1\} and independent \mathbf{z}_{i},\mathbf{z}_{j}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{p}) , where we used the fact that \max\{\|\boldsymbol{\mu}\|,\|\mathbf{w}_{K}\|,\|\mathbf{w}_{Q}\|\}=O(1) under 3 . Similarly, for i=j , \displaystyle\mathbf{x}_{i}^{\top}\mathbf{W}_{K}^{\top}\mathbf{W}_{Q}\mathbf{x}_{i} \displaystyle=\|\mathbf{x}_{i}\|^{2}+\mathbf{x}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{x}_{i} \displaystyle=\underbrace{\|\mathbf{z}_{i}\|^{2}}_{O(p)}+\underbrace{\|\boldsymbol{\mu}\|^{2}+2y_{i}\mathbf{z}_{i}^{\top}\boldsymbol{\mu}+y_{i}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}}_{O(1)}, where we used the fact that for any deterministic vector \mathbf{w}\in{\mathbb{R}}^{p} of bounded norm, one has \mathbf{z}_{i}^{\top}\mathbf{w}\sim{\mathcal{N}}(0,\|\mathbf{w}\|^{2}) . As a consequence, we can Taylor-expand the smooth function f in \mathbf{K}_{\mathbf{X}} defined in ( 1 ) of 1 . We first treat its non-diagonal entry (i,j) , for i\neq j , as \displaystyle\sqrt{p}[\mathbf{K}_{\mathbf{X}}]_{ij} \displaystyle=f(\mathbf{z}_{i}^{\top}\mathbf{z}_{j}/\sqrt{p})+f^{\prime}(\mathbf{z}_{i}^{\top}\mathbf{z}_{j}/\sqrt{p})(y_{i}y_{j}\|\boldsymbol{\mu}\|^{2}+(y_{j}\mathbf{z}_{i}+y_{i}\mathbf{z}_{j})^{\top}\boldsymbol{\mu}+y_{j}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j} \displaystyle+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{j}+y_{i}y_{j}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu})/\sqrt{p}+O(p^{-1}), and for its diagonal entries as \displaystyle\sqrt{p}[\mathbf{K}_{\mathbf{X}}]_{ii} \displaystyle=f(\|\mathbf{z}_{i}\|^{2}/\sqrt{p})+f^{\prime}(\|\mathbf{z}_{i}\|^{2}/\sqrt{p})(\|\boldsymbol{\mu}\|^{2}+2y_{i}\mathbf{z}_{i}^{\top}\boldsymbol{\mu}+y_{i}\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}+y_{i}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\mathbf{z}_{i}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{z}_{i}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu})/\sqrt{p} \displaystyle+O(p^{-1}). Note that under 2 , one has \lim_{t\to\infty}f(t)<\infty so that as n,p\to\infty , \sqrt{p}[\mathbf{K}_{\mathbf{X}}]_{ii}=O(1). This leads to the following spectral norm approximation of \mathbf{K}_{\mathbf{X}} as \displaystyle\sqrt{p}\mathbf{K}_{\mathbf{X}} \displaystyle=\underbrace{f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})-\diag(\cdot)}_{O_{\|\cdot\|}(\sqrt{p})}+\underbrace{f^{\prime}(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})\odot(\|\boldsymbol{\mu}\|^{2}\mathbf{y}\mathbf{y}^{\top}+\mathbf{y}\boldsymbol{\mu}^{\top}\mathbf{Z}+\mathbf{Z}^{\top}\boldsymbol{\mu}\mathbf{y}^{\top}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\cdot\mathbf{Z}^{\top}\mathbf{w}_{K}\mathbf{y}^{\top}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\cdot\mathbf{y}\mathbf{w}_{Q}^{\top}\mathbf{Z})/\sqrt{p}}_{O_{\|\cdot\|}(\sqrt{p})} \displaystyle+\underbrace{\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}\cdot f^{\prime}(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})\odot(\mathbf{y}\mathbf{y}^{\top})/\sqrt{p}}_{O_{\|\cdot\|}(\sqrt{p})}+\underbrace{\mathbf{Z}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{Z}\odot f^{\prime}(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})/\sqrt{p}}_{O_{\|\cdot\|}(\sqrt{p})}-\diag(\cdot)+O_{\|\cdot\|}(1) \displaystyle=\underbrace{f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})-\diag(\cdot)}_{O_{\|\cdot\|}(\sqrt{p})}+\underbrace{a_{1}((\|\boldsymbol{\mu}\|^{2}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu})\mathbf{y}\mathbf{y}^{\top}+\mathbf{y}\boldsymbol{\mu}^{\top}\mathbf{Z}+\mathbf{Z}^{\top}\boldsymbol{\mu}\mathbf{y}^{\top}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\cdot\mathbf{Z}^{\top}\mathbf{w}_{K}\mathbf{y}^{\top}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\cdot\mathbf{y}\mathbf{w}_{Q}^{\top}\mathbf{Z})/\sqrt{p}}_{O_{\|\cdot\|}(\sqrt{p})} \displaystyle+\underbrace{a_{1}\mathbf{Z}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\mathbf{Z}/\sqrt{p}}_{O_{\|\cdot\|}(\sqrt{p})}-\diag(\cdot)+O_{\|\cdot\|}(1), where we used the fact that under 2 for {\mathbb{E}}[f(\xi)]=0 and {\mathbb{E}}[f^{\prime}(\xi)]=a_{1}\neq 0 , it follows from 2 that f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})-\diag(\cdot)=O_{\|\cdot\|}(\sqrt{p}) and f^{\prime}(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})={\mathbb{E}}[f^{\prime}(\xi)]\mathbf{1}_{n}\mathbf{1}_{n}^{\top}+O_{\|\cdot\|}(\sqrt{p}) , and then Item 3 of 3 . Let \mathbf{U}_{K}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{K}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3} , \mathbf{V}_{Q}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{Q}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3} , and \boldsymbol{\Sigma}_{\mathbf{K}}=a_{1}\begin{bmatrix}\|\boldsymbol{\mu}\|^{2}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\mathbf{w}_{Q}^{\top}\boldsymbol{\mu}&1&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 1&0&0\\ \boldsymbol{\mu}^{\top}\mathbf{w}_{Q}&0&1\end{bmatrix}\in{\mathbb{R}}^{3\times 3}. (25) Putting everything in matrix form, we conclude the proof of 1 . C.3 Proof of 1 For the sake of presentation, we provide here the derivation of the Deterministic Equivalent for the resolvent 3 3 3 Note that this is not the same \mathbf{Q}(\gamma) as in ( 3 ) of 3 . It is used here for the sake of notational convenience and only within the proof of 1 . \mathbf{Q}(\gamma)=\left(\frac{1}{p}\mathbf{K}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}+\frac{\gamma}{c}\mathbf{I}_{n}\right)^{-1},\quad\gamma>0, (26) where we denote, with a slight abuse of notation that \mathbf{K}=\mathbf{K}_{N}=f(\mathbf{Z}^{\top}\mathbf{Z}/\sqrt{p})/\sqrt{p}-\diag(\cdot) for the noise-only kernel matrix \mathbf{K}_{N} defined in 1 . The result in 1 can be obtained with a simple scaling. Consider the following normalized traces involving \mathbf{Q}(\gamma) defined in ( 26 ): \frac{1}{n}\tr\mathbf{Q}(\gamma),\quad\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K}),\quad\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K}\cdot\mathbf{Z}^{\top}\mathbf{Z}/p)\quad\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K}),\quad\frac{1}{p}\tr(\mathbf{Z}^{\top}\mathbf{Z}/p\cdot\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K}\cdot\mathbf{Z}^{\top}\mathbf{Z}/p), for which we shall subsequently prove that \displaystyle\frac{1}{n}\tr\mathbf{Q}(\gamma)-m(\gamma)\to 0,\quad\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K})-\delta_{1}(\gamma)\to 0,\quad\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K}\mathbf{Z}^{\top}\mathbf{Z}/p)-\delta_{2}(\gamma)\to 0, (27) \displaystyle\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K})-\delta_{3}(\gamma)\to 0,\quad\frac{1}{p}\tr(\mathbf{Z}^{\top}\mathbf{Z}/p\cdot\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K}\cdot\mathbf{Z}^{\top}\mathbf{Z}/p)-\delta_{4}(\gamma)\to 0, in probability as n,p\to\infty , where m(\gamma) and \delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma),\delta_{4}(\gamma) are Stieltjes transforms satisfying the following self-consistent system of equations \left\{\begin{array}[]{ll}m(\gamma)&=\left(\frac{\gamma}{c}+\frac{\nu}{c}+\frac{a_{1}^{2}}{c^{2}}-\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)^{-1}\\ c\delta_{1}(\gamma)&=-m(\gamma)\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}\\ c\delta_{2}(\gamma)&=\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+c\delta_{1}(\gamma)\left(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)\\ c\delta_{3}(\gamma)&=\mathbf{v}_{1}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+\frac{c^{2}\delta_{1}^{2}(\gamma)}{m(\gamma)}\\ c\delta_{4}(\gamma)&=\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{4}+m(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)^{2}\end{array}\right. (28) where we denote \mathbf{T}(\gamma)\equiv\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\in{\mathbb{R}}^{6\times 6}, (29) that is symmetric, for \boldsymbol{\Delta}_{0}(\gamma)\equiv\begin{bmatrix}\frac{m(\gamma)}{c}&\frac{a_{1}}{c}m(\gamma)&\delta_{1}(\gamma)&a_{1}\delta_{1}(\gamma)&\delta_{2}(\gamma)&a_{1}\delta_{2}(\gamma)\\ \frac{a_{1}}{c}m(\gamma)&\frac{\nu}{c}m(\gamma)&a_{1}\delta_{1}(\gamma)&\nu\delta_{1}(\gamma)&a_{1}\delta_{2}(\gamma)&\nu\delta_{2}(\gamma)\\ \delta_{1}(\gamma)&a_{1}\delta_{1}(\gamma)&\delta_{3}(\gamma)&a_{1}\delta_{3}(\gamma)&\frac{1}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))\\ a_{1}\delta_{1}(\gamma)&\nu\delta_{1}(\gamma)&a_{1}\delta_{3}(\gamma)&\nu\delta_{3}(\gamma)&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{\nu}{c}(1-\frac{\gamma}{c}m(\gamma))\\ \delta_{2}(\gamma)&a_{1}\delta_{2}(\gamma)&\frac{1}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\delta_{4}(\gamma)&a_{1}\delta_{4}(\gamma)\\ a_{1}\delta_{2}(\gamma)&\nu\delta_{2}(\gamma)&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{\nu}{c}(1-\frac{\gamma}{c}m(\gamma))&a_{1}\delta_{4}(\gamma)&\nu\delta_{4}(\gamma)\end{bmatrix}\in{\mathbb{R}}^{6\times 6}, (30) and \boldsymbol{\Lambda}_{0}=\begin{bmatrix}\frac{a_{1}^{2}}{c^{2}}(c+1)&a_{1}/c&a_{1}/c&0&a_{1}&0\\ a_{1}/c&1&1&0&0&0\\ a_{1}/c&1&1&0&0&0\\ 0&0&0&0&0&0\\ a_{1}&0&0&0&0&0\\ 0&0&0&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6\times 6}, (31) that are both symmetric, as well as \left\{\begin{array}[]{ll}\mathbf{v}=\begin{bmatrix}\frac{a_{1}^{2}}{c^{2}}(1+c)&\frac{a_{1}}{c}&\frac{a_{1}}{c}&0&0&1\end{bmatrix}^{\top}\in{\mathbb{R}}^{6}\\ \mathbf{v}_{1}=\begin{bmatrix}0&1&0&0&0&0\end{bmatrix}^{\top}\in{\mathbb{R}}^{6}\\ \mathbf{v}_{2}=\begin{bmatrix}1&0&0&0&0&0\end{bmatrix}^{\top}\in{\mathbb{R}}^{6}\\ \mathbf{v}_{4}=\begin{bmatrix}\frac{a_{1}}{c}&1&1&0&0&0\end{bmatrix}^{\top}\in{\mathbb{R}}^{6}.\end{array}\right. (32) C.3.1 Preliminaries First, let us introduce some notations and preliminary results that will be used in the proof of 1 . Following [ 22 , Section 4.3.3] , we can decompose, up to permutation, the nonlinear kernel matrix \mathbf{K} as \mathbf{K}=\begin{bmatrix}\mathbf{K}_{-i}&f(\mathbf{Z}_{-i}^{\top}\mathbf{z}_{i}/\sqrt{p})/\sqrt{p}\\ f(\mathbf{z}_{i}^{\top}\mathbf{Z}_{-i}/\sqrt{p})/\sqrt{p}&0\end{bmatrix}=\begin{bmatrix}\mathbf{K}_{-i}&f(\boldsymbol{\alpha}_{-i})/\sqrt{p}\\ f(\boldsymbol{\alpha}_{-i})^{\top}/\sqrt{p}&0\end{bmatrix}, (33) where we denote \mathbf{K}_{-i}\equiv f(\mathbf{Z}_{-i}^{\top}\mathbf{Z}_{-i}/\sqrt{p})/\sqrt{p}-\diag(\cdot)\in{\mathbb{R}}^{(n-1)\times(n-1)} , \boldsymbol{\alpha}_{-i}\equiv\mathbf{Z}_{-i}^{\top}\mathbf{z}_{i}/\sqrt{p}\in{\mathbb{R}}^{n-1}, (34) for \mathbf{Z}_{-i}\in{\mathbb{R}}^{p\times(n-1)} the Gaussian random matrix \mathbf{Z} with its i th column removed, and \mathbf{z}_{i}\in{\mathbb{R}}^{p} is the i th column of \mathbf{Z} . Note that in the large p limit, the random vector \boldsymbol{\alpha}_{-i} is standard Gaussian {\mathcal{N}}(\mathbf{0},\mathbf{I}_{n-1}) . Denote the shortcut \check{\mathbf{Z}}=\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z} , we can similarly decompose \check{\mathbf{Z}} as \check{\mathbf{Z}}=\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}=\begin{bmatrix}\mathbf{Z}_{-i}^{\top}\mathbf{Z}_{-i}/p&\boldsymbol{\alpha}_{-i}/\sqrt{p}\\ \boldsymbol{\alpha}_{-i}^{\top}/\sqrt{p}&1\\ \end{bmatrix}+O_{\|\cdot\|}(p^{-1/2})=\begin{bmatrix}\check{\mathbf{Z}}_{-i}&\boldsymbol{\alpha}_{-i}/\sqrt{p}\\ \boldsymbol{\alpha}_{-i}^{\top}/\sqrt{p}&1\\ \end{bmatrix}+O_{\|\cdot\|}(p^{-1/2}), (35) where the O_{\|\cdot\|}(p^{-1/2}) error term is due to the approximation \|\mathbf{z}_{i}\|^{2}/p=1+O(p^{-1/2}) with a CLT argument. Note that, by the decomposition f(x)=a_{1}x+f_{>1}(x) for a_{1} the first Hermite polynomial of f as defined in 2 , we have f(\boldsymbol{\alpha}_{-i})=a_{1}\boldsymbol{\alpha}_{-i}+f_{>1}(\boldsymbol{\alpha}_{-i}), (36) and, for any \mathbf{A}\in{\mathbb{R}}^{(n-1)\times(n-1)} independent of \boldsymbol{\alpha}_{-i} , \frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\mathbf{A}\boldsymbol{\alpha}_{-i}=a_{1}\cdot\frac{1}{p}\tr\mathbf{A}+o(1),\quad\frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\mathbf{A}f(\boldsymbol{\alpha}_{-i})=\nu\cdot\frac{1}{p}\tr\mathbf{A}+o(1). (37) In particular, for \mathbf{A}=\mathbf{I}_{n-1} , we get \frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\mathbf{A}\boldsymbol{\alpha}_{-i}=\frac{a_{1}}{c}+o(1) and \frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\mathbf{A}f(\boldsymbol{\alpha}_{-i})=\frac{\nu}{c}+o(1) , where c=\lim p/n as in 3 . Further denote \mathbf{U}_{0}=\begin{bmatrix}\boldsymbol{\alpha}_{-i}&f(\boldsymbol{\alpha}_{-i})&\mathbf{K}_{-i}^{\perp}\boldsymbol{\alpha}_{-i}&\mathbf{K}_{-i}^{\perp}f(\boldsymbol{\alpha}_{-i})&\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\boldsymbol{\alpha}_{-i}&\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}f(\boldsymbol{\alpha}_{-i})\end{bmatrix}/\sqrt{p}\in{\mathbb{R}}^{(n-1)\times 6}, (38) as well as \mathbf{K}_{-i}^{\perp}=\{f((\mathbf{z}_{j}^{\perp})^{\top}\mathbf{z}_{k}^{\perp}/\sqrt{p})\}_{j,k\neq i}/\sqrt{p}-\diag(\cdot)\in{\mathbb{R}}^{(n-1)\times(n-1)},\quad\check{\mathbf{Z}}_{-i}^{\perp}\equiv\{(\mathbf{z}_{j}^{\perp})^{\top}\mathbf{z}_{k}^{\perp}/p\}_{j,k\neq i}\in{\mathbb{R}}^{(n-1)\times(n-1)}, (39) where, for given i , \mathbf{z}_{j}^{\perp}\equiv\mathbf{z}_{j}-\frac{\mathbf{z}_{i}^{\top}\mathbf{z}_{j}}{\|\mathbf{z}_{i}\|}\frac{\mathbf{z}_{i}}{\|\mathbf{z}_{i}\|}, (40) that is orthogonal to and asymptotically independent of \mathbf{z}_{i}^{\top}\mathbf{z}_{j}/\|\mathbf{z}_{i}\|\approx[\boldsymbol{\alpha}_{-i}]_{j} in ( 34 ). For the two “leave-one-out” matrices \mathbf{K}_{-i} and \check{\mathbf{Z}}_{-i} defined in ( 33 ) and ( 35 ), we have the following result. Lemma 4 (Approximations of \mathbf{K}_{-i} and \check{\mathbf{Z}}_{-i} , [ 22 , Section 4.3.3] ) . For \mathbf{K}_{-i} and \check{\mathbf{Z}}_{-i} defined respectively in ( 33 ) and ( 35 ), we have the following approximations in spectral norm: 1. \mathbf{K}_{-i}=\mathbf{K}_{-i}^{\perp}+\frac{a_{1}}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) ; and 2. \check{\mathbf{Z}}_{-i}=\check{\mathbf{Z}}_{-i}^{\perp}+\frac{1}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) ; for \mathbf{K}_{-i}^{\perp} and \check{\mathbf{Z}}_{-i}^{\perp} defined in ( 39 ) that is asymptotically independent of \boldsymbol{\alpha}_{-i} in ( 34 ). With these preliminary results at hand, we are ready to derive a Deterministic Equivalent for \mathbf{Q}(\gamma) defined in ( 26 ). C.3.2 Self-consistent equation for m(z) Here we present the derivation for the Deterministic Equivalent (of the diagonal entries) of \mathbf{Q}(z) . With the block decomposition of \mathbf{K} and \check{\mathbf{Z}} in ( 33 ) and ( 35 ), we obtain for \mathbf{Q}(\gamma)\equiv\mathbf{Q} in ( 26 ) (where we drop the argument \gamma ) that \mathbf{Q}^{-1}\equiv\mathbf{K}\check{\mathbf{Z}}\mathbf{K}+\frac{\gamma}{c}\mathbf{I}_{n}=\begin{bmatrix}[\mathbf{Q}^{-1}]_{11}&[\mathbf{Q}^{-1}]_{12}\equiv[\mathbf{Q}^{-1}]_{21}^{\top}\\ [\mathbf{Q}^{-1}]_{21}&[\mathbf{Q}^{-1}]_{22}\end{bmatrix} (41) with \displaystyle[\mathbf{Q}^{-1}]_{11} \displaystyle\equiv\mathbf{K}_{-i}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i}+\frac{1}{p}f(\boldsymbol{\alpha}_{-i})\boldsymbol{\alpha}_{-i}^{\top}\mathbf{K}_{-i}+\frac{1}{p}\mathbf{K}_{-i}\boldsymbol{\alpha}_{-i}f(\boldsymbol{\alpha}_{-i})^{\top}+\frac{1}{p}f(\boldsymbol{\alpha}_{-i})f(\boldsymbol{\alpha}_{-i})^{\top}+\frac{\gamma}{c}\mathbf{I}_{n-1}, \displaystyle[\mathbf{Q}^{-1}]_{21} \displaystyle\equiv\frac{1}{\sqrt{p}}f(\boldsymbol{\alpha}_{-i})^{\top}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i}+\frac{a_{1}}{c}\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}, \displaystyle[\mathbf{Q}^{-1}]_{22} \displaystyle\equiv\frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\check{\mathbf{Z}}_{-i}f(\boldsymbol{\alpha}_{-i})+\frac{\gamma}{c}, for which we have, per 4 , 1. \mathbf{K}_{-i}=\mathbf{K}_{-i}^{\perp}+\frac{a_{1}}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) ; and 2. \check{\mathbf{Z}}_{-i}=\check{\mathbf{Z}}_{-i}^{\perp}+\frac{1}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) ; where \mathbf{K}_{-i}^{\perp} and \check{\mathbf{Z}}_{-i}^{\perp} as defined in ( 39 ) that are both asymptotically independent of \boldsymbol{\alpha}_{-i} . This allows for the first approximation of [\mathbf{Q}^{-1}]_{22} as \displaystyle[\mathbf{Q}^{-1}]_{22} \displaystyle=\frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\check{\mathbf{Z}}_{-i}f(\boldsymbol{\alpha}_{-i})+\frac{\gamma}{c}=\frac{1}{p}f(\boldsymbol{\alpha}_{-i})^{\top}\left(\frac{1}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+\check{\mathbf{Z}}_{-i}^{\perp}\right)f(\boldsymbol{\alpha}_{-i})+\frac{\gamma}{c}+o(1) \displaystyle=\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\frac{\gamma}{c}+o(1). (42) Let \mathbf{Q}_{-i}^{\perp}=\left(\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+\frac{\gamma}{c}\mathbf{I}_{n-1}\right)^{-1}\in{\mathbb{R}}^{(n-1)\times(n-1)}, (43) for \mathbf{K}_{-i}^{\perp},\check{\mathbf{Z}}_{-i}^{\perp} defined in ( 39 ), so that \mathbf{Q}_{-i}^{\perp} is asymptotically independent of \boldsymbol{\alpha}_{-i} satisfying \tr(\mathbf{Q}-\mathbf{Q}_{-i}^{\perp})=O(1) . We have the following approximation. Lemma 5 (Approximation of \mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0} ) . For \mathbf{Q}_{-i}^{\perp}\in{\mathbb{R}}^{(n-1)\times(n-1)} as in ( 43 ) and \mathbf{U}_{0}\in{\mathbb{R}}^{(n-1)\times 6} in ( 38 ), we have \mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}=\boldsymbol{\Delta}_{0}(\gamma)+o_{\|\cdot\|}(1), (44) with \boldsymbol{\Delta}_{0}(z)\equiv\begin{bmatrix}\frac{m(\gamma)}{c}&\frac{a_{1}}{c}m(\gamma)&\delta_{1}(\gamma)&a_{1}\delta_{1}(\gamma)&\delta_{2}(\gamma)&a_{1}\delta_{2}(\gamma)\\ \frac{a_{1}}{c}m(\gamma)&\frac{\nu}{c}m(\gamma)&a_{1}\delta_{1}(\gamma)&\nu\delta_{1}(\gamma)&a_{1}\delta_{2}(\gamma)&\nu\delta_{2}(\gamma)\\ \delta_{1}(\gamma)&a_{1}\delta_{1}(\gamma)&\delta_{3}(\gamma)&a_{1}\delta_{3}(\gamma)&\frac{1}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))\\ a_{1}\delta_{1}(\gamma)&\nu\delta_{1}(\gamma)&a_{1}\delta_{3}(\gamma)&\nu\delta_{3}(\gamma)&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{\nu}{c}(1-\frac{\gamma}{c}m(\gamma))\\ \delta_{2}(\gamma)&a_{1}\delta_{2}(\gamma)&\frac{1}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\delta_{4}(\gamma)&a_{1}\delta_{4}(\gamma)\\ a_{1}\delta_{2}(\gamma)&\nu\delta_{2}(\gamma)&\frac{a_{1}}{c}(1-\frac{\gamma}{c}m(\gamma))&\frac{\nu}{c}(1-\frac{\gamma}{c}m(\gamma))&a_{1}\delta_{4}(\gamma)&\nu\delta_{4}(\gamma)\end{bmatrix}\in{\mathbb{R}}^{6\times 6}, (45) as in ( 30 ), for m(\gamma),\delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma),\delta_{4}(\gamma) as defined in ( 27 ) and ( 28 ). Proof of 5 . Since \mathbf{Q}_{-i}^{\perp} is asymptotically independent of \mathbf{U}_{0}=\begin{bmatrix}\boldsymbol{\alpha}_{-i}&f(\boldsymbol{\alpha}_{-i})&\mathbf{K}_{-i}^{\perp}\boldsymbol{\alpha}_{-i}&\mathbf{K}_{-i}^{\perp}f(\boldsymbol{\alpha}_{-i})&\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\boldsymbol{\alpha}_{-i}&\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}f(\boldsymbol{\alpha}_{-i})\end{bmatrix}/\sqrt{p}\in{\mathbb{R}}^{(n-1)\times 6}, (46) we obtain \displaystyle\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0} \displaystyle=\begin{bmatrix}\frac{1}{p}\tr\mathbf{Q}&a_{1}\frac{1}{p}\tr\mathbf{Q}&\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&\frac{1}{p}\tr(\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\\ a_{1}\frac{1}{p}\tr\mathbf{Q}&\nu\frac{1}{p}\tr\mathbf{Q}&a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&\nu\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&\nu\frac{1}{p}\tr(\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\\ \frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K})&a_{1}\tr(\mathbf{K}\mathbf{Q}\mathbf{K})&\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&a_{1}\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\\ a_{1}\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&\nu\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})&a_{1}\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K})&\nu\tr(\mathbf{K}\mathbf{Q}\mathbf{K})&a_{1}\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&\nu\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\\ \frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q})&a_{1}\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q})&\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&a_{1}\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&a_{1}\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\\ a_{1}\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q})&\nu\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q})&a_{1}\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&\nu\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&a_{1}\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})&\nu\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})\end{bmatrix}+o_{\|\cdot\|}(1) \displaystyle=\boldsymbol{\Delta}_{0}(\gamma)+o_{\|\cdot\|}(1), where we recall from ( 27 ) that \displaystyle m(\gamma) \displaystyle=\frac{1}{n}\tr\mathbf{Q}(\gamma)+o(1)=\frac{1}{n}\tr\mathbf{Q}_{-i}^{\perp}(\gamma)+o(1) \displaystyle\delta_{1}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K})+o(1)=\frac{1}{p}\tr\left(\mathbf{Q}_{-i}^{\perp}(\gamma)\mathbf{K}_{-i}^{\perp}\right)+o(1) \displaystyle\delta_{2}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K}\check{\mathbf{Z}})+o(1)=\frac{1}{p}\tr(\mathbf{Q}_{-i}^{\perp}(\gamma)\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp})+o(1) \displaystyle\delta_{3}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K})+o(1)=\frac{1}{p}\tr(\mathbf{K}_{-i}^{\perp}\mathbf{Q}_{-i}^{\perp}(\gamma)\mathbf{K}_{-i}^{\perp})+o(1) \displaystyle\delta_{4}(\gamma) \displaystyle=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K}\check{\mathbf{Z}})+o(1)=\frac{1}{p}\tr(\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}\mathbf{Q}_{-i}^{\perp}(\gamma)\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp})+o(1), and we use the fact that by [ 76 , Lemma 2.6] and [ 22 , Lemma 2.9] , when evaluating normalized traces forms as in ( 27 ) for n,p large, we can ignore terms of finite rank inside the trace, by adding an error term o(1) with high probability, as well as \tr(\tilde{\mathbf{K}}\mathbf{Q}\tilde{\mathbf{K}}\check{\mathbf{Z}})=\tr(\mathbf{Q}\tilde{\mathbf{K}}\check{\mathbf{Z}}\tilde{\mathbf{K}})=\tr[\mathbf{Q}(\mathbf{Q}^{-1}-\gamma\mathbf{I}_{n})]=\tr[\mathbf{I}_{n}-\gamma\mathbf{Q}]=\tr(\mathbf{I}_{n})-\gamma\tr(\mathbf{Q})=n-n\gamma m(\gamma)=n(1-\gamma m(\gamma)) , This concludes the proof of 5 . ∎ Our objective is to compute the (i,i) th diagonal entries of the inverse \mathbf{Q}=(\mathbf{K}\check{\mathbf{Z}}\mathbf{K}+\gamma\mathbf{I}_{n})^{-1} . Using the block inversion lemma, we get [\mathbf{Q}]_{ii}=\left([\mathbf{Q}^{-1}]_{22}-[\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1}[\mathbf{Q}^{-1}]_{12}\right)^{-1}, (47) and \mathbf{Q}=\begin{bmatrix}([\mathbf{Q}^{-1}]_{11}-[\mathbf{Q}^{-1}]_{12}([\mathbf{Q}^{-1}]_{22})^{-1}[\mathbf{Q}^{-1}]_{21})^{-1}&-([\mathbf{Q}^{-1}]_{11})^{-1}[\mathbf{Q}^{-1}]_{12}[\mathbf{Q}]_{ii}\\ -[\mathbf{Q}]_{ii}[\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1}&[\mathbf{Q}]_{ii}\end{bmatrix}. (48) We start with the inverse ([\mathbf{Q}^{-1}]_{11})^{-1} , for which we have the following result. Lemma 6 (Approximation of ([\mathbf{Q}^{-1}]_{11})^{-1} ) . For [\mathbf{Q}^{-1}]_{11}\in{\mathbb{R}}^{(n-1)\times(n-1)} defined in ( 41 ), we have ([\mathbf{Q}^{-1}]_{11})^{-1}=\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}+o_{\|\cdot\|}(1), (49) where we recall \boldsymbol{\Delta}_{0}(\gamma)\in{\mathbb{R}}^{6\times 6} as in ( 30 ), \mathbf{U}_{0}\in{\mathbb{R}}^{(n-1)\times 6} as defined in ( 38 ), and \boldsymbol{\Lambda}_{0}=\begin{bmatrix}\frac{a_{1}^{2}}{c^{2}}(c+1)&a_{1}/c&a_{1}/c&0&a_{1}&0\\ a_{1}/c&1&1&0&0&0\\ a_{1}/c&1&1&0&0&0\\ 0&0&0&0&0&0\\ a_{1}&0&0&0&0&0\\ 0&0&0&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6\times 6}, (50) as in ( 31 ). Proof of 6 . Per its definition in ( 41 ), we have \displaystyle[\mathbf{Q}^{-1}]_{11} \displaystyle\equiv\mathbf{K}_{-i}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i}+\frac{1}{p}f(\boldsymbol{\alpha}_{-i})\boldsymbol{\alpha}_{-i}^{\top}\mathbf{K}_{-i}+\frac{1}{p}\mathbf{K}_{-i}\boldsymbol{\alpha}_{-i}f(\boldsymbol{\alpha}_{-i})^{\top}+\frac{1}{p}f(\boldsymbol{\alpha}_{-i})f(\boldsymbol{\alpha}_{-i})^{\top}+\frac{\gamma}{c}\mathbf{I}_{n-1} \displaystyle=\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+\mathbf{U}_{0}\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}+\frac{\gamma}{c}\mathbf{I}_{n-1}+o_{\|\cdot\|}(1), for \mathbf{U}_{0},\boldsymbol{\Lambda} defined in ( 31 ), \mathbf{K}_{-i}^{\perp},\check{\mathbf{Z}}_{-i}^{\perp} defined in ( 39 ) such that \mathbf{K}_{-i}=\mathbf{K}_{-i}^{\perp}+\frac{a_{1}}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) and \check{\mathbf{Z}}_{-i}=\check{\mathbf{Z}}_{-i}^{\perp}+\frac{1}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top}+o_{\|\cdot\|}(1) by 4 . As such, by Woodbury identity, \displaystyle([\mathbf{Q}^{-1}]_{11})^{-1} \displaystyle=\left(\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+\frac{\gamma}{c}\mathbf{I}_{n-1}+\mathbf{U}_{0}\boldsymbol{\Lambda}\mathbf{U}_{0}^{\top}\right)^{-1}+o_{\|\cdot\|}(1) \displaystyle=\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\right)^{-1}\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}+o_{\|\cdot\|}(1), for \mathbf{Q}_{-i}^{\perp} defined in ( 43 ). Using 5 to approximate \mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}=\boldsymbol{\Delta}_{0}(\gamma)+o_{\|\cdot\|}(1) , we conclude the proof of 6 . ∎ With Lemmas 5 and 6 , we get the following (block-wise) approximation for \mathbf{Q} . Lemma 7 (Block approximation of \mathbf{Q} ) . We have \mathbf{Q}=\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}+o_{\|\cdot\|}(1), (51) where we recall \boldsymbol{\Delta}_{0}(\gamma)\in{\mathbb{C}}^{6\times 6} as in ( 30 ), \boldsymbol{\Lambda}_{0} as in ( 31 ), \mathbf{U}_{0}\in{\mathbb{R}}^{(n-1)\times 6} as defined in ( 38 ), and \boldsymbol{\Lambda}_{1}(\gamma)=\boldsymbol{\Lambda}_{0}-\left(\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\frac{\gamma}{c}\right)^{-1}\mathbf{v}\mathbf{v}^{\top}\in{\mathbb{R}}^{6\times 6},\quad\mathbf{v}^{\top}=\begin{bmatrix}\frac{a_{1}^{2}}{c^{2}}(1+c)&\frac{a_{1}}{c}&\frac{a_{1}}{c}&0&0&1\end{bmatrix}\in{\mathbb{R}}^{6}, (52) as in ( 32 ). We also have, by ( 52 ) and Sherman–Morrison identity that \displaystyle\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}=\boldsymbol{\Delta}_{0}(\gamma)\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)-\left(\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\frac{\gamma}{c}\right)^{-1}\mathbf{v}\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1} \displaystyle=\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}+\frac{\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}}{\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\frac{\gamma}{c}-\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}}, (53) and \mathbf{U}_{0}^{\top}\left(\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\right)\mathbf{U}_{0}=\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}+o_{\|\cdot\|}(1). (54) Proof of 7 . We first work on [\mathbf{Q}^{-1}]_{21} by expanding the term \frac{1}{\sqrt{p}}f(\boldsymbol{\alpha}_{-i})^{\top}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i} as \displaystyle\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i} \displaystyle=\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}(\check{\mathbf{Z}}_{-i}^{\perp}+\frac{1}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top})(\mathbf{K}_{-i}^{\perp}+\frac{a_{1}}{p}\boldsymbol{\alpha}_{-i}\boldsymbol{\alpha}_{-i}^{\top})+o_{\|\cdot\|}(1) \displaystyle=\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+a_{1}\frac{n}{p}\frac{1}{\sqrt{p}}\boldsymbol{\alpha}_{-i}^{\top}\mathbf{K}_{-i}^{\perp}+a_{1}^{2}\frac{n}{p}\left(1+\frac{n}{p}\right)\frac{\boldsymbol{\alpha}_{-i}^{\top}}{\sqrt{p}}+o_{\|\cdot\|}(1), so that [\mathbf{Q}^{-1}]_{21}=\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}\check{\mathbf{Z}}_{-i}\mathbf{K}_{-i}+a_{1}\frac{n}{p}\frac{f(\boldsymbol{\alpha}_{-i})^{\top}}{\sqrt{p}}+o_{\|\cdot\|}(1)=\mathbf{v}^{\top}\mathbf{U}_{0}^{\top}+o_{\|\cdot\|}(1), (55) with \mathbf{v}\in{\mathbb{R}}^{6} defined in ( 52 ). So that \displaystyle[\mathbf{Q}]_{ii}[\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1} \displaystyle=m(\gamma)\mathbf{v}^{\top}\mathbf{U}_{0}^{\top}\left(\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}\right)+o_{\|\cdot\|}(1) \displaystyle=m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}+o_{\|\cdot\|}(1). Then, with the approximation of the inverse ([\mathbf{Q}^{-1}]_{11})^{-1} in 6 and that of [\mathbf{Q}^{-1}]_{21} above, we obtain \displaystyle([\mathbf{Q}^{-1}]_{11}-[\mathbf{Q}^{-1}]_{12}([\mathbf{Q}^{-1}]_{22})^{-1}[\mathbf{Q}^{-1}]_{21})^{-1} \displaystyle=(\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+\mathbf{U}_{0}\boldsymbol{\Lambda}_{0}\mathbf{U}_{0}^{\top}+\frac{\gamma}{c}\mathbf{I}_{n-1}-([\mathbf{Q}^{-1}]_{22})^{-1}\mathbf{U}_{0}\mathbf{v}\mathbf{v}^{\top}\mathbf{U}_{0}^{\top})^{-1}+o_{\|\cdot\|}(1) \displaystyle=(\mathbf{K}_{-i}^{\perp}\check{\mathbf{Z}}_{-i}^{\perp}\mathbf{K}_{-i}^{\perp}+\mathbf{U}_{0}\boldsymbol{\Lambda}_{1}\mathbf{U}_{0}^{\top}+\frac{\gamma}{c}\mathbf{I}_{n-1})^{-1}+o_{\|\cdot\|}(1) \displaystyle=\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}+o_{\|\cdot\|}(1), by ( 42 ) and Woodbury identity, for \boldsymbol{\Lambda}_{1}(\gamma)=\boldsymbol{\Lambda}_{0}-([\mathbf{Q}^{-1}]_{22})^{-1}\mathbf{v}\mathbf{v}^{\top}+o_{\|\cdot\|}(1)=\boldsymbol{\Lambda}_{0}-\left(\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\frac{\gamma}{c}\right)^{-1}\mathbf{v}\mathbf{v}^{\top}+o_{\|\cdot\|}(1), (56) as defined in ( 52 ). This concludes the proof of 7 . ∎ Following the same idea, we expand the quadratic form [\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1}[\mathbf{Q}^{-1}]_{12} in ( 47 ) as [\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1}[\mathbf{Q}^{-1}]_{12}=\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}+o(1), (57) for \mathbf{v}\in{\mathbb{R}}^{6} defined in ( 52 ). Plugging this approximation back to ( 47 ) and ignoring the terms in o(1) , we obtain the following self-consistent equation on m(\gamma) , \frac{1}{m(\gamma)}=\frac{\gamma}{c}+\frac{\nu}{c}+\frac{a_{1}^{2}}{c^{2}}-\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v},\quad\mathbf{T}(\gamma)=\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}. (58) In the following, we determine the (self-consistent) equations for \delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma) and \delta_{4}(\gamma) in \mathbf{T}(\gamma) , so as to retrieve the final self-consistent equations in ( 27 ). C.3.3 Establishing self-consistent equations for \delta(\gamma) s Following the same idea above in Section C.3.2 , we now establish self-consistent equations for the intermediate variables \delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma),\delta_{4}(\gamma) defined in ( 27 ). Self-consistent equation for \delta_{1}(\gamma) . We start with \delta_{1}(\gamma)=\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K})+o(1) by writing \displaystyle\delta_{1}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{Q}\mathbf{K})+o(1)=\frac{1}{p}\sum_{i=1}^{n}[\mathbf{Q}\mathbf{K}]_{ii}+o(1)=\frac{1}{c}[\mathbf{Q}\mathbf{K}]_{ii}+o(1) \displaystyle=-[\mathbf{Q}]_{ii}\frac{1}{c}[\mathbf{Q}^{-1}]_{21}([\mathbf{Q}^{-1}]_{11})^{-1}f(\boldsymbol{\alpha}_{-i})/\sqrt{p}+o(1) \displaystyle=-\frac{m(\gamma)}{c}\mathbf{v}^{\top}\mathbf{U}_{0}^{\top}([\mathbf{Q}^{-1}]_{11})^{-1}\mathbf{U}_{0}\mathbf{v}_{1}+o(1)=-\frac{m(\gamma)}{c}\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+o(1) \displaystyle=-\frac{m(\gamma)}{c}\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+o(1), for \mathbf{v}_{1}^{\top}=\begin{bmatrix}0&1&0&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6}. (59) where we used the fact that f(\boldsymbol{\alpha}_{-i})/\sqrt{p}=\mathbf{U}_{0}\mathbf{v}_{1} , ( 55 ), and 6 . Self-consistent equation for \delta_{2}(\gamma) . We consider now \delta_{2}(\gamma)=\frac{1}{p}\tr(\mathbf{Q}(\gamma)\mathbf{K}\check{\mathbf{Z}})+o(1) and write \displaystyle\delta_{2}(\gamma)=\frac{1}{p}\tr(\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})+o(1)=\frac{1}{c}[\check{\mathbf{Z}}\mathbf{Q}\mathbf{K}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}\boldsymbol{\alpha}_{-i}^{\top}/\sqrt{p}&1\end{bmatrix}\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}\begin{bmatrix}f(\boldsymbol{\alpha}_{-i})/\sqrt{p}\\ 0\end{bmatrix}+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{2}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}-m(\gamma)\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}\right)+o(1) \displaystyle=\frac{1}{c}\left((\mathbf{v}_{2}-m(\gamma)\mathbf{v})^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+\frac{\mathbf{v}_{2}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\times\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}}{\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\gamma-\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}}\right)+o(1) \displaystyle=\frac{1}{c}\left((\mathbf{v}_{2}-m(\gamma)\mathbf{v})^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+m(\gamma)\mathbf{v}_{2}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\times\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+c\delta_{1}(\gamma)\left(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)\right)+o(1), for \mathbf{v}_{1}^{\top}=\begin{bmatrix}0&1&0&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6},\mathbf{v}_{2}^{\top}=\begin{bmatrix}1&0&0&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6} , where we used the fact that \boldsymbol{\alpha}_{-i}/\sqrt{p}=\mathbf{U}_{0}\mathbf{v}_{2} , f(\boldsymbol{\alpha}_{-i})/\sqrt{p}=\mathbf{U}_{0}\mathbf{v}_{1} , 7 , and the relation in ( 53 ). Self-consistent equation for \delta_{3}(\gamma) . We consider now \delta_{3}(\gamma)=\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K})+o(1) and write \displaystyle\delta_{3}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{K}\mathbf{Q}\mathbf{K})+o(1)=\frac{1}{c}[\mathbf{K}\mathbf{Q}\mathbf{K}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}f(\boldsymbol{\alpha}_{-i})^{\top}/\sqrt{p}&0\end{bmatrix}\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix} \displaystyle\times\begin{bmatrix}f(\boldsymbol{\alpha}_{-i})/\sqrt{p}\\ 0\end{bmatrix}+o(1)=\frac{1}{c}\mathbf{v}_{1}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{1}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+\frac{(\mathbf{v}_{1}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v})^{2}}{\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\gamma-\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}}\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{1}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{1}+m(\gamma)(\mathbf{v}_{1}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v})^{2}\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{1}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+m(\gamma)(\mathbf{v}_{1}^{\top}\mathbf{T}(\gamma)\mathbf{v})^{2}\right)+o(1)=\frac{1}{c}\left(\mathbf{v}_{1}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+\frac{c^{2}\delta_{1}^{2}(\gamma)}{m(\gamma)}\right)+o(1). Self-consistent equation for \delta_{4}(\gamma) . We consider now \delta_{4}(\gamma)=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}(\gamma)\mathbf{K}\check{\mathbf{Z}})+o(1) and write \displaystyle\delta_{4}(\gamma)=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}})+o(1)=\frac{1}{c}[\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}(\mathbf{K}_{-i}^{\perp}\boldsymbol{\alpha}_{-i}+\frac{a_{1}}{c}\boldsymbol{\alpha}_{-i}+f(\boldsymbol{\alpha}_{-i}))^{\top}/\sqrt{p}&\frac{a_{1}}{c}\end{bmatrix} \displaystyle\times\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}_{0}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}_{0}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}\begin{bmatrix}*\\ *\end{bmatrix}+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{4}-\frac{2a_{1}m(\gamma)}{c}\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}+\frac{a_{1}^{2}}{c^{2}}m(\gamma)\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{4}+\frac{(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v})^{2}}{\frac{a_{1}^{2}}{c^{2}}+\frac{\nu}{c}+\gamma-\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}}-\frac{2a_{1}m(\gamma)}{c}\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\right) \displaystyle+\frac{a_{1}^{2}}{c^{2}\times c}m(\gamma)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{4}+m(\gamma)\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}-\frac{a_{1}}{c}\right)^{2}\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{4}+m(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)^{2}\right)+o(1), for \mathbf{v}_{4}^{\top}=\begin{bmatrix}\frac{a_{1}}{c}&1&1&0&0&0\end{bmatrix}\in{\mathbb{R}}^{6}. (60) Putting these together, we obtain the system of equations as in ( 28 ). We thus conclude the proof of 1 . C.4 Proof of 1 Here, we provide detailed derivations of 1 on the Deterministic Equivalent of the in-context memorization error E in ( 6 ) 3 . To do this, recall the following structured nonlinear resolvent \mathbf{Q}(\gamma)=\left(\frac{1}{n}\mathbf{K}_{\mathbf{X}}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{K}_{\mathbf{X}}+\gamma\mathbf{I}_{n}\right)^{-1}, (61) in ( 7 ) of 3 . First note that by 1 , we have \mathbf{K}_{\mathbf{X}}=\mathbf{K}_{N}+\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{K}}\mathbf{V}_{Q}^{\top}+O_{\|\cdot\|}(n^{-1/2}), (62) for \boldsymbol{\Sigma}_{\mathbf{K}}\in{\mathbb{R}}^{3\times 3} defined in ( 9 ). Similarly, under 3 , we have \frac{1}{n}\mathbf{X}^{\top}\mathbf{X}=\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}+\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{X}}\mathbf{U}_{K}^{\top}=c\check{\mathbf{Z}}+\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{X}}\mathbf{U}_{K}^{\top},\quad\boldsymbol{\Sigma}_{\mathbf{X}}\equiv c\Big[\begin{smallmatrix}\|\boldsymbol{\mu}\|^{2}&1&0\\ 1&0&0\\ 0&0&0\end{smallmatrix}\Big], (63) that is of bounded norm with probability one as n,p\to\infty at the same rate. As such, we have \displaystyle\mathbf{Q}(\gamma) \displaystyle=\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\mathbf{U}\boldsymbol{\Sigma}\mathbf{U}^{\top}+\gamma\mathbf{I}_{n}\right)^{-1}+O_{\|\cdot\|}(n^{-\frac{1}{2}}) \displaystyle=\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}+O_{\|\cdot\|}(n^{-\frac{1}{2}}) \displaystyle-\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\left(\boldsymbol{\Sigma}^{-1}+\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\right)^{-1}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}, by Woodbury identity, for \mathbf{U}=\begin{bmatrix}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}&\mathbf{K}_{N}\mathbf{U}_{K}&\mathbf{V}_{Q}\end{bmatrix}\in{\mathbb{R}}^{n\times 9}, (64) with \mathbf{U}_{K}\in{\mathbb{R}}^{n\times 3} and \mathbf{V}_{Q}\in{\mathbb{R}}^{n\times 3} defined in 1 , and \boldsymbol{\Sigma}=\begin{bmatrix}\mathbf{0}_{3}&\mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{K}}\\ \mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{X}}&\boldsymbol{\Sigma}_{\mathbf{X}}\mathbf{U}_{K}^{\top}\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{K}}\\ \boldsymbol{\Sigma}_{\mathbf{K}}^{\top}&\boldsymbol{\Sigma}_{\mathbf{K}}^{\top}\mathbf{U}_{K}^{\top}\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{X}}&\boldsymbol{\Sigma}_{\mathbf{K}}^{\top}(\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}+\mathbf{U}_{K}^{\top}\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{X}}\mathbf{U}_{K}^{\top}\mathbf{U}_{K})\boldsymbol{\Sigma}_{\mathbf{K}}\end{bmatrix}\in{\mathbb{R}}^{9\times 9}, (65) Our objective of interest is the the memorization error E defined in ( 7 ) of 3 as E=-\frac{\gamma^{2}}{n}\frac{\partial\mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y}}{\partial\gamma}. (66) Note that \mathbf{y}/\sqrt{p} is the first column of \mathbf{V}_{Q} and thus the seventh column of \mathbf{U} defined in ( 64 ), so that \displaystyle\frac{1}{n}\mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y}=c\cdot\mathbf{e}_{7}^{\top}\mathbf{U}^{\top}\mathbf{Q}(\gamma)\mathbf{U}\mathbf{e}_{7} \displaystyle=c\cdot\mathbf{e}_{7}^{\top}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\cdot\left(\mathbf{I}_{9}+\boldsymbol{\Sigma}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\right)^{-1}\mathbf{e}_{7}+O(n^{-\frac{1}{2}}), (67) where \mathbf{e}_{7}\in{\mathbb{R}}^{9} is the canonical vector at location seven. We have the following approximation for the above objective of interest. Lemma 8 (Further approximations) . For \boldsymbol{\Sigma} defined in ( 65 ) and \mathbf{U} in ( 64 ), we have the following approximations in spectral norm holds with high probability as n,p\to\infty with p/n\to c\in(0,\infty) , \displaystyle\boldsymbol{\Sigma}=\boldsymbol{\Lambda}+O_{\|\cdot\|}(n^{-\frac{1}{2}}) (68) \displaystyle\mathbf{U}^{\top}\left(\frac{1}{p}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\frac{\gamma}{c}\mathbf{I}_{n}\right)^{-1}\mathbf{U}=\boldsymbol{\Delta}(\gamma)+O_{\|\cdot\|}(n^{-\frac{1}{2}}), (69) with \boldsymbol{\Lambda}=\begin{bmatrix}\mathbf{0}_{3}&\mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{K}}\\ \mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{X}}&[\boldsymbol{\Lambda}]_{2,3}\\ \boldsymbol{\Sigma}_{\mathbf{K}}^{\top}&[\boldsymbol{\Lambda}]_{2,3}^{\top}&[\boldsymbol{\Lambda}]_{3,3}\\ \end{bmatrix}\in{\mathbb{R}}^{9\times 9} and \boldsymbol{\Delta}(\gamma)=\begin{bmatrix}[\boldsymbol{\Delta}(\gamma)]_{1,1}&[\boldsymbol{\Delta}(\gamma)]_{1,2}&[\boldsymbol{\Delta}(\gamma)]_{1,3}\\ [\boldsymbol{\Delta}(\gamma)]_{1,2}^{\top}&[\boldsymbol{\Delta}(\gamma)]_{2,2}&[\boldsymbol{\Delta}(\gamma)]_{2,3}\\ [\boldsymbol{\Delta}(\gamma)]_{1,3}^{\top}&[\boldsymbol{\Delta}(\gamma)]_{2,3}^{\top}&[\boldsymbol{\Delta}(\gamma)]_{3,3}\\ \end{bmatrix}\in{\mathbb{C}}^{9\times 9} both three-by-three block symmetric matrices with corresponding blocks given by \displaystyle[\boldsymbol{\Lambda}]_{2,3} \displaystyle=a_{1}\begin{bmatrix}(\|\boldsymbol{\mu}\|^{2}+1)T_{1}&\|\boldsymbol{\mu}\|^{2}&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}(\|\boldsymbol{\mu}\|^{2}+1)\\ T_{1}&1&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 0&0&0\end{bmatrix} \displaystyle[\boldsymbol{\Lambda}]_{3,3} \displaystyle=a_{1}^{2}\begin{bmatrix}\frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}T_{1}^{2}+\frac{1+c}{c}T_{1}+\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\left(\boldsymbol{\mu}^{\top}\mathbf{w}_{K}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\|\mathbf{w}_{K}\|^{2}\right)&(*)&(*)\\ \frac{1+c+\|\boldsymbol{\mu}\|^{2}}{c}T_{1}&1+\frac{\|\boldsymbol{\mu}\|^{2}}{c}&\frac{1+c+\|\boldsymbol{\mu}\|^{2}}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ \frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}T_{1}+\frac{1+c}{c}(\boldsymbol{\mu}^{\top}\mathbf{w}_{K}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\|\mathbf{w}_{K}\|^{2})&(*)&\frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}(\boldsymbol{\mu}^{\top}\mathbf{w}_{K})^{2}+\frac{1+c}{c}\|\mathbf{w}_{K}\|^{2}\end{bmatrix} for T_{1}=\|\boldsymbol{\mu}\|^{2}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}, (70) and \boldsymbol{\Sigma}_{\mathbf{K}}\equiv a_{1}\left[\begin{smallmatrix}T_{1}&1&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 1&0&0\\ \boldsymbol{\mu}^{\top}\mathbf{w}_{Q}&0&1\end{smallmatrix}\right]\in{\mathbb{R}}^{3\times 3} defined in ( 9 ) of 1 , \boldsymbol{\Sigma}_{\mathbf{X}}\equiv c\Big[\begin{smallmatrix}\|\boldsymbol{\mu}\|^{2}&1&0\\ 1&0&0\\ 0&0&0\end{smallmatrix}\Big]\in{\mathbb{R}}^{3\times 3} , as well as \displaystyle[\boldsymbol{\Delta}(\gamma)]_{1,1} \displaystyle=\begin{bmatrix}c^{2}\delta_{4}(\gamma)&0&0\\ 0&c^{2}\|\boldsymbol{\mu}\|^{2}\delta_{7}(\gamma)&c^{2}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{7}(\gamma)\\ 0&(*)&c^{2}\|\mathbf{w}_{K}\|^{2}\delta_{7}(\gamma)&\\ \end{bmatrix}\in{\mathbb{R}}^{3\times 3} \displaystyle[\boldsymbol{\Delta}(\gamma)]_{1,2} \displaystyle=\begin{bmatrix}1-\frac{\gamma}{c}m(\gamma)&0&0\\ 0&c\|\boldsymbol{\mu}\|^{2}\delta_{4}(\gamma)&c\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{4}(\gamma)\\ 0&c\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{4}(\gamma)&c\|\mathbf{w}_{K}\|^{2}\delta_{4}(\gamma)\\ \end{bmatrix}\in{\mathbb{R}}^{3\times 3} \displaystyle[\boldsymbol{\Delta}(\gamma)]_{1,3} \displaystyle=\begin{bmatrix}c\delta_{2}(\gamma)&0&0\\ 0&c\|\boldsymbol{\mu}\|^{2}\delta_{6}(\gamma)&c\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{6}(\gamma)\\ 0&c\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{6}(\gamma)&c\mathbf{w}_{K}^{\top}\mathbf{w}_{Q}\delta_{6}(\gamma)\\ \end{bmatrix}\in{\mathbb{R}}^{3\times 3} \displaystyle[\boldsymbol{\Delta}(\gamma)]_{2,2} \displaystyle=\begin{bmatrix}\delta_{3}(\gamma)&0&0\\ 0&\frac{1}{c}\|\boldsymbol{\mu}\|^{2}(1-\frac{\gamma}{c}m(\gamma))&\frac{1}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}(1-\frac{\gamma}{c}m(\gamma))\\ 0&(*)&\frac{1}{c}\|\mathbf{w}_{K}\|^{2}(1-\frac{\gamma}{c}m(\gamma))\\ \end{bmatrix}\in{\mathbb{R}}^{3\times 3} \displaystyle[\boldsymbol{\Delta}(\gamma)]_{2,3} \displaystyle=\begin{bmatrix}\delta_{1}(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{2}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{2}(\gamma)\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{2}(\gamma)&\mathbf{w}_{K}^{\top}\mathbf{w}_{Q}\delta_{2}(\gamma)\\ \end{bmatrix}\in{\mathbb{R}}^{3\times 3} \displaystyle[\boldsymbol{\Delta}(\gamma)]_{3,3} \displaystyle=\begin{bmatrix}\frac{1}{c}m(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{5}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{5}(\gamma)\\ 0&(*)&\|\mathbf{w}_{Q}\|^{2}\delta_{5}(\gamma)\end{bmatrix}\in{\mathbb{R}}^{3\times 3}, for \delta_{1}(\gamma),\delta_{2}(\gamma),\delta_{3}(\gamma),\delta_{4}(\gamma) as defined in ( 27 ) of the proof of 1 , and \left\{\begin{array}[]{ll}c\delta_{5}(\gamma)&=m(\gamma)\left(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)\\ c\delta_{6}(\gamma)&=\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{2}+m(\gamma)(\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}-1)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\\ c\delta_{7}(\gamma)&=\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{7}+m(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\left(\mathbf{v}_{7}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\left(2+\frac{1}{c}\right)\right)\end{array}\right. (71) with \mathbf{v}_{7}^{\top}=\begin{bmatrix}2\frac{a_{1}}{c}+\frac{a_{1}}{c^{2}}&\frac{1}{c}+1&\frac{1}{c}+1&0&1&0\end{bmatrix}\in{\mathbb{R}}^{6}. (72) Proof of 8 . We first work on the approximation of \boldsymbol{\Sigma} defined in ( 65 ), for which we exploit the following concentration results: \displaystyle\mathbf{U}_{K}^{\top}\mathbf{U}_{K} \displaystyle=\frac{1}{c}\begin{bmatrix}1&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}&\|\mathbf{w}_{K}\|^{2}\end{bmatrix}+O_{\|\cdot\|}(n^{-\frac{1}{2}}), \displaystyle\frac{1}{n}\mathbf{U}_{K}^{\top}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K} \displaystyle=\begin{bmatrix}1&0&0\\ 0&\frac{1+c}{c}\|\boldsymbol{\mu}\|^{2}&\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 0&\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}&\frac{1+c}{c}\|\mathbf{w}_{K}\|^{2}\end{bmatrix}+O_{\|\cdot\|}(n^{-\frac{1}{2}}), where we used the Gaussian moments, we thus get \mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}+\mathbf{U}_{K}^{\top}\mathbf{U}_{K}\boldsymbol{\Sigma}_{\mathbf{X}}\mathbf{U}_{K}^{\top}\mathbf{U}_{K}=\begin{bmatrix}1+\|\boldsymbol{\mu}\|^{2}/c&\|\boldsymbol{\mu}\|^{2}/c&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}/c\\ \|\boldsymbol{\mu}\|^{2}/c&\frac{1+c}{c}\|\boldsymbol{\mu}\|^{2}&\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ \boldsymbol{\mu}^{\top}\mathbf{w}_{K}/c&\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}&\frac{1+c}{c}\|\mathbf{w}_{K}\|^{2}\end{bmatrix}+O_{\|\cdot\|}(n^{-\frac{1}{2}}), (73) and therefore \boldsymbol{\Sigma}=\boldsymbol{\Lambda}+O_{\|\cdot\|}(n^{-\frac{1}{2}}) with \boldsymbol{\Sigma}=\begin{bmatrix}\mathbf{0}_{3}&\mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{K}}\\ \mathbf{0}_{3}&\boldsymbol{\Sigma}_{\mathbf{X}}&[\boldsymbol{\Lambda}]_{2,3}\\ \boldsymbol{\Sigma}_{\mathbf{K}}^{\top}&[\boldsymbol{\Lambda}]_{2,3}^{\top}&[\boldsymbol{\Lambda}]_{3,3}\\ \end{bmatrix}, (74) and \displaystyle[\boldsymbol{\Lambda}]_{2,3} \displaystyle=a_{1}\begin{bmatrix}(\|\boldsymbol{\mu}\|^{2}+1)T_{1}&\|\boldsymbol{\mu}\|^{2}&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}(\|\boldsymbol{\mu}\|^{2}+1)\\ T_{1}&1&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ 0&0&0\end{bmatrix} \displaystyle[\boldsymbol{\Lambda}]_{3,3} \displaystyle=a_{1}^{2}\begin{bmatrix}\frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}T_{1}^{2}+\frac{1+c}{c}T_{1}+\frac{1+c}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\left(\boldsymbol{\mu}^{\top}\mathbf{w}_{K}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\|\mathbf{w}_{K}\|^{2}\right)&(*)&(*)\\ \frac{1+c+\|\boldsymbol{\mu}\|^{2}}{c}T_{1}&1+\frac{\|\boldsymbol{\mu}\|^{2}}{c}&\frac{1+c+\|\boldsymbol{\mu}\|^{2}}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\\ \frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}\boldsymbol{\mu}^{\top}\mathbf{w}_{K}T_{1}+\frac{1+c}{c}(\boldsymbol{\mu}^{\top}\mathbf{w}_{K}+\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\|\mathbf{w}_{K}\|^{2})&(*)&\frac{2+c+\|\boldsymbol{\mu}\|^{2}}{c}(\boldsymbol{\mu}^{\top}\mathbf{w}_{K})^{2}+\frac{1+c}{c}\|\mathbf{w}_{K}\|^{2}\end{bmatrix}. where we denote the shortcut T_{1}=\|\boldsymbol{\mu}\|^{2}+\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\boldsymbol{\mu}^{\top}\mathbf{w}_{Q} , This concludes the proof of the approximation of \boldsymbol{\Sigma} in 8 . We then proceed to the approximation of \mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U} . Note that for \mathbf{U} defined in ( 64 ) and \mathbf{Q}_{0}\equiv\left(\frac{1}{p}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\frac{\gamma}{c}\mathbf{I}_{n}\right)^{-1}, (75) we have \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U}=\begin{bmatrix}\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}&\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{U}_{K}&\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{V}_{Q}\\ \mathbf{U}_{K}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}&\mathbf{U}_{K}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{U}_{K}&\mathbf{U}_{K}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{V}_{Q}\\ \mathbf{V}_{Q}^{\top}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}&\mathbf{V}_{Q}^{\top}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{U}_{K}&\mathbf{V}_{Q}^{\top}\mathbf{Q}_{0}\mathbf{V}_{Q}\end{bmatrix}\in{\mathbb{R}}^{9\times 9}, (76) which writes as a three-by-three block matrix, for \mathbf{U}_{K}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{K}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3} , \mathbf{V}_{Q}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu},~\mathbf{Z}^{\top}\mathbf{w}_{Q}]/\sqrt{p}\in{\mathbb{R}}^{n\times 3} as in 1 . In the following, we further evaluate the nine (in fact six by symmetry) blocks of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} , in the limit of n,p\to\infty with p/n\to c\in(0,\infty) . To that end, we need the following intermediate results. Lemma 9 (Further Deterministic Equivalents) . Under the same settings and notations as in 1 , we have the following Deterministic Equivalent results (in the sense of 4 ) \displaystyle\frac{1}{n^{2}}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z} \displaystyle\leftrightarrow c^{3}\delta_{4}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N} \displaystyle\leftrightarrow\left(c-\gamma m(\gamma)\right)\cdot\mathbf{I}_{n}, \displaystyle\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0} \displaystyle\leftrightarrow c^{2}\delta_{2}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N} \displaystyle\leftrightarrow c\delta_{3}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\mathbf{K}_{N}\mathbf{Q}_{0} \displaystyle\leftrightarrow c\delta_{1}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top} \displaystyle\leftrightarrow c^{2}\delta_{7}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top} \displaystyle\leftrightarrow c\delta_{4}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow c\delta_{6}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{1}{c}\left(1-\frac{\gamma}{c}m(\gamma)\right)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\delta_{2}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\delta_{5}(\gamma)\cdot\mathbf{I}_{p}. Proof of 9 . Note that for \mathbf{Q}_{0} defined in ( 75 ), we have, by the proof of 1 in Section C.3 ., the following Deterministic Equivalent results. \displaystyle\frac{1}{n^{2}}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z} \displaystyle\leftrightarrow\frac{1}{n^{2}}{\mathbb{E}}[\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}]\leftrightarrow\frac{p^{3}}{n^{3}}\tr\frac{1}{p}\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{n}\leftrightarrow c^{3}\delta_{4}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N} \displaystyle\leftrightarrow\frac{1}{n}{\mathbb{E}}[\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}]\leftrightarrow\frac{p}{n}\frac{1}{n}\tr\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\right)\cdot\mathbf{I}_{n}\leftrightarrow\left(c-\gamma m(\gamma)\right)\cdot\mathbf{I}_{n}, \displaystyle\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0} \displaystyle\leftrightarrow\frac{1}{n}{\mathbb{E}}[\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}]\leftrightarrow\frac{p^{2}}{n^{2}}\frac{1}{p}\tr\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\right)\cdot\mathbf{I}_{n}\leftrightarrow c^{2}\delta_{2}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N} \displaystyle\leftrightarrow{\mathbb{E}}[\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}]\leftrightarrow\frac{p}{n}\frac{1}{p}\tr\left(\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\right)\cdot\mathbf{I}_{n}\leftrightarrow c\delta_{3}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\mathbf{K}_{N}\mathbf{Q}_{0} \displaystyle\leftrightarrow{\mathbb{E}}[\mathbf{K}_{N}\mathbf{Q}_{0}]\leftrightarrow\frac{p}{n}\frac{1}{p}\tr\left(\mathbf{K}_{N}\mathbf{Q}_{0}\right)\cdot\mathbf{I}_{n}\leftrightarrow c\delta_{1}(\gamma)\cdot\mathbf{I}_{n}, \displaystyle\mathbf{Q}_{0} \displaystyle\leftrightarrow m(\gamma)\cdot\mathbf{I}_{n}. Similarly, we have \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{p^{2}}{n^{2}}\frac{1}{p}\tr\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{p}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow c^{2}\delta_{7}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{p}{n}\frac{1}{p}\tr\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow c\delta_{4}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{p}{n}\frac{1}{p}\tr\left(\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow c\delta_{6}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{1}{p}\tr\left(\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow\frac{1}{c}\left(1-\frac{\gamma}{c}m(\gamma)\right)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{1}{p}\tr\left(\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow\delta_{2}(\gamma)\cdot\mathbf{I}_{p}, \displaystyle\frac{1}{p}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top} \displaystyle\leftrightarrow\frac{1}{p}\tr\left(\mathbf{Q}_{0}\frac{1}{p}\mathbf{Z}^{\top}\mathbf{Z}\right)\cdot\mathbf{I}_{p}\leftrightarrow\delta_{5}(\gamma)\cdot\mathbf{I}_{p}, for \delta_{5}(\gamma),\delta_{6}(\gamma),\delta_{7}(\gamma) as defined in ( 71 ). To complete the proof of 9 , we establish, in the following as similar to Section C.3.3 , self-consistent equations for \delta_{5}(\gamma),\delta_{6}(\gamma) and \delta_{7}(\gamma) . Self-consistent equation for \delta_{5}(\gamma) . Consider \delta_{5}(\gamma)=\frac{1}{p}\tr(\mathbf{Q}\check{\mathbf{Z}})+o(1) and write \displaystyle\delta_{5}(\gamma) \displaystyle=\frac{1}{p}\tr(\mathbf{Q}\check{\mathbf{Z}})+o(1)=\frac{1}{c}[\mathbf{Q}\check{\mathbf{Z}}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}-m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}\begin{bmatrix}\boldsymbol{\alpha}_{-i}/\sqrt{p}\\ 1\end{bmatrix}+o(1) \displaystyle=\frac{1}{c}\left(-m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}\mathbf{U}\mathbf{v}_{2}+m(\gamma)\right)+o(1) \displaystyle=\frac{m(\gamma)}{c}\left(-\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\boldsymbol{\Delta}_{0}(\gamma)\mathbf{v}_{2}+1\right)+o(1) \displaystyle=\frac{m(\gamma)}{c}\left(-\mathbf{v}_{2}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}+1\right)+o(1) \displaystyle=\frac{m(\gamma)}{c}\left(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)+o(1). Self-consistent equation for \delta_{6}(\gamma) . Consider now \delta_{6}(\gamma)=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\check{\mathbf{Z}})+o(1) and write \displaystyle\delta_{6}(\gamma) \displaystyle=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\check{\mathbf{Z}})+o(1)=\frac{1}{c}[\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\check{\mathbf{Z}}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}\mathbf{v}_{4}^{\top}\mathbf{U}^{\top}&\frac{a_{1}}{c}\end{bmatrix}\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}\begin{bmatrix}\mathbf{U}\mathbf{v}_{2}\\ 1\end{bmatrix}+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{2}-m(\gamma)\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\right. \displaystyle\left.-\frac{a_{1}}{c}m(\gamma)\mathbf{v}^{\top}(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0})^{-1}\boldsymbol{\Delta}_{0}(\gamma)\mathbf{v}_{2}+\frac{a_{1}}{c}m(\gamma)\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{2}+m(\gamma)\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}\times\mathbf{v}^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}_{2}\right. \displaystyle\left.-m(\gamma)\left(\frac{a_{1}}{c}\mathbf{v}_{2}+\mathbf{v}_{4}\right)^{\top}\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\mathbf{v}+\frac{a_{1}}{c}m(\gamma)\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{2}+m(\gamma)(\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}-1)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\right)+o(1). Self-consistent equation for \delta_{7}(\gamma) . Consider \delta_{7}(\gamma)=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}}\check{\mathbf{Z}})+o(1) and write \displaystyle\delta_{7}(\gamma) \displaystyle=\frac{1}{p}\tr(\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}}\check{\mathbf{Z}})+o(1)=\frac{1}{c}[\check{\mathbf{Z}}\mathbf{K}\mathbf{Q}\mathbf{K}\check{\mathbf{Z}}\check{\mathbf{Z}}]_{ii}+o(1) \displaystyle=\frac{1}{c}\begin{bmatrix}\mathbf{v}_{4}^{\top}\mathbf{U}^{\top}&\frac{a_{1}}{c}\end{bmatrix}\begin{bmatrix}\mathbf{Q}_{-i}^{\perp}-\mathbf{Q}_{-i}^{\perp}\mathbf{U}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}(\gamma))^{-1}\boldsymbol{\Lambda}_{1}(\gamma)\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}&-m(\gamma)\mathbf{Q}_{-i}^{\perp}\mathbf{U}\left(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma)\right)^{-1}\mathbf{v}\\ -m(\gamma)\mathbf{v}^{\top}\left(\mathbf{I}_{6}+\boldsymbol{\Delta}(\gamma)\boldsymbol{\Lambda}_{0}\right)^{-1}\mathbf{U}^{\top}\mathbf{Q}_{-i}^{\perp}&[\mathbf{Q}]_{ii}\end{bmatrix}\begin{bmatrix}\mathbf{U}\mathbf{v}_{7}\\ 2\frac{a_{1}}{c}+\frac{a_{1}}{c^{2}}\end{bmatrix}+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{1}(\gamma)\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}_{7}-\left(2\frac{a_{1}}{c}+\frac{a_{1}}{c^{2}}\right)m(\gamma)\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}\right. \displaystyle\left.-\frac{a_{1}}{c}m(\gamma)\left(\mathbf{v}^{\top}(\mathbf{I}_{6}+\boldsymbol{\Delta}(\gamma)\boldsymbol{\Lambda}_{0})^{-1}\boldsymbol{\Delta}(\gamma)\mathbf{v}_{7}-\left(2\frac{a_{1}}{c}+\frac{a_{1}}{c^{2}}\right)\right)\right)+o(1) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}_{7}+m(\gamma)\mathbf{v}_{4}^{\top}\boldsymbol{\Delta}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}\mathbf{v}^{\top}\boldsymbol{\Delta}(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}_{7}\right. \displaystyle\left.-\frac{a_{1}}{c}m(\gamma)\left(\left(\frac{1}{c}+2\right)\mathbf{v}_{4}+\mathbf{v}_{7}\right)^{\top}\boldsymbol{\Delta}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}(\gamma))^{-1}\mathbf{v}+\frac{a_{1}^{2}}{c^{2}}\left(\frac{1}{c}+2\right)m(\gamma)\right)+o(1). With these self-consistent equations for \delta_{5}(\gamma),\delta_{6}(\gamma),\delta_{7}(\gamma) , we conclude the proof of 9 . ∎ With 9 at hand, we are now ready to evaluate the blocks of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} as follows. Approximation of the (1,1) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}(\gamma)\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{U}_{K}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\end{bmatrix}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{y}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{y}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=c^{2}\begin{bmatrix}\delta_{4}(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{7}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{7}(\gamma)\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{7}(\gamma)&\|\mathbf{w}_{K}\|^{2}\delta_{7}(\gamma)\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we used 9 for the approximation in the last line. Approximation of the (1,2) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}(\gamma)\mathbf{K}_{N}\mathbf{U}_{K}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\end{bmatrix}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=\begin{bmatrix}1-\frac{\gamma}{c}m(\gamma)&0&0\\ 0&c\|\boldsymbol{\mu}\|^{2}\delta_{4}(\gamma)&c\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{4}(\gamma)\\ 0&c\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{4}(\gamma)&c\|\mathbf{w}_{K}\|^{2}\delta_{4}(\gamma)\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we used 9 for the approximation in the last line. Approximation of the (1,3) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{U}_{K}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}(\gamma)\mathbf{V}_{Q}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\end{bmatrix}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{K}^{\top}\mathbf{Z}\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=c\begin{bmatrix}\delta_{2}(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{6}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{6}(\gamma)\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{6}(\gamma)&\mathbf{w}_{K}^{\top}\mathbf{w}_{Q}\delta_{6}(\gamma)\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we used 9 for the approximation in the last line. Approximation of the (2,2) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{U}_{K}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}(\gamma)\mathbf{K}_{N}\mathbf{U}_{K}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\end{bmatrix}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{y}&\mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{w}_{K}\end{bmatrix} \displaystyle=\frac{1}{c}\begin{bmatrix}c\delta_{3}(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}(1-\frac{\gamma}{c}m(\gamma))&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}(1-\frac{\gamma}{c}m(\gamma))\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}(1-\frac{\gamma}{c}m(\gamma))&\|\mathbf{w}_{K}\|^{2}(1-\frac{\gamma}{c}m(\gamma))\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we used 9 for the approximation in the last line. Approximation of the (2,3) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{U}_{K}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}(\gamma)\mathbf{V}_{Q}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\end{bmatrix}\mathbf{K}_{N}\mathbf{Q}_{0}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{y}&\mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{K}^{\top}\mathbf{Z}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=\begin{bmatrix}\delta_{1}(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{2}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{2}(\gamma)\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{K}\delta_{2}(\gamma)&\mathbf{w}_{K}^{\top}\mathbf{w}_{Q}\delta_{2}(\gamma)\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we again use 9 for the approximation in the last line. Approximation of the (3,3) block of \mathbf{U}^{\top}\mathbf{Q}_{0}\mathbf{U} . \displaystyle\mathbf{V}_{Q}^{\top}\mathbf{Q}_{0}(\gamma)\mathbf{V}_{Q}=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\\ \mathbf{w}_{Q}^{\top}\mathbf{Z}\end{bmatrix}\mathbf{K}_{N}\mathbf{Q}_{0}\mathbf{K}_{N}\begin{bmatrix}\mathbf{y}&\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=\frac{1}{p}\begin{bmatrix}\mathbf{y}^{\top}\mathbf{Q}_{0}\mathbf{y}&\mathbf{y}^{\top}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{y}^{\top}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{y}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\boldsymbol{\mu}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\\ \mathbf{w}_{Q}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{y}&\mathbf{w}_{Q}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top}\boldsymbol{\mu}&\mathbf{w}_{Q}^{\top}\mathbf{Z}\mathbf{Q}_{0}\mathbf{Z}^{\top}\mathbf{w}_{Q}\end{bmatrix} \displaystyle=\begin{bmatrix}\frac{n}{p}m(\gamma)&0&0\\ 0&\|\boldsymbol{\mu}\|^{2}\delta_{5}(\gamma)&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{5}(\gamma)\\ 0&\boldsymbol{\mu}^{\top}\mathbf{w}_{Q}\delta_{5}(\gamma)&\|\mathbf{w}_{Q}\|^{2}\delta_{5}(\gamma)\end{bmatrix}+O_{\|\cdot\|}(n^{-1/2}), where we used 9 for the approximation in the last line. This concludes the proof of the approximation of the quadratic form \mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U} in 8 . ∎ With 8 at hand, it follows from ( 67 ) that \displaystyle\frac{1}{n}\mathbf{y}^{\top}\mathbf{Q}(\gamma)\mathbf{y} \displaystyle=c\cdot\mathbf{e}_{7}^{\top}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\cdot\left(\mathbf{I}_{9}+\boldsymbol{\Sigma}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{K}_{N}\mathbf{Z}^{\top}\mathbf{Z}\mathbf{K}_{N}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{U}\right)^{-1}\mathbf{e}_{7}+O(n^{-\frac{1}{2}}) \displaystyle=c\cdot\mathbf{e}_{7}^{\top}\boldsymbol{\Delta}(\gamma)\cdot\left(\mathbf{I}_{9}+\boldsymbol{\Lambda}\boldsymbol{\Delta}(\gamma)\right)^{-1}\mathbf{e}_{7}+O(n^{-\frac{1}{2}}). To assess the high-dimensional behavior of the memorization error E defined in ( 7 ) of 3 , it thus remains to evaluate the following derivative (with respective to \gamma ) as \displaystyle E \displaystyle=-\frac{\gamma^{2}}{n}\frac{\partial y^{\top}\mathbf{Q}(\gamma)y}{\partial\gamma}=-\gamma^{2}c^{2}\cdot\mathbf{e}_{7}^{\top}\left(c\mathbf{I}_{9}+\boldsymbol{\Delta}(\gamma)\boldsymbol{\Lambda}\right)^{-1}\boldsymbol{\Delta}^{\prime}(\gamma)\left(c\mathbf{I}_{9}+\boldsymbol{\Lambda}\boldsymbol{\Delta}(\gamma)\right)^{-1}\mathbf{e}_{7}+O(n^{-\frac{1}{2}}), where we denote \boldsymbol{\Delta}^{\prime}(\gamma) the derivative (with respect to \gamma ) of \boldsymbol{\Delta}(\gamma) defined in ( 69 ). To evaluate \boldsymbol{\Delta}^{\prime}(\gamma) , we need the following result on the derivatives of m^{\prime}(\gamma) and \delta_{(}\gamma) s. Lemma 10 (Derivatives of the \delta(\gamma) s) . Under the settings and notations of 1 , we have that m^{\prime}(\gamma),\delta_{1}^{\prime}(\gamma),\delta_{2}^{\prime}(\gamma),\delta_{3}^{\prime}(\gamma),\delta_{4}^{\prime}(\gamma) satisfy the following system of equations \left\{\begin{array}[]{ll}m^{\prime}(\gamma)&=\left(\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}-\frac{1}{c}\right)m^{2}(\gamma)\\ c\delta_{1}^{\prime}(\gamma)&=-m^{\prime}(\gamma)\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}-m(\gamma)\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{1}\\ c\delta_{2}^{\prime}(\gamma)&=\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)(\mathbf{v}_{1}-c\delta_{1}(\gamma)\mathbf{v})+c\delta^{\prime}_{1}(\gamma)(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v})\\ c\delta_{3}^{\prime}(\gamma)&=\mathbf{v}_{1}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{1}+\frac{c^{2}\delta_{1}(\gamma)\left(2\delta_{1}^{\prime}(\gamma)m(\gamma)-\delta_{1}(\gamma)m^{\prime}(\gamma)\right)}{m^{2}(\gamma)}\\ c\delta_{4}^{\prime}(\gamma)&=\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{4}+m^{\prime}(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)^{2}+2m(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\end{array}\right. (77) for \mathbf{T}(\gamma) and \boldsymbol{\Delta}_{0}(\gamma) defined in ( 29 ) and ( 30 ), receptively, so that their derivatives (with respective to \gamma ) satisfy \displaystyle\mathbf{T}^{\prime}(\gamma) \displaystyle=(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0})^{-1}\boldsymbol{\Delta}_{0}^{\prime}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}, and \boldsymbol{\Delta}_{0}^{\prime}(\gamma)\equiv\begin{bmatrix}\frac{m^{\prime}(\gamma)}{c}&\frac{a_{1}}{c}m^{\prime}(\gamma)&\delta_{1}^{\prime}(\gamma)&a_{1}\delta_{1}^{\prime}(\gamma)&\delta_{2}^{\prime}(\gamma)&a_{1}\delta_{2}^{\prime}(\gamma)\\ \frac{a_{1}}{c}m^{\prime}(\gamma)&\frac{\nu}{c}m^{\prime}(\gamma)&a_{1}\delta_{1}^{\prime}(\gamma)&\nu\delta_{1}^{\prime}(\gamma)&a_{1}\delta_{2}^{\prime}(\gamma)&\nu\delta_{2}^{\prime}(\gamma)\\ \delta_{1}^{\prime}(\gamma)&a_{1}\delta_{1}^{\prime}(\gamma)&\delta_{3}^{\prime}(\gamma)&a_{1}\delta_{3}^{\prime}(\gamma)&-\frac{1}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&-\frac{a_{1}}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))\\ a_{1}\delta_{1}^{\prime}(\gamma)&\nu\delta_{1}^{\prime}(\gamma)&a_{1}\delta_{3}^{\prime}(\gamma)&\nu\delta_{3}^{\prime}(\gamma)&-\frac{a_{1}}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&-\frac{\nu}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))\\ \delta_{2}^{\prime}(\gamma)&a_{1}\delta_{2}^{\prime}(\gamma)&-\frac{1}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&-\frac{a_{1}}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&\delta_{4}^{\prime}(\gamma)&a_{1}\delta_{4}^{\prime}(\gamma)\\ a_{1}\delta_{2}^{\prime}(\gamma)&\nu\delta_{2}^{\prime}(\gamma)&-\frac{a_{1}}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&-\frac{\nu}{c^{2}}(m(\gamma)+\gamma m^{\prime}(\gamma))&a_{1}\delta_{4}^{\prime}(\gamma)&\nu\delta_{4}^{\prime}(\gamma)\end{bmatrix} Also, the derivatives \delta_{5}^{\prime}(\gamma),\delta_{6}^{\prime}(\gamma),\delta_{7}^{\prime}(\gamma) are given by \displaystyle c\delta^{\prime}_{5}(\gamma) \displaystyle=m^{\prime}(\gamma)\left(-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}+1\right)-m(\gamma)\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v} \displaystyle c\delta^{\prime}_{6}(\gamma) \displaystyle=\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{2}+m^{\prime}(\gamma)\left(\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}-1\right)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)+m(\gamma)\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right) \displaystyle+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\left(\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}-1\right) \displaystyle c\delta^{\prime}_{7}(\gamma) \displaystyle=\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{7}+m^{\prime}(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\left(\mathbf{v}_{7}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\left(1+\frac{1}{c}\right)\right)+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\left(\mathbf{v}_{7}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\left(1+\frac{1}{c}\right)\right) \displaystyle+m(\gamma)\mathbf{v}_{7}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right). Proof of 10 . By their definitions in 1 and 8 , we have \displaystyle m^{\prime}(\gamma) \displaystyle=-\frac{\frac{1}{c}-\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}}{\left(\frac{\gamma}{c}+\frac{\nu}{c}+\frac{a_{1}^{2}}{c^{2}}-\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}\right)^{2}}=\left(\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}-\frac{1}{c}\right)m^{2}(\gamma) \displaystyle c\delta_{1}^{\prime}(\gamma) \displaystyle=-\left(m^{\prime}(\gamma)\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{1}+m(\gamma)\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{1}\right) \displaystyle c\delta_{2}^{\prime}(\gamma) \displaystyle=\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{1}+c\delta^{\prime}_{1}(\gamma)(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v})-c\delta_{1}(\gamma)\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}=\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)(\mathbf{v}_{1}-c\delta_{1}(\gamma)\mathbf{v})+c\delta^{\prime}_{1}(\gamma)(1-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}) \displaystyle c\delta_{3}^{\prime}(\gamma) \displaystyle=\mathbf{v}_{1}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{1}+\frac{c^{2}\delta_{1}(\gamma)\left(2\delta_{1}^{\prime}(\gamma)m(\gamma)-\delta_{1}(\gamma)m^{\prime}(\gamma)\right)}{m^{2}(\gamma)} \displaystyle c\delta_{4}^{\prime}(\gamma) \displaystyle=\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{4}+m^{\prime}(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)^{2}+2m(\gamma)\left(\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}\right)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}, with \displaystyle\mathbf{T}^{\prime}(\gamma) \displaystyle=\boldsymbol{\Delta}_{0}^{\prime}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}-\boldsymbol{\Delta}_{0}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}^{\prime}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1} \displaystyle=(\mathbf{I}_{6}+\boldsymbol{\Delta}_{0}(\gamma)\boldsymbol{\Lambda}_{0})^{-1}\boldsymbol{\Delta}_{0}^{\prime}(\gamma)(\mathbf{I}_{6}+\boldsymbol{\Lambda}_{0}\boldsymbol{\Delta}_{0}(\gamma))^{-1}, and \boldsymbol{\Delta}_{0}^{\prime}(\gamma) as in the statement of 10 . Similarly, by their definition in 9 , we obtain the derivatives of \delta_{5},\delta_{6} and \delta_{7} as \displaystyle\delta^{\prime}_{5}(\gamma) \displaystyle=\frac{1}{c}\left(m^{\prime}(\gamma)\left(-\mathbf{v}_{2}^{\top}\mathbf{T}(\gamma)\mathbf{v}+1\right)-m(\gamma)\mathbf{v}_{2}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\right) \displaystyle\delta^{\prime}_{6}(\gamma) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{2}+m^{\prime}(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{2}+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{2}\right. \displaystyle\left.+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{2}-m^{\prime}(\gamma)\left(\frac{a_{1}}{c}\mathbf{v}_{2}+\mathbf{v}_{4}\right)^{\top}\mathbf{T}(\gamma)\mathbf{v}-m(\gamma)\left(\frac{a_{1}}{c}\mathbf{v}_{2}+\mathbf{v}_{4}\right)^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}+\frac{a_{1}}{c}m^{\prime}(\gamma)\right) \displaystyle\delta^{\prime}_{7}(\gamma) \displaystyle=\frac{1}{c}\left(\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{7}+m^{\prime}(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{7}+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}(\gamma)\mathbf{v}_{7}+m(\gamma)\mathbf{v}_{4}^{\top}\mathbf{T}(\gamma)\mathbf{v}\mathbf{v}^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}_{7}\right. \displaystyle\left.-\frac{a_{1}}{c}m^{\prime}(\gamma)\left(\left(\frac{1}{c}+2\right)\mathbf{v}_{4}+\mathbf{v}_{7}\right)^{\top}\mathbf{T}(\gamma)\mathbf{v}-\frac{a_{1}}{c}m(\gamma)\left(\left(\frac{1}{c}+2\right)\mathbf{v}_{4}+\mathbf{v}_{7}\right)^{\top}\mathbf{T}^{\prime}(\gamma)\mathbf{v}+\frac{a_{1}^{2}}{c^{2}}\left(\frac{1}{c}+2\right)m^{\prime}(\gamma)\right). This concludes the proof of 10 . ∎ Putting these together, we conclude the proof of 1 . C.5 Proof of 2 Here, we provide the proof of 2 . By the definition of ridge regression (in-context) memorization error E_{\rm RR} in ( 14 ) of 5 , it suffices to evaluate the following quadratic form \frac{1}{n}\mathbf{y}^{\top}\left(\frac{1}{n}\mathbf{X}^{\top}\mathbf{X}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y}, (78) and its derivative with respect to \gamma , for \mathbf{X}=\boldsymbol{\mu}\mathbf{y}^{\top}+\mathbf{Z}\in{\mathbb{R}}^{p\times n} as in 1 . By Woodbury identity, we have \displaystyle\frac{1}{n}\mathbf{y}^{\top}\left(\frac{1}{n}\mathbf{X}^{\top}\mathbf{X}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y}=\mathbf{e}_{1}^{\top}\mathbf{U}^{\top}\left(\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}+\gamma\mathbf{I}_{n}+\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{\top}\right)^{-1}\mathbf{U}\mathbf{e}_{1} \displaystyle=\mathbf{e}_{1}^{\top}\mathbf{U}^{\top}\mathbf{Q}_{0}(\gamma)\mathbf{U}\left(\mathbf{I}_{2}+\boldsymbol{\Lambda}\mathbf{U}^{\top}\mathbf{Q}_{0}(\gamma)\mathbf{U}\right)^{-1}\mathbf{e}_{1}, where \mathbf{e}_{1}=[1,~0]^{\top} and with a slight abuse of notations, we denote \mathbf{U}=[\mathbf{y},~\mathbf{Z}^{\top}\boldsymbol{\mu}]/\sqrt{n}\in{\mathbb{R}}^{n\times 2},\quad\boldsymbol{\Lambda}=\begin{bmatrix}\|\boldsymbol{\mu}\|^{2}&1\\ 1&0\end{bmatrix}\in{\mathbb{R}}^{2\times 2},\quad\mathbf{Q}_{0}(\gamma)=\left(\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}+\gamma\mathbf{I}_{n}\right)^{-1}. (79) Similar to 1 , we have the following Deterministic Equivalent result for the linear resolvent \mathbf{Q}_{0}(\gamma) . Lemma 11 (Deterministic Equivalent for \mathbf{Q}_{0} , [ 22 , Theorem 2.4] ) . Let \mathbf{Z}\in{\mathbb{R}}^{p\times n} have i.i.d. standard Gaussian entries. Then, as n,p\to\infty at the same pace with p/n\to c\in(0,\infty) and \gamma>0 , the following Deterministic Equivalent (see 4 ) holds \left(\frac{1}{n}\mathbf{Z}\mathbf{Z}^{\top}+\gamma\mathbf{I}_{p}\right)^{-1}\leftrightarrow m_{\rm RR}(\gamma)\cdot\mathbf{I}_{p},\quad\left(\frac{1}{n}\mathbf{Z}^{\top}\mathbf{Z}+\gamma\mathbf{I}_{n}\right)^{-1}\leftrightarrow\left(cm_{\rm RR}(\gamma)+\frac{1-c}{\gamma}\right)\mathbf{I}_{n}. with m(\gamma) is the unique Stieltjes transform solution to the following Marc̆enko-Pastur equation [ 59 , 22 ] c\gamma m_{\rm RR}^{2}(\gamma)+\left(1-c+\gamma\right)m_{\rm RR}(\gamma)-1=0. (80) By 11 , we have \displaystyle\frac{1}{n}\mathbf{y}^{\top}\left(\frac{1}{n}\mathbf{X}^{\top}\mathbf{X}+\gamma\mathbf{I}_{n}\right)^{-1}\mathbf{y}=\mathbf{e}_{1}^{\top}\mathbf{U}^{\top}\mathbf{Q}_{0}(\gamma)\mathbf{U}\left(\mathbf{I}_{2}+\boldsymbol{\Lambda}\mathbf{U}^{\top}\mathbf{Q}_{0}(\gamma)\mathbf{U}\right)^{-1}\mathbf{e}_{1} \displaystyle=\left[\begin{bmatrix}cm_{\rm RR}(\gamma)+\frac{1-c}{\gamma}&0\\ 0&\|\boldsymbol{\mu}\|^{2}(1-\gamma m_{\rm RR}(\gamma))\end{bmatrix}\begin{bmatrix}1+\|\boldsymbol{\mu}\|^{2}\left(cm_{\rm RR}(\gamma)+\frac{1-c}{\gamma}\right)&\|\boldsymbol{\mu}\|^{2}(1-\gamma m_{\rm RR}(\gamma))\\ cm_{\rm RR}(\gamma)+\frac{1-c}{\gamma}&1\end{bmatrix}^{-1}\right]_{1,1} \displaystyle=\frac{cm_{\rm RR}(\gamma)+\frac{1-c}{\gamma}}{1+\|\boldsymbol{\mu}\|^{2}(1-\gamma m_{\rm RR}(\gamma))}, so that by ( 14 ), we obtain E_{\rm RR}-\bar{E}_{\rm RR}\to 0,\quad\bar{E}_{\rm RR}=-\frac{c\gamma^{2}m^{\prime}(\gamma)+c-1+\|\boldsymbol{\mu}\|^{2}\left(\gamma^{2}m^{\prime}(\gamma)+(1-c-\gamma)(\gamma m(\gamma)-1)\right)}{\left(1+\|\boldsymbol{\mu}\|^{2}-\|\boldsymbol{\mu}\|^{2}\gamma m_{\rm RR}(\gamma)\right)^{2}}, (81) in probability as n,p\to\infty , with m_{\rm RR}(\gamma) the Stieltjes transform solution to the Marc̆enko-Pastur equation in ( 80 ), and m_{\rm RR}^{\prime}(\gamma)=-\frac{cm^{2}(\gamma)+m(\gamma)}{2c\gamma m(\gamma)+1-c+\gamma} its derivative with respect to \gamma . This concludes the proof of 2 . Appendix D Additional Numerical Results and Discussions In this section, we present additional numerical results . 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (a) p/n=1/4 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (b) p/n=1 10^{-1} 10^{0} 10^{1} 0 0.2 0.4 0.6 0.8 1 SNR \|\boldsymbol{\mu}\|^{2} (c) p/n=4 Figure 4 : Theoretical in-context memorization error for f(t)=\tanh(t) ( blue ) from 1 versus that of ridge regression ( green ) from 2 , as a function of SNR, for different dimension ratio p/n , synthetic data drawn from the Gaussian signal-plus-noise model in 2 with \mathbf{w}_{K}=\mathbf{w}_{Q}=\boldsymbol{\mu}_{\rm base}\sim\mathcal{N}(\mathbf{0},\mathbf{1}_{p}/p) , \boldsymbol{\mu}\propto\boldsymbol{\mu}_{\rm base} , and \gamma=1 . Figure 4 compare the theoretical in-context memorization errors of nonlinear Attention (as characterized in 1 ) with those of linear ridge regression (from 2 ) on synthetic Gaussian mixture data. We observe that, while ridge regression generally achieves lower memorization error than nonlinear Attention in the under-determined p>n regime, this advantage is reversed in the over-determined setting with p/n<1 . In such cases, nonlinear Attention yields lowers error, for structured inputs and Attention weights aligned to the data signal. Furthermore, compared to ridge regression, the memorization error of nonlinear Attention exhibits remarkably less sensitivity to the dimension ratio p/n , especially when the Attention weights are well aligned with the underlying signal in the input data. 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (a) p/n=1/2 , \gamma=10 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (b) p/n=1/2 , \gamma=1 10^{-1} 10^{0} 10^{1} 0 0.2 0.4 0.6 0.8 1 SNR \|\boldsymbol{\mu}\|^{2} (c) p/n=1/2 , \gamma=0.1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (d) p/n=1/4 , \gamma=10 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (e) p/n=1/4 , \gamma=1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (f) p/n=1/4 , \gamma=0.1 10^{-1} 10^{0} 10^{1} 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (g) p/n=1/8 , \gamma=10 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (h) p/n=1/8 , \gamma=1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (i) p/n=1/8 , \gamma=0.1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (j) p/n=1/16 , \gamma=10 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (k) p/n=1/16 , \gamma=1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (l) p/n=1/16 , \gamma=0.1 Figure 5 : Theoretical in-context memorization error for f(t)=\tanh(t) ( blue ) versus f(t)=\max(-5,\min(5,t)) ( purple ) and that of ridge regression ( green ) in the over-determined regime, as a function of SNR, for different dimension ratio p/n and regularization parameter \gamma , synthetic data drawn from the Gaussian signal-plus-noise model in 2 with \mathbf{w}_{K}=\mathbf{w}_{Q}=\boldsymbol{\mu}_{\rm base}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{p}/p) , \boldsymbol{\mu}\propto\boldsymbol{\mu}_{\rm base} . Figure 5 further illustrates the impact of the Attention nonlinearity, the dimension ratio p/n , and the regularization parameter \gamma on the in-context memorization errors of nonlinear/linear Attention and linear ridge regression. Reading the subfigures from left to right, we observe that the difference in memorization error between different Attention (i.e., \tanh nonlinear or truncated linear) and ridge regression vanishes either as the regularization strength \gamma decreases or as the SNR increases. Moreover, the advantage of nonlinear Attention over ridge regression—in terms of reduced memorization error—critically depends on both the dimension ratio p/n (as already confirmed in Figure 4 ) and the choice of regularization \gamma , see for example Figure 5(d) versus Figure 5(e) . Reading the subfigures from top to bottom, we further observe that in the over-determined p/n<1 regime, the memorization error of nonlinear Attention is considerably less sensitive to the changes in the dimension ratio p/n compared to ridge regression. 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (a) p/n=1/4 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (b) p/n=1 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} (c) p/n=4 Figure 6 : Theoretical in-context memorization error of \tanh ( blue ) and truncated linear (with f(t)=\max(-5,\min(5,t)) in purple ) Transformer, for key/query weights aligned with the signal direction in solid lines: \mathbf{w}_{K}=\mathbf{w}_{Q}=\boldsymbol{\mu}_{\rm base}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{p}/p) and \boldsymbol{\mu}\propto\boldsymbol{\mu}_{\rm base} ; versus the case where both weights orthogonal to the signal in dotted lines: \mathbf{w}_{K}\perp\boldsymbol{\mu}_{\rm base},\mathbf{w}_{Q}\perp\boldsymbol{\mu}_{\rm base} , \mathbf{w}_{K}\perp\mathbf{w}_{Q} and \boldsymbol{\mu}\propto\boldsymbol{\mu}_{\rm base} ; for regularization strength \gamma=1 . Figure 6 illustrates the impact of alignment between the Attention weights (the query \mathbf{w}_{Q} and key \mathbf{w}_{K} vectors in 1 ) and the input data signal \boldsymbol{\mu} . A consistent pattern emerges from Figure 6 : when the Attention weights are aligned in direction with \boldsymbol{\mu} , the resulting in-context memorization error is significantly lower compared to the case where the weights are orthogonal to \boldsymbol{\mu} . This effect is observed across both nonlinearities considered: f(t)=\tanh(t) and truncated linear function f(t)=\max(-5,\min(5,t)) , and persists across a range of SNR values and dimension ratios p/n . The performance gain from the weight alignment is particularly pronounced in the over-determined p/n<1 setting. 10^{-2} 10^{-1} 10^{0} 10^{1} 10^{2} 10^{3} 0 0.2 0.4 0.6 0.8 1 Regularization penalty \gamma Memorization error E (a) 1st layer, p/n=4 , and \boldsymbol{\mu}=\mathbf{0} 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (b) 1st layer, p/n=1/4 , and \gamma=1 10^{-2} 10^{-1} 10^{0} 10^{1} 10^{2} 10^{3} 0 0.2 0.4 0.6 0.8 1 Regularization penalty \gamma Memorization error E (c) 7th layer, p/n=4 , and \boldsymbol{\mu}=\mathbf{0} 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (d) 7th layer, p/n=1/4 , and \gamma=1 10^{-2} 10^{-1} 10^{0} 10^{1} 10^{2} 10^{3} 0 0.2 0.4 0.6 0.8 1 Regularization penalty \gamma Memorization error E (e) 12th layer, p/n=4 , and \boldsymbol{\mu}=\mathbf{0} 10^{-1} 10^{0} 10^{1} 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SNR \|\boldsymbol{\mu}\|^{2} Memorization error E (f) 12th layer, p/n=1/4 , \gamma=1 Figure 7 : Theoretical in-context memorization error of Softmax ( cyan ) and entry-wise tanh ( blue ), truncated exponential ( f(t)=\min(5,\exp(t)) in red ) Attention. Theoretical predictions under 1 in solid lines and key/query weights using pretrained Attention weights in dotted lines. Figure 7(a) , Figure 7(c) , and Figure 7(e) : theoretical predictions obtained by assuming \mathbf{w}_{K}=\mathbf{w}_{Q}=\boldsymbol{\mu}=\mathbf{0} ; Figure 7(b) , Figure 7(d) , and Figure 7(f) : theoretical predictions obtained by assuming \mathbf{w}_{K}=\mathbf{w}_{Q}=\boldsymbol{\mu}_{\rm base}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{p}/p) , \boldsymbol{\mu}\propto\boldsymbol{\mu}_{\rm base} and \gamma=1 . Figure 7 compares the in-context memorization error curves of nonlinear Attention using weights extracted from a pretrained GPT-2 model against our theoretical predictions from 1 , across varying regularization strengths, SNR levels, and activation nonlinearities. This numerical experiment serves to empirically validate the full-plus-low-rank decomposition of Attention weights posited in 1 . To extract the Attention weights \mathbf{W}_{Q} and \mathbf{W}_{K} , we use the first Attention head from the 1st, 7th, and 12th Transformer layers of a pretrained GPT-2 model (accessed via HuggingFace). Specifically, we extract the first and second m -sized column blocks from the projection matrix model.transformer.h[l].attn.c_attn.weight (of shape m\times 3m with m=768 ) as query and key weight matrices. The weights for a single head are then obtained by selecting the first m_{\rm head}=m/n_{\text{heads}}=64 columns from each matrix, consistent with the model’s n_{\text{heads}}=12 -head configuration. As shown in Figure 7 , the empirical memorization curves obtained from pretrained Attention weights closely match the theoretical trends predicted by 1 , as a function of both regularization strength \gamma and SNR. In particular, we observe that 1. in the absence of input data signal ( \boldsymbol{\mu}=\mathbf{0} ), pretrained Attention weights yield slightly lower errors than theory; and 2. in the presence of signal, pretrained Attentions perform marginally worse than theory from (manually) aligned weights. These discrepancies are generally modest in scale and consistent across both the tanh and truncated exponential nonlinearities. Additionally, we observe that Softmax Attention incurs substantially higher memorization error than entrywise exponential Attention, but only when meaningful input structure is present.