On the Spectral Flattening of Quantized Embeddings Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu Abstract Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data. By formalizing the connection between Zipfian statistics and random matrix theory, we prove that the power-law decay in the singular value spectra of embeddings is a fundamental requisite for semantic encoding. We derive theoretical bounds showing that uniform quantization introduces a noise floor that disproportionately truncates this spectral tail, which induces spectral flattening and a strictly provable increase in the stable rank of representations. Empirical validation across diverse architectures including GPT-2 and TinyLlama corroborates that this geometric degradation precipitates representational collapse. This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization. 1 Introduction The escalating computational demands of training Large Language Models (LLMs) have mandated a paradigm shift towards low-bit quantization to sustain scalability. In recent years, the community has witnessed a successful migration from standard FP32 and BF16 formats to FP8 training 33 ; 39 ; 40 , realizing substantial gains in memory efficiency and throughput without sacrificing convergence stability. Currently, the research frontier is pivoting further toward 4-bit precision. Notably, NVIDIA’s technical report on the Blackwell architecture reveals that the emerging NVFP4 format reduces memory consumption by 1.8\times , enabling the B200 GPU to accelerate General Matrix Multiplications (GeMM) by more than 5\times compared to the FP8 throughput of the Hopper-based H100 36 , underscoring the transformative potential of FP4 training for next-generation foundation models 37 . However, extending the training frontier to FP4 presents distinct theoretical hurdles that cannot be resolved by merely extrapolating previous successes. As precision descends to 4 bits, the representation imposes exponentially tighter constraints on dynamic range and resolution. This introduces a fundamental conflict with the inherently high-dynamic-range distributions of parameters, activations, and gradients characteristic of LLMs. Crucially, these distributional properties are not arbitrary artifacts; rather, they are intrinsic manifestations of the low-rank hypothesis of deep learning 1 ; 32 . While the weight matrices of LLMs are nominally full-rank, they exhibit a heavy-tailed spectral structure—governed by Zipfian laws 10 —wherein a few dominant singular values capture the bulk energy, while a long, fragile tail encodes fine-grained semantic information. We argue that the primary peril of ultra-low-bit quantization, such as FP4 , extends beyond simple scalar approximation error; it resides in the structural degradation of the spectral tail. When the quantization noise floor encroaches upon the magnitude of these delicate tail singular values, the model undergoes spectral flattening. Crucially, while a higher stable rank is often associated with beneficial isotropy in representation learning, we prove that in the context of quantization, this increase signifies a pathological loss of discriminative capacity. This phenomenon effectively truncates the subtle feature subspaces essential for complex reasoning, leading to what we formalize as representational collapse. Existing analyses often overlook this critical interplay between the discrete constraints of quantization and the continuous Zipfian laws governing natural language. In this work, we bridge this gap by establishing a rigorous theoretical framework to quantify this feature: Contribution 1: We formalize the connection between the statistical properties of natural language and the spectral structure of linear layers. Consider a linear layer \mathbf{Y}=\mathbf{XW} , where \mathbf{X} represents input token embeddings and \mathbf{W} is the weight matrix. Based on empirical studies showing that word frequencies in large-scale corpora follow a Zipfian distribution, we prove that this distribution dictates a power-law decay in the singular values of \mathbf{X} and the weight gradient \nabla_{\mathbf{W}}=\mathbf{X}^{T}\mathbf{G} , where \mathbf{G} is the gradient of the loss with respect to the output. Specifically, we derive the following bounds for the top- r singular values: \sigma_{k}(\mathbf{X})\leq C\cdot k^{-\alpha/2},\quad\sigma_{k}(\nabla_{\mathbf{W}})\leq(CM)\cdot k^{-\alpha/2}, (1) where 1\leq k\leq r , C is a constant, \sigma_{k}(\cdot) is the k -th largest singular value, \alpha is the decay exponent, and \|\mathbf{G}\|_{2}\leq M . Contribution 2: We demonstrate that block-wise symmetric uniform quantization, the mechanism widely adopted in modern low-precision formats like MXFP4 42 and NVFP4 35 , effectively alter the spectral properties of \mathbf{X} and \nabla_{\mathbf{W}} . We prove that quantization disrupts the aforementioned power-law decay, increasing the stable rank of the representations. The remainder of this paper is organized as follows. Section 2 outlines the statistical foundations. Section 3 derives the heavy-tailed spectral structure of LLMs using random matrix theory. Section 4 proves that quantization truncates this spectral tail and increases the stable rank. Section 5 provides empirical verification of these theoretical predictions across diverse model architectures. Section 6 concludes the paper and outlines directions for future work. 2 Preliminaries and Theoretical Foundations This section establishes notations and probabilistic frameworks necessary for analysis. We begin by defining the asymptotic behaviors used to describe high-dimensional limits. Subsequently, we formalize the statistical properties of language data and the geometric structure of embedding spaces. Finally, we introduce critical tools from matrix perturbation theory and concentration inequalities that allow us to bound the spectral impact of quantization noise. The supplementary definitions are in appendix Section B . 2.1 Asymptotic Notations To analyze the spectral properties of large-scale models, we consider the regime where the dimension d\to\infty . We define the following asymptotic relations to characterize the convergence behavior of sequences. Definition 2.1 (Asymptotic Equivalence) . Define two sequences A(d) and B(d) to be asymptotically equivalent, denoted as A(d)\sim B(d) , if and only if their ratio converges to 1 as d\to\infty : A(d)\sim B(d)\iff\lim_{d\to\infty}\frac{A(d)}{B(d)}=1. (2) Or equivalently, the relative error tends to 0: \lim_{d\to\infty}\left|\frac{A(d)-B(d)}{B(d)}\right|=0. (3) Definition 2.2 (Asymptotic Upper Bound) . For two sequences A(d) and B(d) that depend on d , B(d) is the asymptotic upper bound of A(d) if and only if there exists C>0,d_{0}\in\mathbb{R} such that for all d>d_{0} the following holds: |A(d)|\leq C\cdot|B(d)|, (4) which is denoted as A(d)=\mathcal{O}(B(d)) . 2.2 Statistical Structure of Language Representations To analyze the origin of the heavy-tailed spectra in LLMs, we first model the input data distribution that drives the training dynamics. We establish the spectral properties of the data embedding matrix by two widely accepted assumptions regarding token frequency and embedding geometry. Assumption 2.3 ( 10 ) . Assume the data stream is generated from a vocabulary \mathcal{V}=\{w_{1},w_{2},\dots,w_{V}\} of size V . Let p_{k} denote the occurrence probability of the k -th token. We assume \{p_{k}\}_{1\leq k\leq V} asymptotically follows a Zipfian power-law distribution: p_{k}\sim k^{-\alpha},\quad k=1,2,\dots,V, (5) where \alpha>1 is the decay parameter. Assumption 2.3 is grounded in quantitative linguistics. Empirical studies on large-scale corpora (e.g., Wikipedia 52 , Common Crawl 13 ) consistently demonstrate a power-law relationship between word frequency and rank. This implies that the embedding matrix constructed from these tokens will inherently exhibit a skewed, heavy-tailed singular value distribution. Assumption 2.4 . Let v_{k}\in\mathbb{R}^{d} denote the embedding vector corresponding to the k -th token, normalized to unit norm such that \|v_{k}\|_{2}=1 . The covariance matrix \Sigma is defined as the probability-weighted outer product: \Sigma\triangleq\sum_{k=1}^{V}p_{k}v_{k}v_{k}^{T}. (6) We posit that in the high-dimensional limit ( d\to\infty ), these embedding vectors exhibit quasi-orthogonality. Specifically, for any distinct pair of indices i\neq j : |\langle v_{i},v_{j}\rangle|=\mathcal{O}\left(\frac{1}{\sqrt{d}}\right). (7) Assumption 2.4 is grounded in the fundamental geometry of high-dimensional spaces, where independent random vectors exhibit an tendency toward orthogonality. This phenomenon is corroborated by high-dimensional probability theory (often associated with the Johnson-Lindenstrauss Lemma 24 ), which establishes that the inner product of two random unit vectors has an expectation of zero and a variance of 1/d . Consequently, in the large d regime, the cosine similarity between distinct semantic vectors scales as \mathcal{O}(d^{-1/2}) . Equation 7 is required for the proof of Lemma 3.1 . 2.3 Tools for Spectral Perturbation Analysis To quantify the spectral distortion induced by quantization noise, we leverage fundamental tools from random matrix theory 5 . Theorem 2.5 . Let \{\mu_{n}\}_{n\geq 1} be a sequence of probability measures on \mathbb{R} , and let \{m_{n}(z)\}_{n\geq 1} denote their corresponding Stieltjes transforms. If there exists a function m(z) defined on the upper complex plane \mathbb{C}^{+} , such that for all z\in\mathbb{C}^{+} the pointwise limit exists: \lim_{n\to\infty}m_{n}(z)=m(z), (8) and m(z) is the Stieltjes transform of a probability measure \mu , then \{\mu_{n}\} converges weakly to \mu : \mu_{n}\xrightarrow{w.}\mu. (9) Theorem 2.5 allows us to characterize the limiting spectral distribution of large covariance matrices by analyzing their Stieltjes transforms, a standard technique in random matrix theory. Theorem 2.6 ( 7 ) . Let A\in\mathbb{R}^{m\times n} be a matrix and \tilde{A}=A+E be its perturbed counterpart, with singular values \sigma_{1}\geq\sigma_{2}\geq\dots\geq 0 and \tilde{\sigma}_{1}\geq\tilde{\sigma}_{2}\geq\dots\geq 0 , respectively. Then, for every k , the absolute difference between the k -th singular values is bounded by the spectral norm of the perturbation matrix: |\tilde{\sigma}_{k}-\sigma_{k}|\leq\|E\|_{2}. (10) Analyzing the distribution of each specific \tilde{\sigma}_{k} directly is intractable due to complex eigenvector interactions. Theorem 2.6 allows us to decouple the individual singular value shifts from the specific noise structure, bounding them uniformly using the global spectral norm \|E\|_{2} . Theorem 2.7 ( 47 ) . Consider a finite sequence of independent, centered random matrices \{X_{k}\}\subseteq\mathbb{R}^{d_{1}\times d_{2}} . Assume that each matrix is bounded uniformly in spectral norm: \|X_{k}\|_{2}\leq R\quad\text{almost surely}, (11) for some constant R . Let \delta^{2} be the variance statistic defined as: \delta^{2}=\max\left\{\left\|\sum_{k}\mathbb{E}[X_{k}X_{k}^{T}]\right\|_{2},\left\|\sum_{k}\mathbb{E}[X_{k}^{T}X_{k}]\right\|_{2}\right\}. (12) Then, for any threshold t>0 , the probability that the spectral norm of random matrices Y\triangleq\sum_{k}X_{k} exceeds t is bounded by the exponent term: P(\|Y\|_{2}\geq t)\leq(d_{1}+d_{2})\cdot\exp\left(\frac{-t^{2}/2}{\sigma^{2}+Rt/3}\right). (13) Theorem 2.7 establishes that the spectral norm of the accumulated quantization noise does not grow linearly with the matrix dimensions. Instead, it concentrates sharply around zero, exhibiting sub-Gaussian tail behavior governed by the variance statistic \delta^{2} . This exponential decay of tail probabilities implies that large spectral deviations are extremely rare, thereby justifying our treatment of quantization noise as a bounded perturbation with high confidence. 3 Low-rank Nature of Language Model Embedding In this section, we provide a rigorous derivation of the spectral properties of the embedding. Our analysis proceeds in four steps: (1) connecting token frequency to the population covariance spectrum; (2) modeling the tail spectrum as effective white noise; (3) applying random matrix theory to characterize the sample covariance eigenvalues; and (4) deriving the singular value bounds for the input embeddings and gradients. The detailed proofs are in appendix Section C . 3.1 Population Covariance and Spectral Decomposition We first establish the link between the linguistic distribution of tokens and the spectral decay of their embedding covariance matrix. Lemma 3.1 . Under Assumption 2.3 and Assumption 2.4 , the asymptotic behavior of the k -th largest eigenvalue of \Sigma , denoted as \tau_{k} , is governed by the probability weights p_{k} : \tau_{k}\sim p_{k}\sim k^{-\alpha}, (14) where \alpha>1 . Lemma 3.1 establishes that the spectral structure of \Sigma inherits the power-law decay characteristic of the token probability distribution p_{k} . Based on this, we adopt the following explicit spectral model for the remainder of our analysis: \tau_{k}=L\cdot k^{-\alpha},\quad\forall k\in\{1,\dots,d\}, (15) where L>0 is the energy scale and \alpha>1 is the decay exponent. To apply spectral analysis tools effectively, we must distinguish between the dominant signal components and the residual noise floor. The following lemma formalizes this decomposition. Lemma 3.2 . Let the eigenvalues of the covariance matrix \Sigma\in\mathbb{R}^{d\times d} follow the power law \tau_{k}=L\cdot k^{-\alpha} with \alpha>1 . Define the background noise level \nu^{2}(d) as the mean energy of the residual eigenvalues after truncating the first r principal components: \nu^{2}(d)\triangleq\frac{1}{d-r}\sum_{j=r+1}^{d}\tau_{j}=\frac{L}{d-r}\sum_{j=r+1}^{d}j^{-\alpha}. (16) Let c be an arbitrary non-negative real constant. For any fixed head rank k\in[1,r] , the signal dominates the noise threshold: \tau_{k}>\nu^{2}(d)\cdot(1+\sqrt{c}). (17) Conversely, for any fixed tail rank k\in(r,d] , the eigenvalues fall below the threshold: \tau_{k}\leq\nu^{2}(d)\cdot(1+\sqrt{c}). (18) Furthermore, the noise level scales as \nu^{2}(d)=\mathcal{O}(d^{-1}) . 3.2 Matrix Phase Transitions A critical step in our proof is to demonstrate that the heavy-tailed residual spectrum behaves asymptotically like white noise. We achieve this by analyzing the Stieltjes transform of the spectral measure. Lemma 3.3 . Consider the set of residual eigenvalues \mathcal{T}_{A}\triangleq\{\tau_{j}=Lj^{-\alpha}\}_{j=r+1}^{d} defined in Lemma 3.2 . The Stieltjes transform of \mathcal{T}_{A} is given by: m_{Noise}(z)\triangleq\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{1}{\tau_{j}-z}. (19) Consider a reference white noise spectrum \mathcal{T}_{B} consisting of d-r identical values equal to the background noise level \nu^{2}(d) : \mathcal{T}_{B}\triangleq\underbrace{\{\nu^{2}(d),\dots,\nu^{2}(d)\}}_{d-r}, (20) whose Stieltjes transform is: m_{White}(z)\triangleq\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{1}{\nu^{2}(d)-z}=\frac{1}{\nu^{2}(d)-z}. (21) Then, as d\to\infty , the two transforms are asymptotically equivalent: \lim_{d\to\infty}\left|m_{Noise}(z)-m_{White}(z)\right|=0. (22) Combining Lemma 3.3 with the continuity property of Stieltjes transforms (Theorem 2.5 ), we derive the limiting distribution of the noise spectrum. Lemma 3.4 . Let the covariance matrix \Sigma be decomposed into a signal part and a noise part: \Sigma=\Sigma_{Signal}+\Sigma_{Noise}, (23) where \begin{split}&\text{Spec}(\Sigma_{Signal})=(\tau_{1},\dots,\tau_{r},0,\dots,0),\\ &\text{Spec}(\Sigma_{Noise})=(0,\dots,0,\tau_{r+1},\dots,\tau_{d})\end{split}. (24) As d\rightarrow\infty , the empirical spectral distribution (ESD) of \Sigma_{Noise} weakly converges to a Dirac mass centered at the noise variance \nu^{2}(d) : \text{ESD}(\Sigma_{Noise})\xrightarrow{w.}\delta_{\nu^{2}(d)}, (25) where \delta_{\nu^{2}(d)} is defined by a probability mass of 1 located at \nu^{2}(d) and 0 elsewhere. Lemma 3.4 establishes a crucial fact: despite the underlying power-law decay, the tail of the covariance matrix exhibits a spectral concentration phenomenon, effectively behaving as white noise with variance \nu^{2}(d) . We now turn our attention to the sample covariance matrix observed during training, defined as: S_{N}=\frac{1}{N}\mathbf{X}\mathbf{X}^{T}\in\mathbb{R}^{d\times d}, (26) where \mathbf{X}\in\mathbb{R}^{d\times N} is the data matrix of token embeddings, and N represents the effective batch size. Crucially, the columns of \mathbf{X} are sampled independently from the vocabulary \{v_{k}\}_{k=1}^{V} according to the token probabilities \{p_{k}\} . Consequently, S_{N} serves as the empirical estimator of the population covariance \Sigma defined in Assumption 2.4 , satisfying the unbiasedness condition: \mathbb{E}[S_{N}]=\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{N}x_{i}x_{i}^{T}\right]=\sum_{k=1}^{V}p_{k}v_{k}v_{k}^{T}=\Sigma, (27) where x_{i} is the i -th column of \mathbf{X} . In the asymptotic regime where d,N\to\infty with the ratio d/N\to c\in(0,\infty) , the behavior of the sample eigenvalues \lambda_{k} of S_{N} is characterized by the Baik-Ben Arous-Péché (BBP) phase transition theorem 6 . Theorem 3.5 (BBP Phase Transition) . Let \tau_{k} be the k -th largest eigenvalue of the population covariance \Sigma , and \lambda_{k} be the corresponding eigenvalue of the sample covariance S_{N} . Given that \text{ESD}(\Sigma_{Noise})\xrightarrow{w}\delta_{\nu^{2}(d)} , a spectral phase transition occurs at the threshold \nu^{2}(d)(1+\sqrt{c}) : 1. Super-critical regime: If the population eigenvalue satisfies the separation condition: \tau_{k}>\nu^{2}(d)(1+\sqrt{c}), (28) then the corresponding sample eigenvalue \lambda_{k} separates from the bulk spectrum and converges almost surely to a deterministic limit dependent on \tau_{k} : \lambda_{k}\xrightarrow{a.s.}\rho(\tau_{k})=\tau_{k}\left(1+\frac{c\cdot\nu^{2}(d)}{\tau_{k}-\nu^{2}(d)}\right). (29) 2. Sub-critical regime: If the population eigenvalue falls below the threshold: \tau_{k}\leq\nu^{2}(d)(1+\sqrt{c}), (30) then \lambda_{k} collapses to the right edge of the Marchenko-Pastur bulk distribution 31 : \lambda_{k}\xrightarrow{a.s.}\nu^{2}(d)(1+\sqrt{c})^{2}. (31) Specifically, \text{Spec}(\Sigma_{Signal}) falls into the super-critical regime and \text{Spec}(\Sigma_{Noise}) falls into the sub-critical regime. Theorem 3.5 defines a detectability boundary at \nu^{2}(d)(1+\sqrt{c}) . In the super-critical regime, signals exceeding this threshold separate from the noise bulk as outliers. In the sub-critical regime, signals collapse onto the edge of the Marchenko-Pastur bulk and become indistinguishable from noise. This justifies separating \text{Spec}(\Sigma_{Signal}) from \text{Spec}(\Sigma_{Noise}) . To simplify the non-linear limit \rho(\tau_{k}) , the following lemma derives an asymptotic equivalence for d\to\infty , showing that dominant sample eigenvalues reconstruct the population power-law while the tail converges to a noise floor. Lemma 3.6 . The sample eigenvalues \lambda_{k} exhibit the following asymptotic behavior: \lambda_{k}\sim\begin{cases}\tau_{k}&1\leq k\leq r\\ \nu^{2}(d)(1+\sqrt{c})^{2}&r0 is the spectral magnitude constant, and \alpha>1 is the decay rate parameter characteristic of the model’s domain. Let \tilde{A} denote the component-wise quantized counterpart of A . The quantization error matrix is defined as the difference E=\tilde{A}-A , with entries E_{ij}=e_{ij} . To facilitate a component-wise spectral analysis, we decompose the error matrix E into a sum of independent basis matrices: E=\sum_{i=1}^{m}\sum_{j=1}^{n}Z_{ij}, (37) where each Z_{ij}\in\mathbb{R}^{m\times n} is a sparse matrix containing the error e_{ij} at entry (i,j) and zeros elsewhere. In the context of modern low-precision training mechanisms, such as MXFP4 and NVFP4, the underlying quantization operators can be unified under a block-wise symmetric uniform quantization formulation. Specifically, for an input scalar a_{ij}=A_{ij} , the quantizer Q_{s}(a_{ij}) is defined as: Q_{s}(a_{ij})=s\cdot\text{round}\left(\frac{a_{ij}}{s}\right), (38) where s=a_{\max}/L represents the quantization step size. Here, a_{\max}=\max_{i,j}{|A_{ij}|} represents the dynamic range, and L is the maximum representable integer value for the given bit-width (e.g., L=7 for signed 4-bit integer representation). We adopt the standard statistical assumption regarding the distribution of the quantization error e_{ij}=Q_{s}(a_{ij})-a_{ij} , as stated in Assumption 4.2 . Assumption 4.2 . The quantization error e_{ij}=Q_{s}(a_{ij})-a_{ij} is assumed to be a centered random variable with zero mean \mathbb{E}[e_{ij}]=0 and finite variance \text{Var}(e_{ij})=B\propto s^{2} . 4.2 Bounding the Spectral Norm of Noise A primary challenge in analyzing quantization effects is the stochastic nature of the error. To derive a deterministic control over the error’s spectral norm \|E\|_{2} , we employ Theorem 2.7 . To apply this inequality to our specific error decomposition E=\sum_{i,j}Z_{ij} , we must first compute the variance statistic \delta^{2} ( Equation 12 ), which requires evaluating the spectral norms of the expected covariance sums. The following lemma simplifies this calculation by establishing the independence of cross-terms. Lemma 4.3 . For the quantization error matrix E decomposed into independent basis matrices Z_{ij} with e_{ij} following Assumption 4.2 , the expected second-moment matrix sums satisfy: \displaystyle\sum_{i,j}\mathbb{E}[Z_{ij}Z_{ij}^{T}]=\mathbb{E}[EE^{T}], (39) \displaystyle\sum_{i,j}\mathbb{E}[Z_{ij}^{T}Z_{ij}]=\mathbb{E}[E^{T}E]. Lemma 4.3 is critical as it decouples the spatial correlations within the error matrix. It essentially states that, due to independence, the variance of the sum equals the sum of the variances, even in the matrix domain. By Theorem 2.7 and Lemma 4.3 , the following theorem holds. Theorem 4.4 . Consider a series of independent basis matrices \{Z_{ij}\} of dimension m\times n where the error terms e_{ij} follow Assumption 4.2 . Let \delta^{2} be the variance statistic defined as: \delta^{2}=\max\left\{\left\|\sum_{i,j}\mathbb{E}[Z_{ij}Z_{ij}^{T}]\right\|_{2},\left\|\sum_{i,j}\mathbb{E}[Z_{ij}^{T}Z_{ij}]\right\|_{2}\right\}. (40) Then, for any threshold t>0 , the probability that the spectral norm of the sum E=\sum_{i,j}Z_{ij} exceeds t is bounded by: P(\|E\|_{2}\geq t)\leq(m+n)\cdot\exp\left(-\frac{t^{2}}{C\cdot\max(m,n)\cdot s^{2}}\right), (41) where C is a universal constant related to the sub-Gaussian norm of the noise. The derivation above yields a Gaussian-like tail bound, quantitatively demonstrating that the global spectral noise \|E\|_{2} scales approximately with \sqrt{\max(m,n)}\cdot s . While this suggests that the absolute spectral error increases for extremely wide layers, the stability of the representation is dictated by the relative impact of this noise, which we analyze next. Let us define the tail bound function F(t) as the right-hand side of Equation ( 41 ): F(t)\triangleq(m+n)\cdot\exp\left(-\frac{t^{2}}{C\cdot\max(m,n)\cdot s^{2}}\right). (42) Note that the derivative F^{\prime}(t)<0 for all t>0 , ensuring that F(t) is strictly monotonically decreasing. This monotonicity is crucial for the existence of its inverse function in the subsequent analysis. 4.3 Relative Error Analysis of Singular Values Having established a probabilistic upper bound for the global spectral error \|E\|_{2} , we now investigate how this global noise translates into perturbations of individual singular values. To bridge the gap between the spectral norm of the error matrix and the specific deviations of singular values, we define the relative error violation event \mathcal{R}(\epsilon_{k}) over the sample space \Omega . Specifically, \mathcal{R}(\epsilon_{k}) collects all outcomes \omega\in\Omega where the relative perturbation exceeds \epsilon_{k} : \mathcal{R}(\epsilon_{k})\triangleq\left\{\omega\in\Omega:\frac{|\sigma_{k}(\tilde{A}(\omega))-\sigma_{k}(A)|}{\sigma_{k}(A)}>\epsilon_{k}\right\}. (43) For brevity, we suppress the dependence on \omega in the notation and refer to the event as \mathcal{R}(\epsilon_{k})=\left\{\frac{|\sigma_{k}(\tilde{A})-\sigma_{k}(A)|}{\sigma_{k}(A)}>\epsilon_{k}\right\} . Theorem 4.5 . For a fixed confidence level 1-\theta where \theta\in(0,1) , the maximum relative error bound \epsilon_{k}(\theta) induced by quantization is strictly inversely proportional to the magnitude of the singular value \sigma_{k}(A) : \epsilon_{k}(\theta)\propto\frac{1}{\sigma_{k}(A)}. (44) Furthermore, for any two singular values with magnitudes \sigma_{i}(A)>\sigma_{j}(A)>0 , under the same confidence level 1-\theta , it strictly holds that: \epsilon_{i}(\theta)<\epsilon_{j}(\theta). (45) Theorem 4.5 establishes the theoretical core of our analysis, revealing a fundamental spectral asymmetry: while quantization adds a roughly uniform noise floor to the entire spectrum, the relative damage is highly non-uniform. Dominant singular values are sufficiently large to suppress this noise, resulting in negligible relative perturbation. Conversely, tail singular values operate in a low signal-to-noise regime, making them highly susceptible to distortion. 4.4 Failure Probability under Power-Law Decay In this subsection, we concretize the relative error analysis by calculating the explicit failure probability. Define \theta_{k} as the failure probability for the k -th singular value. The following theorem gives an upper bound of \theta_{k} Theorem 4.6 . Consider a stochastic matrix A\in\mathbb{R}^{m\times n} satisfying Assumption 4.1 and Assumption 4.2 . Let R_{k}\triangleq\frac{|\sigma_{k}(\tilde{A})-\sigma_{k}(A)|}{\sigma_{k}(A)} be the relative error. For any fixed relative error tolerance \eta>0 , define \theta_{k} to be the probability that R_{k}>\eta . Then \theta_{k} is bounded by: \theta_{k}\lesssim(m+n)\cdot\exp\left(-\frac{\eta^{2}\mu^{2}}{CDs^{2}}\cdot\xi_{k}\right), (46) where \xi_{k}=\begin{cases}k^{-\alpha},&1\leq k\leq r\\ r^{-\alpha},&rS_{r}(A). (52) 5 Experiments 5.1 Model Architecture and Dataset To verify the analysis, we employ a decoder-only Transformer architecture following the configuration of GPT-2-124M 41 . Specifically, the model comprises L=12 Transformer layers with a hidden dimension of d_{\text{model}}=768 and H=12 attention heads. The context window size is set to 1024 tokens. We train all models on the OpenWebText dataset 21 , a large-scale collection of web text widely utilized for language model pre-training. All experiments are implemented using the PyTorch framework and executed on a computational node equipped with 8 GPUs. We utilize a global batch size of 1280, which is achieved using a local batch size of 4 per GPU combined with 40 steps of gradient accumulation. The models are trained with AdamW optimizer 30 with a weight decay coefficient of \beta=0.1 . The learning rate is initialized at 1.5\times 10^{-4} and features a linear warm-up over the first 20 steps. Furthermore, to mitigate the risk of gradient explosion, we apply gradient clipping with a maximum norm of 1.0 . All FP4 training in this work adopts W4A4G4 quantization, where weights, activations, and gradients are represented in the E2M1 NVFP4 format. Due to Nvidia’s closed-source FP4 training software stack, native hardware-supported FP4 training is not currently accessible; consequently, our experiments with NVFP4 are conducted through simulation in BF16 . More experiments are in appendix Section E . 5.2 Empirical Verification of Spectral Flattening We empirically validate the predicted spectral degradation through a layer-wise analysis of the Multilayer Perceptron (MLP) blocks in GPT-2-124M. Specifically, we evaluate the singular value spectra of the BF16 baseline against its NVFP4 quantized counterpart. The left panel of Figure 1 elucidates the mechanism via the component-wise distortion ratio \epsilon_{k} . We observe disproportionate attenuation in dominant singular values (top 30%, ratio \approx 0.02 – 0.05 ), which directly inflates the stable rank by suppressing the spectral norm. Conversely, the tail components exhibit relative preservation ( \epsilon_{k}\to 0.18 ). This asymmetry corroborates Theorem 4.5 , confirming that the quantization noise floor effectively obstructs the decay of head spectra. The right panel of Figure 1 confirms that pre-quantization embeddings exhibit a log-linear spectral decay, validating the power-law result in Theorem 3.7 . Following NVFP4 quantization, we detect a deterministic increase in stable rank ( 2.52\to 2.71 ), consistent with the spectral flattening predicted by Theorem 4.7 . Figure 1: Spectral analysis of MLP embedding matrices. (Left) The ratio of quantized to original singular values across indices. (Right) Logarithmic distribution of singular values with BF16 and NVFP4 quantization. 5.3 Spectral Evolution and Representational Collapse Extending the analysis of Theorem 4.7 , we investigate the temporal evolution of spectral properties. Figure 2 contrasts the singular value spectra of BF16 and NVFP4 embedding matrices in MLP layers across representative training steps. The empirical evidence reveals a bifurcation in spectral dynamics, characterizing two distinct phases of low-precision training. In the initial iterations (Steps 0–10), observations corroborate our theoretical predictions. As shown in the top row of Figure 2 , the quantized embeddings ( NVFP4 ) display an elevated tail spectrum relative to BF16 baseline. Consequently, the stable rank of the quantized representations increases. This confirms that quantization introduces a noise floor that obscures fine-grained spectral structures, resulting in a generalized flattening of the distribution. However, as training progresses (Steps 50–100), a pathological divergence emerges. The bottom row of Figure 2 demonstrates that the stable rank of quantized embedding declines rapidly, deviating from the monotonic increase observed in the BF16 baseline. By Step 100, the BF16 model attains a high stable rank (SR=1.08), indicative of effective dimensionality expansion and feature encoding. Conversely, the NVFP4 embedding degenerates into a low-rank state (SR=1.00). This crossover suggests that cumulative gradient errors shift the regime from static flattening to dynamic representational collapse. This process compresses the model capacity into a few dominant singular values, effectively eliminating the heavy-tailed spectral structure essential for semantic encoding. Figure 2: Spectral evolution of training. Top: Initial noise inflates Stable Rank (SR). Bottom: Later steps show representational collapse. 6 Conclusion and Future Work This work unifies Zipfian statistics and Random Matrix Theory to demonstrate that uniform quantization disrupts the heavy-tailed spectral structure of LLMs, causing spectral flattening and a strict increase in stable rank. Our empirical analysis shows that this distortion precipitates a phase transition into representational collapse. This finding challenges unstructured perturbation models by characterizing quantization as a geometric degradation of the optimization landscape rather than simple additive noise. Future research should focus on designing non-uniform quantization schemes grounded in power-law statistics to preserve tail fidelity, alongside employing spectral regularization to prevent rank collapse. Furthermore, extending this spectral analysis to non-linearities such as softmax and attention is essential to fully characterize quantization effects. These directions collectively establish the theoretical foundation for robust low-precision training. Appendix A More Related Works The challenge of ultra-low-bit quantization in Large Language Models (LLMs) is a problem of preserving spectral structure under extreme compression. In this section, we review the theoretical foundations of this challenge, tracing the trajectory from the statistical physics of natural language to the spectral properties of neural networks, and subsequently analyzing the evolution of quantization algorithms designed to mitigate spectral degradation. A.1 Statistical and Spectral Foundations The spectral attributes of LLMs are not mere artifacts of random initialization; rather, they are intrinsically shaped by the statistical distribution of training data. As established by [ 10 ] , word frequency in natural language adheres to a strict power-law distribution f\propto r^{-\alpha} . This distribution has been extensively validated across web-scale corpora, including Common Crawl [ 13 ] and Wikipedia [ 52 ] . Recent theoretical analysis by [ 15 ] suggests that observed Neural Scaling Laws are deductive consequences of these Zipfian statistics. In the context of neural representations, this heavy-tailed data distribution induces geometric anisotropy in the embedding space. [ 17 ] demonstrated that contextual representations occupy a narrow cone topology, while [ 18 ] formalized this phenomenon as representation degeneration. Although early frameworks postulated that this generative process governs geometric structure [ 3 ] , recent analysis by [ 12 ] empirically confirms that high-frequency tokens exhibit distinct variability patterns compared to rare tokens, establishing that token frequency dictates the local curvature of the embedding manifold. This phenomenon is often exacerbated by the frequency bias inherent in the softmax layer [ 49 ] . Chronologically, strategies to mitigate this collapse have evolved from heuristic subtraction [ 34 ] to density-aware calibration, such as normalizing flows [ 27 ] and standard whitening [ 44 ] . Most recently, [ 55 ] introduced Zipfian whitening, advancing this methodology by explicitly leveraging word frequency statistics to flatten the spectrum. While [ 2 ] leveraged the resulting low intrinsic dimensionality to show models can be fine-tuned in low-rank subspaces, [ 23 ] argues that the fragile long tail—often dismissed as noise—encodes critical semantic distinctions via a tail-first growth mechanism. This emphasis on the heavy-tailed spectrum aligns with rigorous characterizations from Random Matrix Theory (RMT). [ 32 ] identified heavy-tailed self-regularization (HT-SR) as a hallmark of high-performance deep networks. Parallel research on the Hessian spectrum [ 43 , 20 ] confirms that the loss landscape consists of a massive bulk and a few extreme outliers, often associated with flat minima [ 22 ] . Early work on Optimal Brain Damage (OBD) by [ 26 ] utilized these properties for pruning. Subsequently, [ 16 ] leveraged this in Hessian-Aware Quantization, demonstrating that layers with larger spectral norms are disproportionately sensitive to noise. More recently, the Muon optimizer [ 29 ] explicitly manages these spectral properties during training, reinforcing the necessity of maintaining specific spectral profiles for scalable learning. Table 1 summarized the literature on statistical and spectral foundations. Table 1: Summary of Literature on Statistical and Spectral Foundations. Category Key Contributions & Findings Citations Zipfian Statistics Establishes power-law distribution ( f\propto r^{-\alpha} ) in natural language and validates it on web-scale corpora; links Zipf’s law to Neural Scaling Laws. [ 10 , 13 , 52 , 15 ] Anisotropy & Geometry Identifies representation degeneration and the narrow cone effect; links token frequency to local manifold curvature and softmax bias. [ 17 , 18 , 19 , 3 , 12 , 49 ] Isotropy Restoration Evolution from heuristic subtraction (All-but-the-Top) to Normalizing Flows. [ 34 , 27 , 44 , 55 ] Dimensionality & Tails Debates on intrinsic dimensionality vs. the importance of the fragile long tail for semantic distinctions and capacity growth. [ 2 , 23 ] Spectral Theory (RMT) Characterizes heavy-tailed self-regularization (HT-SR) in trained weights; analyzes empirical spectral densities. [ 32 ] Hessian & Optimization Analyzes Hessian spectrum (bulk vs. outliers, flat minima); Optimal Brain Damage (OBD); Hessian-Aware Quantization (HAWQ); Spectral control via Muon optimizer. [ 43 , 20 , 22 , 26 , 16 , 29 ] A.2 Quantization Pathology and Algorithmic Evolution These theoretical properties manifest practically as the outlier phenomenon [ 8 ] , which constitutes the primary barrier to low-bit quantization. [ 14 ] observed that when parameter counts exceed 6.7B, systematic massive activations emerge [ 45 ] . According to classic quantization theory [ 51 ] , these outliers compel uniform quantizers to adopt excessive step sizes, causing the fine-grained information in the distribution bulk to collapse. [ 50 ] further identified LayerNorm as a key factor amplifying these variances. To address the spectral challenges posed by outliers, quantization strategies have evolved from simple decomposition to complex geometric transformations. Early mitigation strategies, such as LLM.int8() [ 14 ] , employed mixed-precision decomposition to isolate outliers. Subsequent approaches utilizing mathematical equivalences, such as SmoothQuant [ 53 ] and AWQ [ 28 ] , sought to migrate quantization difficulty from activations to weights. Recent research has shifted towards geometric transformations to fundamentally alter the spectral distribution. Methods like QuIP [ 11 ] and QuIP# [ 48 ] introduce Randomized Hadamard Transforms to spectrally whiten the distribution, approximating a Gaussian distribution by leveraging the Johnson-Lindenstrauss lemma [ 25 ] . [ 4 ] extended this approach to end-to-end inference in QuaRot. Most recently, [ 54 ] proposed limiting signal propagation spectral norms via manifold constraints. Concurrently, hardware-aware formats have evolved to support these algorithmic advances. While initial efforts explored FP8 formats [ 33 ] , newer standards like NVFP4 [ 38 ] and MXFP4 [ 42 ] adopt block-wise scaling to perform local spectral normalization, thereby preserving representational capacity in low-precision regimes. Despite these engineering advances, the specific theoretical mechanism linking the breakdown of reasoning capabilities in quantized models to the fragility of the Zipfian spectral tail remains an open question. Table 2 summarized the literature on quantization pathology and algorithmic evolution. Table 2: Summary of Literature on Quantization Pathology and Algorithmic Evolution. Category Key Contributions & Mechanisms Citations Outlier Pathology Identification of the outlier phenomenon and massive activations in LLMs (>6.7B); analyzes the role of LayerNorm in variance amplification. [ 8 , 14 , 45 , 50 ] Classic Theory Foundational quantization noise theories assuming uniform distribution (often violated by LLM outliers). [ 51 ] Decomp. & Migration Decomposition : Mixed-precision (INT8/FP16) isolation. Migration : Mathematically shifting quantization difficulty from activations to weights (SmoothQuant, AWQ). [ 14 , 53 , 28 ] Rotation & Whitening Uses Randomized Hadamard Transforms (RHT) and JL Lemma to whiten spectral distribution and delocalize outliers (QuIP, QuIP#, QuaRot). [ 11 , 48 , 25 , 4 ] Manifold Constraints Explicitly limits signal propagation spectral norms via manifold projections (mHC). [ 54 ] Hardware Formats Evolution from FP8 to block-wise scaling formats ( NVFP4 , MXFP4 ) designed for local spectral normalization. [ 33 , 38 , 42 ] Appendix B Supplementary Definitions To ensure the self-contained nature of this work, we provide formal definitions of the mathematical concepts and theoretical frameworks utilized in our spectral analysis. Definition B.1 (Stable Rank) . For a stochastic matrix A\in\mathbb{R}^{m\times n} , the stable rank, denoted by S_{r}(A) , is defined as the ratio of its squared Frobenius norm to its squared spectral norm: S_{r}(A)\triangleq\frac{\|A\|_{F}^{2}}{\|A\|_{2}^{2}}, (53) where \|A\|_{F}=\sqrt{\sum_{i,j}A_{ij}^{2}} is the Frobenius norm and \|A\|_{2}=\sigma_{\max}(A) is the spectral norm. Definition B.2 (Stieltjes Transform) . The Stieltjes transform of a probability measure \mu is defined for z\in\mathbb{C}\setminus\text{supp}(\mu) as: m_{\mu}(z)=\int_{\mathbb{R}}\frac{1}{\lambda-z}d\mu(\lambda). (54) For a matrix with n discrete eigenvalues \{\lambda_{i}\}_{i=1}^{n} , the transform takes the empirical form: m_{n}(z)=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{\lambda_{i}-z}. (55) Stieltjes transform is used to prove the spectral convergence of noise models in Lemma 3.3 and Lemma 3.4 . Definition B.3 (Empirical Spectral Distribution, ESD) . For a symmetric matrix M\in\mathbb{R}^{d\times d} with eigenvalues \lambda_{1},\dots,\lambda_{d} , the ESD is defined as: \mu_{M}=\frac{1}{d}\sum_{i=1}^{d}\delta_{\lambda_{i}} (56) where \delta is the Dirac delta measure. The ESD converges to a deterministic measure as d\to\infty . The ESD is employed to characterize the global density of eigenvalues, serving as the basis for proving that heavy-tailed quantization noise converges to a deterministic noise bulk in Lemma 3.4 and Theorem 3.5 . Appendix C Math Proof of Low-rank Nature of Language Model Embedding We write Equation 6 in matrix form. Let U=[v_{1},v_{2},\dots,v_{V}]\in\mathbb{R}^{d\times V} be the matrix of eigenvectors, \Lambda_{p}=\text{diag}(p_{1},\dots,p_{V}) be the probability diagonal matrix. Then, \Sigma=U\Lambda_{p}U^{T}. (57) By Assumption 2.3 , the probability weights satisfy p_{1}>p_{2}>\dots>p_{V}>0 , and p_{k}\sim k^{-\alpha} , which means that p_{k} has a large spectral gap. According to the singular values decomposition (SVD) duality property in linear algebra, \Sigma ’s non-zero eigenvalues are exactly the same as those of the matrix M=\Lambda_{p}^{1/2}U^{T}U\Lambda_{p}^{1/2} . We analyze the matrix M\in\mathbb{R}^{V\times V} . Let G=U^{T}U be the Gram matrix [ 7 ] of the eigenvectors. Due to vector normalization, (G)_{ii}=\|v_{i}\|^{2}=1. (58) By Assumption 2.4 , (G)_{ij}=\langle v_{i},v_{j}\rangle ( i\neq j ), and |(G)_{ij}|=\mathcal{O}(1/\sqrt{d}) . G can be written as: G=I_{V}+E, (59) where I_{V} is the identity matrix, E is the full interference matrix with diagonal elements equal to 0 and off-diagonal absolute values equal to \mathcal{O}(1/\sqrt{d}) , i.e., E=\begin{cases}|E_{ij}|=0,\quad&i=j\\ |E_{ij}|=\mathcal{O}\left(\frac{1}{\sqrt{d}}\right),\quad&i\neq j\end{cases}. (60) Therefore, the target matrix M becomes: M=\Lambda_{p}^{1/2}(I_{V}+E)\Lambda_{p}^{1/2}=\Lambda_{p}+\Lambda_{p}^{1/2}E\Lambda_{p}^{1/2}. (61) Let: A=\Lambda_{p}=\text{diag}(p_{1},\dots,p_{V}),\quad\Delta=\Lambda_{p}^{1/2}E\Lambda_{p}^{1/2}. (62) Since A is a diagonal matrix, the eigenvalues of A are p_{k} . By Theorem 2.6 , the difference between \tau_{k} and p_{k} is bounded by the 2-norm of perturbation matrix \Delta : |\tau_{k}-p_{k}|\leq\|\Delta\|_{2}. (63) We need to compute the upper bound of \|\Delta\|_{2} , where \Delta=\begin{cases}(\Delta)_{ij}=\sqrt{p_{i}p_{j}}\cdot\langle v_{i},v_{j}\rangle,\quad&i\neq j\\ (\Delta)_{ij}=0,\quad&i=j\end{cases}. (64) Using the matrix norm inequality: \|\Delta\|_{2}\leq\|\Delta\|_{F}=\sqrt{\sum_{i\neq j}p_{i}p_{j}\langle v_{i},v_{j}\rangle^{2}}, (65) where ||\cdot||_{F} is the Frobenius norm. By 2.4 , |\langle v_{i},v_{j}\rangle|=\mathcal{O}(1/\sqrt{d}) . Then, \|\Delta\|_{2}\leq\mathcal{O}\left(\frac{1}{\sqrt{d}}\right)\cdot\sqrt{\sum_{i\neq j}p_{i}p_{j}}<\mathcal{O}\left(\frac{1}{\sqrt{d}}\right)\cdot\sum_{i=1}^{V}p_{i}. (66) Due to probability normalization \sum p_{i}=1 : \|\Delta\|_{2}\leq\mathcal{O}\left(\frac{1}{\sqrt{d}}\right)\cdot 1=\mathcal{O}\left(\frac{1}{\sqrt{d}}\right). (67) By Equation 63 and Equation 67 , |\tau_{k}-p_{k}|\leq\mathcal{O}\left(\frac{1}{\sqrt{d}}\right). (68) Note that d\rightarrow\infty leads to \mathcal{O}(1/\sqrt{d})\rightarrow 0 . Thus, for a fixed p_{k} : \frac{|\tau_{k}-p_{k}|}{p_{k}}\to 0\quad(d\to\infty). (69) By Definition 2.1 , the asymptotic equivalence relation holds: \tau_{k}\sim p_{k}. (70) By Assumption 2.3 , the following relation holds: \tau_{k}\sim p_{k}\sim k^{-\alpha}. (71) ∎ By Cauchy’s integral test [ 9 ] , for a monotonically decreasing function f(x)=x^{-\alpha} , there is an inequality holds: \sum_{j=r+1}^{d}j^{-\alpha}<\int_{r}^{d}x^{-\alpha}dx. (72) Calculate the definite integral: \int_{r}^{d}x^{-\alpha}dx=\left[\frac{x^{1-\alpha}}{1-\alpha}\right]_{r}^{d}=\frac{1}{\alpha-1}\left(r^{1-\alpha}-d^{1-\alpha}\right). (73) Since d>r and \alpha>1 , d^{1-\alpha}>0 . Discard the subtraction term -d^{1-\alpha} for further scale: \int_{r}^{d}x^{-\alpha}dx<\frac{r^{1-\alpha}}{\alpha-1}. (74) Substituting Equation 74 into Equation 16 leads to: \nu^{2}(d)<\frac{L}{d-r}\cdot\frac{r^{1-\alpha}}{\alpha-1}. (75) For a fixed head rank k\in[1,r] , define a signal-to-noise ratio function R(d) as: R(d)\triangleq\frac{\tau_{k}}{\nu^{2}(d)}. (76) By Equation 75 : \begin{split}R(d)=\frac{L\cdot k^{-\alpha}}{\nu^{2}(d)}>\frac{L\cdot k^{-\alpha}}{\left(\frac{L}{d-r}\cdot\frac{r^{1-\alpha}}{\alpha-1}\right)}=\frac{k^{-\alpha}(\alpha-1)}{r^{1-\alpha}}\cdot(d-r).\end{split} (77) Since r and \frac{k^{-\alpha}(\alpha-1)}{r^{1-\alpha}} are positive real number, the following limit holds: \lim_{d\to\infty}\frac{k^{-\alpha}(\alpha-1)}{r^{1-\alpha}}\cdot(d-r)=+\infty. (78) By comparison test, Equation 77 leads to \lim_{d\to\infty}R(d)=+\infty. (79) Due to \lim_{d\to\infty}R(d)=\lim_{d\to\infty}\frac{\tau_{k}}{\sigma^{2}(d)}=+\infty , for any given constant 1+\sqrt{c} , there is a dimension d_{1} such that for all d>d_{1} , the following relation holds: \frac{\tau_{k}}{\nu^{2}(d)}>1+\sqrt{c}. (80) Namely: \tau_{k}>\nu^{2}(d)\cdot(1+\sqrt{c}). (81) On the other hand, \lim_{d\to\infty}\tau_{k}=0 when k\in(r,d] . For any given constant 1+\sqrt{c} , there is a dimension d_{2} , such that for all d>d_{2} , the following relation holds: \tau_{k}\leq\nu^{2}(d)\cdot(1+\sqrt{c}). (82) Take d=\max\{d_{1},d_{2}\} , then Equation 81 and Equation 82 hold simultaneously. Furthermore, recall the definition of \nu^{2}(d) : \nu^{2}(d)=\frac{L}{d-r}\sum_{j=r+1}^{d}j^{-\alpha}. (83) When d\rightarrow\infty , \sum_{j=r+1}^{d}j^{-\alpha} can be regarded as the tail of Riemann Zeta function \zeta(\alpha) [ 46 ] : \zeta(\alpha)=\sum_{j=1}^{\infty}j^{-\alpha}. (84) Since \zeta(\alpha) converges, \sum_{j=r+1}^{d}j^{-\alpha} converges. As L,r are constants, there are constants C>0,C,d_{0}\in\mathbb{R} such that for all d>d_{0} the following condition is satisfied |\nu^{2}(d)|=\left|\frac{L}{d-r}\sum_{j=r+1}^{d}j^{-\alpha}\right|\leq C\cdot\left|\frac{1}{d}\right|. (85) By Definition 2.2 , that is \nu^{2}(d)=\mathcal{O}(d^{-1}) . ∎ For any a and z ( a\neq z ), the following identity holds: \frac{1}{a-z}=-\frac{1}{z}-\frac{a}{z^{2}}+\frac{a^{2}}{z^{2}(a-z)}. (86) Substituting a=\tau_{j} into Equation 86 and expanding m_{Noise}(z) gives: \begin{aligned} m_{Noise}(z)&=\frac{1}{d-r}\sum_{j=r+1}^{d}\left(-\frac{1}{z}-\frac{\tau_{j}}{z^{2}}+\frac{\tau_{j}^{2}}{z^{2}(\tau_{j}-z)}\right)\\ &=-\frac{1}{z}-\frac{1}{z^{2}}\left(\frac{1}{d-r}\sum_{j=r+1}^{d}\tau_{j}\right)+\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{\tau_{j}^{2}}{z^{2}(\tau_{j}-z)}\\ &=-\frac{1}{z}-\frac{\nu^{2}(d)}{z^{2}}+R_{Noise}(z)\end{aligned}, (87) where R_{Noise}(z)\triangleq\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{\tau_{j}^{2}}{z^{2}(\tau_{j}-z)}. (88) Substituting a=\nu^{2}(d) into Equation 86 and expanding m_{White}(z) gives: \begin{aligned} m_{White}(z)&=-\frac{1}{z}-\frac{\nu^{2}(d)}{z^{2}}+\frac{(\nu^{2}(d))^{2}}{z^{2}(\nu^{2}(d)-z)}\end{aligned}. (89) Denote R_{White}(z)\triangleq\frac{(\nu^{2}(d))^{2}}{z^{2}(\nu^{2}(d)-z)}. (90) Now examine the difference \Delta(z)\triangleq|m_{Noise}(z)-m_{White}(z)| . The first two terms of Equation 87 and Equation 89 are the same, so by the triangle inequality: \Delta(z)=|R_{Noise}(z)-R_{White}(z)|\leq|R_{Noise}(z)|+|R_{White}(z)|. (91) For a fixed z , since \tau_{j}\in\{\tau_{j}=Lj^{-\alpha}\}_{j=r+1}^{d} , \frac{\tau_{j}^{2}}{z^{2}(\tau_{j}-z)}=\mathcal{O}(\tau_{j}). (92) Therefore, |R_{Noise}|=\mathcal{O}\left(\frac{1}{d-r}\sum_{j=r+1}^{d}\tau_{j}\right)=\mathcal{O}\left(\frac{L}{d-r}\sum_{j=r+1}^{d}j^{-\alpha}\right). (93) When d\rightarrow\infty in Equation 93 , \sum_{j=r+1}^{d}j^{-\alpha} can be regarded as the tail of Riemann Zeta function \zeta(\alpha) . Since \zeta(\alpha) converges, \sum_{j=r+1}^{d}j^{-\alpha} converges. Thus, \frac{L}{d-r}\sum_{j=r+1}^{d}j^{-\alpha}=\mathcal{O}(d^{-1}) (94) Equation 87 and Equation 94 give: |R_{Noise}|=\mathcal{O}(\mathcal{O}(d^{-1}))=\mathcal{O}(d^{-1}). (95) Similarly, |R_{White}|=\mathcal{O}(\nu^{2}(d)). (96) Lemma 3.2 shows that \nu^{2}(d)=\mathcal{O}(d^{-1}) , so |R_{White}|=\mathcal{O}(\mathcal{O}(d^{-1}))=\mathcal{O}(d^{-1}). (97) Substituting Equation 95 and Equation 97 into Equation 91 gives: \lim_{d\to\infty}\Delta(z)=\lim_{d\to\infty}[\mathcal{O}(d^{-1})+\mathcal{O}(d^{-1})]=0 (98) By the Squeeze theorem, \lim_{d\to\infty}|m_{Noise}(z)-m_{White}(z)|=0 (99) ∎ Examine the normalization matrix \tilde{\Sigma}=\frac{1}{\nu^{2}(d)}\Sigma : \tilde{\Sigma}=\frac{1}{\nu^{2}(d)}\Sigma_{Signal}+\frac{1}{\nu^{2}(d)}\Sigma_{Noise}. (100) Define the set of eigenvalues belonging to \frac{1}{\nu^{2}(d)}\Sigma_{Noise} \tilde{\mathcal{T}}_{A}=\{\tilde{\tau}_{j}=\frac{\tau_{j}}{\nu^{2}(d)}\}_{j=r+1}^{d}. (101) The Stieltjes transform of \tilde{\mathcal{T}}_{A} is: \tilde{m}_{Noise}(z)\triangleq\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{1}{\tilde{\tau}_{j}-z}. (102) Substituting \tilde{\tau}_{j} ’s definition and multiplying both the numerator and denominator by \nu^{2}(d) give: \displaystyle\tilde{m}_{Noise}(z) \displaystyle=\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{1}{\frac{\tau_{j}}{\nu^{2}(d)}-z} (103) \displaystyle=\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{\nu^{2}(d)}{\tau_{j}-\nu^{2}(d)z} (104) \displaystyle=\nu^{2}(d)\cdot\underbrace{\left(\frac{1}{d-r}\sum_{j=r+1}^{d}\frac{1}{\tau_{j}-\nu^{2}(d)z}\right)}_{m_{Noise}(\nu^{2}(d)z)} (105) The remaining summation term is m_{Noise}(z) , with the independent variable becoming \nu^{2}(d)z . Therefore, the following relation holds: \tilde{m}_{Noise}(z)=\nu^{2}(d)\cdot m_{Noise}(\nu^{2}(d)z) (106) By Lemma 3.3 \lim_{d\to\infty}|m_{Noise}(z)-m_{White}(z)|=0 , so the relation holds: \displaystyle\lim_{d\to\infty}\tilde{m}_{Noise}(z) \displaystyle=\lim_{d\to\infty}\nu^{2}(d)\cdot\left(\frac{1}{\nu^{2}(d)-(\nu^{2}(d)z)}\right) (107) \displaystyle=\lim_{d\to\infty}\nu^{2}(d)\cdot\frac{1}{\nu^{2}(d)(1-z)} (108) \displaystyle=\lim_{d\to\infty}\frac{\nu^{2}(d)}{\nu^{2}(d)}\cdot\frac{1}{1-z} (109) \displaystyle=\lim_{d\to\infty}\frac{1}{1-z} (110) \displaystyle=\frac{1}{1-z}. (111) Theorem 2.5 states that if a sequence of Stieltjes transforms m_{n}(z) converges to a limit m(z) , and m(z) corresponds to a unique probability measure \mu , then the underlying distributions \mu_{n} must weakly converge to \mu . Equation 107 gives the asymptotic limit of the noise matrix’s transform: \lim_{d\to\infty}\tilde{m}_{Noise}(z)=\frac{1}{1-z}. (112) Note that the limit function (1-z)^{-1} is the unique Stieltjes transform of the Dirac mass \delta_{1} . In fact, m_{\delta_{1}}(z)=\int_{\mathbb{R}}\frac{1}{t-z}d\delta_{1}(t)=\frac{1}{1-z}. (113) Consequently, identifying the ESD as the sequence of measures \{\mu_{n}\} , we invoke Theorem 2.5 to translate the algebraic limit directly into the geometric collapse of the spectrum: \text{ESD}\left(\frac{1}{\nu^{2}(d)}\Sigma_{Noise}\right)\xrightarrow{w}\delta_{1}. (114) By the linear property of random variables, the ESD of \Sigma_{Noise} weakly converges to the Dirac measure concentrated at \nu^{2}(d) , i.e., \text{ESD}(\Sigma_{Noise})\xrightarrow{w.}\delta_{\nu^{2}(d)},\quad d\rightarrow\infty (115) ∎ By Lemma 3.4 , \text{ESD}(\Sigma_{Noise}) weakly converges to the Dirac measure \delta_{\nu^{2}(d)} : \text{ESD}(\Sigma_{Noise})\xrightarrow{w.}\delta_{\nu^{2}(d)},\quad d\rightarrow\infty (116) Obviously, \delta_{\nu^{2}(d)} is a deterministic probability measure. Since \nu^{2}(d)=\mathcal{O}(d^{-1}) , the support set \{\nu^{2}(d)\} is a compact set. Therefore, take \delta_{\nu^{2}(d)} as \mu . By Lemma 3.2 , for d\rightarrow\infty,k=1,...,r , the relation: \tau_{k}>\nu^{2}(d)\cdot(1+\sqrt{c}) (117) holds, which means that \text{Spec}(\Sigma_{signal})=(\tau_{1},\dots,\tau_{r},0,\dots,0) falls into the super-critical regime. For d\rightarrow\infty,k=r+1,...,d , the relation: \tau_{k}\leq\nu^{2}(d)\cdot(1+\sqrt{c}) (118) holds, which means that \text{Spec}(\Sigma_{tail})=(0,\dots,0,\tau_{r+1},\dots,\tau_{d}) falls into the sub-critical regime. ∎ By Theorem 3.5 , since \text{Spec}(\Sigma_{Signal}) falls into the super-critical regime, the relation holds: \lambda_{k}\xrightarrow{a.s.}\rho(\tau_{k})=\tau_{k}\left(1+\frac{c\cdot\nu^{2}(d)}{\tau_{k}-\nu^{2}(d)}\right). (119) Notice that: \frac{1}{\tau_{k}-\nu^{2}(d)}=\frac{1}{\tau_{k}\left(1-\frac{\nu^{2}(d)}{\tau_{k}}\right)}=\frac{1}{\tau_{k}}\cdot\left(1-\frac{\nu^{2}(d)}{\tau_{k}}\right)^{-1} (120) Applying the power series expansion formula (1-x)^{-1}=1+x+x^{2}+\mathcal{O}(x^{3}) , where x=\frac{\nu^{2}(d)}{\tau_{k}} , gives: \left(1-\frac{\nu^{2}(d)}{\tau_{k}}\right)^{-1}=1+\frac{\nu^{2}(d)}{\tau_{k}}+\left(\frac{\nu^{2}(d)}{\tau_{k}}\right)^{2}+\mathcal{O}(\tau_{k}^{-3}). (121) Substituting Equation 121 into Equation 119 gives: \displaystyle\rho(\tau_{k}) \displaystyle=\tau_{k}\left[1+c\nu^{2}(d)\cdot\left(\frac{1}{\tau_{k}}\left(1+\frac{\nu^{2}(d)}{\tau_{k}}+\frac{(\nu^{2}(d))^{2}}{\tau_{k}^{2}}+\dots\right)\right)\right] (122) \displaystyle=\tau_{k}\left[1+\frac{c\nu^{2}(d)}{\tau_{k}}+\frac{c(\nu^{2}(d))^{2}}{\tau_{k}^{2}}+\frac{c(\nu^{2}(d))^{4}}{\tau_{k}^{3}}+\mathcal{O}(\tau_{k}^{-4})\right] \displaystyle=\tau_{k}+c\nu^{2}(d)+\frac{c(\nu^{2}(d))^{2}}{\tau_{k}}+\mathcal{O}(\tau_{k}^{-2}) By the definition of almost surely convergence, \lim_{\tau_{k}\to\infty}\frac{\lambda_{k}}{\tau_{k}}=\lim_{\tau_{k}\to\infty}\left(\frac{\tau_{k}+c\nu^{2}(d)+\frac{c(\nu^{2}(d))^{2}}{\tau_{k}}+O(\tau_{k}^{-2})}{\tau_{k}}\right)=1 (123) By Definition 2.1 , the relation holds: \lambda_{k}\sim\tau_{k},\quad k=1,...,r. (124) Similarly, since \text{Spec}(\Sigma_{tail}) falls into the sub-critical regime of Theorem 3.5 , \lambda_{k}\sim\nu^{2}(d)(1+\sqrt{c})^{2},\quad k=r+1,...,r. (125) ∎ Let the SVD of \mathbf{X} be \mathbf{X}=U\Sigma_{svd}V^{T} , where \Sigma_{svd} is a diagonal matrix with diagonal elements being singular values \sigma_{1}(\mathbf{X})\geq\sigma_{2}(\mathbf{X})\geq\dots\geq\sigma_{d}(\mathbf{X})\geq 0 . Recall that the sample covariance matrix is defined as: S_{N}=\frac{1}{N}\mathbf{X}\mathbf{X}^{T}\in\mathbb{R}^{d\times d}. (126) Substituting \mathbf{X}=U\Sigma_{svd}V^{T} into S_{N} yields: \displaystyle S_{N} \displaystyle=\frac{1}{N}(U\Sigma_{svd}V^{T})(U\Sigma_{svd}V^{T})^{T} (127) \displaystyle=\frac{1}{N}U\Sigma_{svd}V^{T}V\Sigma_{svd}^{T}U^{T} Since V is an orthogonal matrix and \Sigma_{svd} is a diagonal matrix, V^{T}V=I , \Sigma_{svd}^{T}=\Sigma_{svd} . Therefore: \displaystyle S_{N} \displaystyle=\frac{1}{N}U(\Sigma_{svd}\Sigma_{svd})U^{T} (128) \displaystyle=U\left(\frac{1}{N}\Sigma_{svd}^{2}\right)U^{T} Note that Equation 128 is the SVD of S_{N} , where U is the eigenvector matrix. Therefore, \lambda_{k}=\frac{1}{N}\sigma_{k}^{2}(\mathbf{X}), (129) which gives: \sigma_{k}(\mathbf{X})=\sqrt{N\cdot\lambda_{k}}. (130) Since N is a scalar constant, in the limit process of d,N\to\infty , it only affects the amplitude and does not affect the shape of the distribution. By Lemma 3.6 , \begin{aligned} \sigma_{k}(\mathbf{X})&\sim\sqrt{N\cdot\tau_{k}}=\begin{cases}\sqrt{NL}\cdot k^{-\alpha/2},&\quad k\leq r\\ \sqrt{N\nu^{2}(d)}\cdot(1+\sqrt{c}),&\quad r\epsilon_{k}\right\}\\ &\iff\left\{|\sigma_{k}(\tilde{A})-\sigma_{k}(A)|>\epsilon_{k}\cdot\sigma_{k}(A)\right\}.\end{split} (159) By applying Theorem 2.6 , the spectral norm of the noise \|E\|_{2} is a deterministic upper bound on the absolute perturbation |\sigma_{k}(\tilde{A})-\sigma_{k}(A)| . This leads to a crucial logical implication: if the absolute perturbation exceeds \epsilon_{k}\cdot\sigma_{k}(A) , then the noise norm \|E\|_{2} must necessarily exceed this threshold as well. In set-theoretic terms, the relative error violation event is a subset of the spectral norm tail event: \left\{|\sigma_{k}(\tilde{A})-\sigma_{k}(A)|>\epsilon_{k}\cdot\sigma_{k}(A)\right\}\subseteq\left\{\|E\|_{2}>\epsilon_{k}\cdot\sigma_{k}(A)\right\}. (160) Consequently, by the monotonicity of probability measures, the probability of the relative error violation is strictly bounded by the tail probability of the spectral norm: P(\mathcal{R}(\epsilon_{k}))\leq P(\|E\|_{2}>\epsilon_{k}\cdot\sigma_{k}(A)). (161) Recalling the tail bound function F(t)=P(\|E\|_{2}\geq t) derived in Equation ( 42 ), we obtain: P(\mathcal{R}(\epsilon_{k}))\leq F(\epsilon_{k}\cdot\sigma_{k}(A)). (162) To guarantee the specified confidence level 1-\theta , we must ensure that the violation probability does not exceed \theta . We strictly equate the upper bound to \theta : F(\epsilon_{k}\cdot\sigma_{k}(A))=\theta. (163) This equation defines the boundary of our confidence interval. Since the tail function F(t) is strictly monotonically decreasing for t>0 , its inverse function F^{-1} exists. Solving for the specific relative error threshold \epsilon_{k} yields: \epsilon_{k}\cdot\sigma_{k}(A)=F^{-1}(\theta)\implies\epsilon_{k}(\theta)\triangleq\epsilon_{k}=\frac{F^{-1}(\theta)}{\sigma_{k}(A)}. (164) In this final expression, the numerator F^{-1}(\theta) represents a constant noise budget determined solely by the matrix dimensions, noise variance, and the confidence level \theta . It is independent of the index k . This explicitly demonstrates the inverse proportionality: \epsilon_{k}(\theta)\propto\frac{1}{\sigma_{k}(A)}. (165) Therefore, for any two singular values with magnitudes \sigma_{i}>\sigma_{j}>0 , under the same confidence level 1-\theta , it strictly holds that: \epsilon_{i}(\theta)<\epsilon_{j}(\theta). (166) This concludes the proof. ∎ The proof relies on perturbation theory and concentration inequalities. By Theorem 4.4 , the probability that the noise norm ||E||_{2} exceeds a threshold t is bounded by: P(\|E\|_{2}\geq t)\leq(m+n)\cdot\exp\left(-\frac{t^{2}}{C\cdot D\cdot s^{2}}\right). (167) Based on Theorem 2.6 , the absolute perturbation is bounded by the noise norm: |\sigma_{k}(\tilde{A})-\sigma_{k}(A)|\leq\|E\|_{2}. (168) Thus, the failure condition R_{k}>\eta implies \|E\|_{2}>\eta\cdot\sigma_{k}(A) . Substituting t=\eta\cdot\sigma_{k}(A) into the concentration bound yields: \theta_{k}=P(R_{k}>\eta)\leq(m+n)\cdot\exp\left(-\frac{\eta^{2}\cdot\sigma_{k}^{2}(A)}{C\cdot D\cdot s^{2}}\right). (169) We now evaluate Equation ( 169 ) by substituting the piecewise asymptotic spectral profile from Assumption 4.1 . Case 1: In regime 1\leq k\leq r , \sigma_{k}(A)\sim\mu\cdot k^{-\alpha/2} . Squaring this term yields \sigma_{k}^{2}(A)\sim\mu^{2}\cdot k^{-\alpha} . Substituting this into the exponent: \begin{split}\frac{\eta^{2}\cdot\sigma_{k}^{2}(A)}{C\cdot D\cdot s^{2}}&\sim\frac{\eta^{2}\cdot(\mu\cdot k^{-\alpha/2})^{2}}{C\cdot D\cdot s^{2}}\\ &=\frac{\eta^{2}\cdot\mu^{2}\cdot k^{-\alpha}}{C\cdot D\cdot s^{2}}\\ &=\underbrace{\frac{\eta^{2}\mu^{2}}{CDs^{2}}}_{\text{Constant term}}\cdot\underbrace{k^{-\alpha}}_{\text{Decay term}}.\end{split} (170) Applying Equation ( 170 ) to Equation ( 169 ) yields: \theta_{k}\lesssim(m+n)\cdot\exp\left(-\frac{\eta^{2}\mu^{2}}{CDs^{2}}\cdot\frac{1}{k^{\alpha}}\right), (171) Case 2: In regime r\alpha . Consequently, the scaling factor satisfies \frac{1+\beta}{1+\alpha}>1 . It follows that the denominator in the expression for F_{\tilde{A}}(k) increases relative to the numerator, leading to: F_{\tilde{A}}(k)\alpha , \beta>-1 . Finally, we observe that the stable rank S_{r}(A)=\frac{\|A\|_{F}^{2}}{\|A\|_{2}^{2}} is mathematically equivalent to the reciprocal of the energy concentration at k=1 : S_{r}(A)=\frac{\sum_{i}\sigma_{i}^{2}(A)}{\sigma_{1}^{2}(A)}=\frac{1}{F_{A}(1)}. (180) Applying the inequality from Eq. ( 177 ) at k=1 , we conclude: S_{r}(\tilde{A})=\frac{1}{F_{\tilde{A}}(1)}>\frac{1}{F_{A}(1)}=S_{r}(A). (181) ∎ Appendix E More Experiments E.1 Empirical Validation of Quantization Error Unbiasedness The validity of our spectral analysis relies heavily on Assumption 4.2 , which posits that quantization noise acts as a centered random variable with zero mean ( \mathbb{E}[e_{ij}]=0 ). To empirically substantiate this theoretical premise, we analyze the distribution of quantization errors in a pre-trained LLM. We employ a standard decoder-only Transformer architecture following the GPT-2-124M configuration ( L=12 layers, H=12 heads, d_{model}=768 ). The model is trained on the OpenWebText dataset using the AdamW optimizer with a peak learning rate of 1.5\times 10^{-4} and a global batch size of 2,560 (achieved via gradient accumulation on 8 GPUs). The detailed hyperparameters are summarized in Table 3 . All FP4 training in this work adopts a strict W4A4G4 quantization scheme, where weights, activations, and gradients are universally represented in the E2M1 NVFP4 format. We note that due to the proprietary nature of NVIDIA’s closed-source FP4 training software stack, native hardware-supported FP4 training is not currently accessible for open research. Consequently, consistent with standard practices in algorithmic research, our experiments with NVFP4 are conducted through high-fidelity simulation in BF16 . This simulation accurately emulates the quantization noise and dynamic range constraints of the E2M1 format while executing computations on standard BF16 -capable hardware. To verify the error properties, we extracted the quantization error matrices E=Q(W)-W from the Multi-Layer Perceptron (MLP) weights across all 24 sub-layers of the model. Figure 3 illustrates the error frequency histograms for each layer, overlaid with fitted normal distribution curves. Figure 3: The histograms display the frequency of quantization errors for MLP linear layers in the GPT-2-124M model quantized with NVFP4 . The red curves represent normal distribution fits. The annotated \mu values indicate the empirical mean of the error for each layer. Table 3: This table summarizes the model architecture, optimization hyperparameters, and quantization settings used to validate the unbiasedness of quantization error in Section E.1 . Model & Data Training Hyperparameters Architecture GPT-2 (Decoder-only) Optimizer AdamW Parameters \approx 124M Peak Learning Rate 1.5\times 10^{-4} Layers ( L ) 12 Weight Decay 0.1 Hidden Size ( d ) 768 Global Batch Size 2,560 Attention Heads ( H ) 12 Gradient Accumulation 40 steps Context Window 1024 tokens Warmup Steps 20 Dataset OpenWebText Gradient Clipping 1.0 Hardware 8 \times GPUs Epochs 1 Quantization Configuration ( NVFP4 ) Format NVFP4 (E2M1) - 4-bit Floating Point (2 exponent bits, 1 mantissa bit) Scope Weights, Activations, Gradients (Full FP4 Training) Scheme Block-wise Symmetric Uniform Quantization Results. As evidenced in Figure 3 , the quantization error distributions exhibit a high degree of symmetry around zero. Crucially, the empirical mean values ( \mu ) for all layers are negligible, ranging from order 10^{-4} to 10^{-2} (e.g., Layer 0: \mu=-1.50\times 10^{-3} , Layer 10: \mu=-5.36\times 10^{-4} ), which are statistically insignificant compared to the dynamic range of the weights. This empirical evidence strongly supports Assumption 4.2 . While the theoretical assumption simplifies the noise to a uniform distribution, the critical property required for our spectral perturbation bounds is the unbiased nature of the noise (i.e., the first moment \mathbb{E}[E]=0 ). The observed distributions confirm that the NVFP4 quantization mechanism does not introduce a systematic bias to the weight matrices, thereby validating the use of centered random matrix theory in our subsequent spectral analysis. E.2 Validation of Relative Error Bounds Building on the confirmation of error unbiasedness, this experiment aims to rigorously validate Theorem 4.5 , which posits a fundamental spectral asymmetry: the relative quantization error \epsilon_{k} is strictly inversely proportional to the magnitude of the singular value \sigma_{k} . Mathematically, this relationship is expressed as \epsilon_{k}\propto 1/\sigma_{k}(A) . This experiment inherits the identical experimental setup described in Section E.1 and summarized in Table 3 . We utilize the same pre-trained GPT-2-124M model and apply the simulated NVFP4 (W4A4G4) quantization to all linear layers. For each layer l , we compute the Singular Value Decomposition (SVD) of the original parameter matrix A^{(l)} and its quantized counterpart \tilde{A}^{(l)} . We then calculate the component-wise relative error \epsilon_{k} for each rank index k : \epsilon_{k}=\frac{|\sigma_{k}(\tilde{A}^{(l)})-\sigma_{k}(A^{(l)})|}{\sigma_{k}(A^{(l)})} (182) To test the linearity predicted by Theorem 4.5 , we perform a linear regression analysis of \epsilon_{k} against the inverse singular values 1/\sigma_{k}(A^{(l)}) . Figure 4: The scatter plots illustrate the relationship between the relative quantization error \epsilon_{k} (y-axis) and the inverse of the singular value 1/\sigma_{k} (x-axis) across varying layers of the GPT-2-124M model. The red dashed lines represent the linear regression fit. The coefficient of determination ( R^{2} ) is annotated for each layer, indicating the goodness of fit. Results. The results are presented in Figure 4 , covering layers 0 through 23. The visual evidence strongly corroborates our theoretical derivation. Across the majority of layers, we observe a striking linear relationship between the relative error \epsilon_{k} and the reciprocal of the singular value 1/\sigma_{k} . This confirms the prediction of Equation 44 that \epsilon_{k}(\theta)\propto\frac{1}{\sigma_{k}(A)} . The linear regression fits yield high coefficients of determination ( R^{2} ). For instance, deeper layers such as Layer 9 ( R^{2}=0.99 ), Layer 19 ( R^{2}=0.97 ), and Layer 23 ( R^{2}=0.98 ) exhibit near-perfect adherence to the theoretical model. While some initial layers (e.g., Layer 0 with R^{2}=0.42 ) show higher variance due to distinct initialization distributions, the global trend remains consistent. The data demonstrates that dominant singular values (where 1/\sigma_{k} is small) suffer from negligible relative error, effectively preserving the head information. Conversely, as we move to the tail (where 1/\sigma_{k} is large), the relative error scales linearly, leading to significant signal distortion. This mechanism explains the spectral flattening phenomenon: the quantization noise floor disproportionately destroys the low-energy components responsible for fine-grained semantic distinctions, while leaving the coarse-grained structural information relatively intact. This experiment provides robust empirical support for Theorem 4.5 , confirming that uniform quantization does not degrade information uniformly; rather, it selectively erodes the heavy-tailed spectral structure essential for LLM performance. E.3 Universality of Stable Rank Increase Theorem 4.7 predicts that uniform quantization inherently flattens the spectral distribution, leading to a strict increase in the stable rank of parameter matrices ( S_{r}(\tilde{A})>S_{r}(A) ). To demonstrate the universality of this phenomenon, we conducted full FP4 training (simulated W4A4G4) on a diverse set of Large Language Models. The experimental suite includes GPT-2-124M and five state-of-the-art architectures: TinyLlama-1.1B , Phi-1.5-1.3B , Pythia-1.4B , DeepSeek-Coder-1.3B , and Qwen2.5-1.5B . These models cover a wide spectrum of design choices, including varying depths (12 to 28 layers), embedding dimensions (768 to 2048), head counts (12 to 32), and vocabulary sizes (32k to 152k). The detailed architectural specifications and training hyperparameters are summarized in Table 4 . All models were trained on the OpenWebText dataset using 8 GPUs with NVFP4 quantization. We compared the singular value spectra of the pre-quantization weights ( BF16 ) and their post-quantization counterparts ( NVFP4 ) across all layers for each model. The results are visualized in Figure 5 , Figure 6 , and Figure 7 . Table 4: Model Architectures and Training Hyperparameters for Section E.3 . This table details the diverse set of LLMs used to validate the universality of Theorem 4.7 . All models utilize AdamW optimizer, Cosine learning rate schedule, and W4A4G4 ( NVFP4 ) quantization. Model Family Params Layers Hidden Dim Heads KV Heads Vocab Size Peak LR Context Window GPT-2 124M 12 768 12 12 50,257 1.5\times 10^{-4} 1024 TinyLlama 1.1B 22 2048 32 8 32,000 4.0\times 10^{-3} 1024 Phi-1.5 1.3B 24 2048 32 32 51,200 4.0\times 10^{-4} 1024 Pythia 1.4B 24 2048 16 16 50,304 4.0\times 10^{-4} 1024 DeepSeek-Coder 1.3B 24 2048 32 32 50,304 4.0\times 10^{-4} 1024 Qwen2.5 1.5B 28 1536 12 2 151,936 4.0\times 10^{-4} 1024 GPT-2-124M TinyLlama-1.1B Figure 5: Spectral analysis of GPT-2-124M and TinyLlama-1.1B , where quantization induces a consistent positive shift in Stable Rank (SR+) in the majority of layers. Results. As shown in Figures 5 - 7 , every tested architecture exhibits the characteristic spectral flattening pattern. The head singular values remain relatively stable, while the tail singular values are significantly elevated by the quantization noise floor. Quantitatively, this results in a strict increase in the Stable Rank for the vast majority of layers (indicated by SR+ in the plots). Starting with the GPT-2-124M , we observe the foundational spectral degradation. This trend is amplified in larger, deeper models. In Qwen2.5-1.5B , which features a deep narrow structure (28 layers, 1536 dim), the stable rank increase is pronounced across middle layers. Similarly, TinyLlama-1.1B and Phi-1.5-1.3B show consistent spectral degradation. This cross-model consistency validates Theorem 4.7 as a fundamental law governing low-precision embeddings, suggesting that the loss of representational capacity via tail truncation is an unavoidable consequence of uniform quantization in heavy-tailed distributions. Phi-1.5-1.3B Pythia-1.4B Figure 6: Spectral analysis of Phi-1.5-1.3B and Pythia-1.4B , where quantization induces a consistent positive shift in Stable Rank (SR+) in the majority of layers. DeepSeek-Coder-1.3B Qwen2.5-1.5B Figure 7: Spectral analysis of DeepSeek-Coder-1.3B and Qwen2.5-1.5B , where quantization induces a consistent positive shift in Stable Rank (SR+) in the majority of layers.