MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration Lianhai Ren, Yucheng Ding, Xiao Liu, Qianxiao Li, Peng Cheng, Yeyun Gong Abstract Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational resources. We study training failures in a 5M-parameter NanoGPT model scaled via μ \mu P, identifying two key phenomena preceding collapse: (1) rapid decline in weight matrix stable rank (ratio of squared Frobenius norm to squared spectral norm), and (2) increasing alignment between adjacent layer Jacobians. We prove theoretically that these two conditions jointly cause exponential gradient norm growth with network depth. To break this instability mechanism, we propose MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank. Experiments on models from 5M to 3B parameters demonstrate that MSign effectively prevents training failures with a computational overhead of less than 7.0%. † † affiliationnotice: † Work done during their internship at Microsoft Research. 1 Introduction Modern deep learning optimizers and training techniques – including Adam ( Kingma & Ba 2015 ) , layer normalization ( Ba et al. 2016 ) , and learning rate schedules – generally enable reliable training. However, as the model scale increases, pretraining large language models (LLMs) becomes increasingly fragile. Training failures manifest as sudden, unrecoverable gradient explosions and corresponding loss growth, which are difficult to predict and can waste substantial computational resources ( Chowdhery et al. 2023 ; Zhang et al. 2022 ) . Empirical Investigation: Identifying the Failure Mechanism. We systematically study training failures using a 5M-parameter NanoGPT model derived from \mu P scaling ( Yang et al. 2022 ) , which provides a controlled environment for reproducible failure analysis. Through extensive monitoring, we identify two critical phenomena that consistently precede training collapse (Figure 1 ): • Observation 1: Stable Rank Collapse. The stable rank of weight matrices – defined as \mathrm{srank}(\mathbf{W})=\|\mathbf{W}\|_{F}^{2}/\|\mathbf{W}\|_{2}^{2} – declines sharply in the steps preceding failure. This indicates that spectral energy becomes concentrated in the top singular directions, reducing the “effective dimensionality” of the weight matrices. • Observation 2: Jacobian Alignment Growth. The alignment between adjacent layer Jacobians increases, meaning the top singular subspaces of consecutive layers become increasingly correlated. It prevents the typical cancellation effects in matrix products. We prove that these two phenomena jointly cause training instability: low stable rank implies high layer Jacobian spectral norms (since \|\mathbf{W}\|_{2}=\|\mathbf{W}\|_{F}/\sqrt{\mathrm{srank}(\mathbf{W})} for fixed Frobenius norm), and high alignment ensures these norms multiply constructively across layers. The total Jacobian norm grows as (aM)^{L} where a is alignment, M is layer Jacobian norm, and L is depth, leading to exponential gradient explosion when aM>1 . Our Solution: The MSign Optimizer. To break the stable rank collapse condition, we propose MSign , an optimizer that periodically applies the matrix sign operation \mathrm{sign}(\mathbf{W})=\mathbf{U}\mathbf{V}^{T} (where \mathbf{W}=\mathbf{U}\mathbf{S}\mathbf{V}^{T} is the SVD). This operation maximizes stable rank by equalizing all non-zero singular values to 1, while preserving the column and row spaces. We restore the original Frobenius norm after the operation to maintain training dynamics. MSign is applied every P steps (typically P=100 ) to projection weights, with computational overhead below 7.0%. Experimental Validation. We validate MSign across four model configurations spanning 5M to 3B parameters: NanoGPT-5M 1 1 1 https://github.com/karpathy/nanoGPT , Sigma-40M (hybrid MHA/MLA attention) ( Qu et al. 2025 ; Hu et al. 2025 ) , LLaMA-1B ( Kumar et al. 2025 ) , and LLaMA-MoE-3B (mixture of experts). In all cases, baseline training with standard hyperparameters fails via gradient explosion, while training with MSign converges stably. The intervention maintains stable rank above critical thresholds, controls Jacobian alignment, and keeps gradient norms bounded. Ablation studies reveal that applying MSign to attention layers (particularly output projections) is sufficient, MLP-only application does not prevent failures. Contributions. We summarize our contributions: 1. Mechanism identification and theoretical analysis : We identify stable rank collapse and Jacobian alignment as consistent precursors to LLM training failures, and prove that their combination causes exponential gradient growth with network depth. 2. Practical solution : We propose MSign, a new optimizer that prevents training failures by periodically restoring weight stable rank. 3. Extensive validation : We demonstrate MSign’s effectiveness across diverse architectures (dense, MoE) and scales (5M–3B parameters) with minimal overhead. 2 Literature Review Training Instability in Large Language Models. Training instability in large language models has been widely observed across major LLM projects. Chowdhery et al. 2023 documented loss spikes during PaLM training that required manual intervention and checkpoint rollbacks, while Zhang et al. 2022 provided detailed logs of OPT-175B training showing dozens of restarts due to hardware failures and gradient explosions. Zeng et al. 2023 reported similar challenges in GLM-130B training. Several factors have been proposed to explain instability: Kaplan et al. 2020 studied the sensitivity of training dynamics to learning rate in the context of scaling laws; Pascanu et al. 2012 analyzed gradient clipping as a remedy for exploding gradients in RNNs; Dong et al. 2021 identified attention entropy collapse as a failure mode in Vision Transformers. More recently, Wortsman et al. 2024 showed that small-scale proxy models can predict instabilities at larger scales, and Moniri et al. 2024 proposed a theoretical framework connecting loss spikes to heavy-tailed gradient noise. However, these explanations typically address symptoms rather than underlying mechanisms. Low-Rank Structure in Neural Networks. The prevalence of low-rank structure in neural network weights and gradients has been extensively documented. Denil et al. 2013 demonstrated that neural network weights exhibit significant redundancy, with up to 95% of parameters predictable from the remaining 5%. Arora et al. 2019 proved that gradient descent in deep linear networks implicitly biases toward low-rank solutions. In the transformer context, Hu et al. 2022 exploited low-rank structure for parameter-efficient fine-tuning, while Zhao et al. 2024 showed that gradients during transformer training maintain consistently low stable rank, motivating memory-efficient optimizers. Huh et al. 2024 found that representations across different models and modalities converge to similar low-dimensional structures. Our work extends these observations by connecting low-rank gradient structure directly to training instability through the stable rank mechanism. Jacobian Analysis in Deep Networks. The role of Jacobians in neural network optimization has received substantial theoretical attention. Classical work by Glorot & Bengio 2010 established the importance of proper weight initialization for maintaining stable gradient flow. Saxe et al. 2013 analyzed dynamics in deep linear networks through the lens of Jacobian singular values. Pennington et al. 2017 and Yang & Schoenholz 2017 developed mean-field theories for Jacobian evolution in residual networks and general deep networks respectively. Fort et al. 2019 empirically studied the relationship between Jacobian eigenvalue spectra and model trainability, finding that successful training correlates with specific spectral properties. Xiao et al. 2018 proposed dynamical isometry as a condition for stable training. Our contribution is identifying inter-layer Jacobian alignment as a key mechanism amplifying gradient explosion, distinct from prior work that focused on individual layer Jacobian norms. Stable Rank and Matrix Analysis. Stable rank, introduced by Rudelson & Vershynin 2007 in the context of random matrix theory, provides a continuous relaxation of matrix rank that is more robust to small perturbations. Vershynin 2018 provides comprehensive treatment of stable rank properties and its applications in high-dimensional probability. In neural network analysis, stable rank has been used to derive generalization bounds: Neyshabur et al. 2017 used stable rank to obtain PAC-Bayes bounds. Li et al. 2018 analyzed intrinsic rank dynamics during optimization, finding that it correlates with model compressibility. Sanyal et al. 2020 proposed stable rank regularization to improve generalization. Our work reveals a novel connection between stable rank dynamics and training stability, showing that stable rank collapse precedes and causes gradient explosion. Figure 1 : Correlation between training failure indicators and gradient norm explosion. Left (Observation 1): Stable rank (geometric mean across early layers) vs. gradient norm over training steps. As stable rank declines sharply around step 20000, gradient norms begin explosive growth. Right (Observation 2): Jacobian alignment (average between adjacent layers) vs. gradient norm. Increasing alignment precedes and accompanies gradient explosion. 3 Empirical Observations: Training Failure Phenomena We identify reproducible training failures in transformer models under moderate learning rates. Our analysis reveals two consistent phenomena preceding training collapse. 3.1 Experimental Setup We construct a reproducible failure scenario using a modified NanoGPT configuration with standard hyperparameters (refer to Appendix A for details). From \mu P perspective ( Yang et al. 2022 ) , this configuration corresponds to a 0.02 std and 6\times 10^{-4} learning rate at 100 M scale, which is within typical ranges. Model and Jacobian Notation. We consider a standard decoder-only transformer with L stacked blocks. At each block \ell , hidden states \mathbf{H}^{(\ell-1)}\in\mathbb{R}^{T\times d} (sequence length T , hidden dimension d ) are first processed by multi-head self-attention with query/key/value/output projections \mathbf{W}_{Q}^{(\ell)},\mathbf{W}_{K}^{(\ell)},\mathbf{W}_{V}^{(\ell)},\mathbf{W}_{O}^{(\ell)} , and then by a position-wise MLP with weights \mathbf{W}_{1}^{(\ell)}\in\mathbb{R}^{d\times d_{\mathrm{ff}}} and \mathbf{W}_{2}^{(\ell)}\in\mathbb{R}^{d_{\mathrm{ff}}\times d} , with residual connections and LayerNorm around each sublayer: \mathbf{H}^{(\ell)}=F^{(\ell)}(\mathbf{H}^{(\ell-1)}),\quad\ell=1,\dots,L. (1) We define the layer Jacobian as \mathbf{J}^{(\ell)}=\frac{\partial\,\mathrm{vec}(\mathbf{H}^{(\ell)})}{\partial\,\mathrm{vec}(\mathbf{H}^{(\ell-1)})} . Later references to the stable ranks of \mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V},\mathbf{W}_{O},\mathbf{W}_{1},\mathbf{W}_{2} and to ”Jacobian alignment between adjacent layers” all refer to this architecture and these \mathbf{J}^{(\ell)} . 3.2 Observations during training failure Observation 1: Sharp Stable Rank Decline The first observable phenomenon is the rapid decline in weight stable rank. As shown in Figure 1 (left panel), the geometric averaged stable rank of the first several layers drops sharply around step 20000, preceding the gradient explosion. This phenomenon indicates energy concentration in top singular values. Definition 3.1 (Stable Rank) . The stable rank of a matrix \mathbf{W}\in\mathbb{R}^{m\times n} is: \mathrm{srank}(\mathbf{W})=\frac{\|\mathbf{W}\|_{F}^{2}}{\|\mathbf{W}\|_{2}^{2}}=\frac{\sum_{i}s_{i}^{2}}{s_{1}^{2}}, (2) where s_{1}\geq s_{2}\geq\cdots\geq 0 are the singular values. Observation 2: Increasing Jacobian Alignment Between Adjacent Layers The second critical phenomenon is the increasing alignment between Jacobians of adjacent layers, as illustrated in Figure 1 (right panel). Definition 3.2 (Matrix Alignment) . For any two matrices \mathbf{A},\mathbf{B} for which the product \mathbf{A}\mathbf{B} is well-defined, their alignment is the cosine similarity between the top right singular vector of \mathbf{A} and the top left singular vector of \mathbf{B} : \text{Align}(\mathbf{A},\mathbf{B})=|\mathbf{v}_{A,1}^{T}\mathbf{u}_{B,1}|, (3) where \mathbf{v}_{A,1} is the first right singular vector of \mathbf{A} and \mathbf{u}_{B,1} is the first left singular vector of \mathbf{B} . For simplicity, we denote the alignment of Jacobians for adjacent layers, \text{Align}(\mathbf{J}^{(\ell+1)},\mathbf{J}^{(\ell)}) , as \text{Align}(\ell+1,\ell) . High alignment correlates strongly with both weight scale growth and stable rank decline, as well as gradient growth during the failure phase. 3.3 Conjecture: Low Stable Rank + Jacobian Alignment Drives Training Failure Based on these two consistent phenomena, we conjecture that the combination of low weight stable rank and high Jacobian alignment creates a destabilizing feedback mechanism that leads to training failure. The remainder of this paper develops this hypothesis theoretically and proposes a solution. 4 Theoretical Analysis: Understanding the Failure Mechanism We now provide theoretical analysis to explain the observed phenomena and their causal relationships. Our analysis is divided into two parts: (1) explaining why Jacobian alignment and low stable rank lead to training failure, and (2) analyzing the positive feedback mechanism that prevents stable rank from increasing and accelerates its collapse. 4.1 Part I: From Observations to Training Failure In this section, we establish the causal chain: Low stable rank + Jacobian alignment \Rightarrow High total Jacobian norm \Rightarrow Large weight gradient norm ( Training instability ). Remark 4.1 (Simplifying Assumption) . We adopt the standard assumption that large gradient norms indicate training instability ( Pascanu et al. 2012 ; Philipp et al. 2017 ) . Our analysis derives lower bounds for gradient norms, with the understanding that such bounds indicate increased risk of divergence. 4.1.1 High Jacobian Alignment + High Layer Jacobian Norm \Rightarrow High Total Jacobian Norm The total Jacobian \mathbf{J}_{total}=\prod_{\ell=1}^{L}\mathbf{J}^{(\ell)} determines how perturbations at the input propagate to the output. In general, the norm of a matrix product can be much smaller than the product of norms due to cancellation effects. However, when the singular subspaces of adjacent Jacobians are aligned , these cancellations are suppressed, and the norms multiply constructively. To formalize this, we use the definition of matrix alignment (Definition 3.2). This quantity measures how well the ”output direction” of \mathbf{J}^{(\ell)} matches the ”input direction” of \mathbf{J}^{(\ell+1)} , determining whether the composition \mathbf{J}^{(\ell+1)}\mathbf{J}^{(\ell)} preserves or diminishes the spectral norm. Theorem 4.2 (Jacobian Product Norm Lower Bound) . For a deep network with L layers, let \mathbf{J}^{(\ell)}=\frac{\partial\mathbf{h}^{(\ell)}}{\partial\mathbf{h}^{(\ell-1)}} denote the Jacobian at layer \ell . If each layer Jacobian satisfies \|\mathbf{J}^{(\ell)}\|_{2}\geq M and the alignment between adjacent Jacobians satisfies \text{Align}(\ell+1,\ell)\geq a>0 for all \ell , then the total Jacobian from input to output has 2-norm: \|\mathbf{J}_{total}\|_{2}=\|\mathbf{J}^{(L)}\mathbf{J}^{(L-1)}\cdots\mathbf{J}^{(1)}\|_{2}\geq\frac{(aM)^{L}}{a}. (4) The proof is provided in Appendix B.1 . Remark 4.3 (Exponential Growth Condition) . The key insight is that when aM>1 , the lower bound in Theorem 4.2 grows exponentially with depth L , providing a sufficient condition for large total Jacobian norms. Observation 1 (stable rank collapse) suggests that M can become large (via Theorem 4.4 under approximately fixed Frobenius norms), and Observation 2 shows that a tends to increase during training. In the failure regimes we study, these trends empirically drive aM above 1 , which is consistent with the observed gradient explosion, although we do not claim that aM>1 holds at all times or is necessary for failure. 4.1.2 Low Stable Rank \Rightarrow High Layer Jacobian Norm We analyze the relationship between stable rank and layer Jacobian norm for the three primary layer types in transformers. Linear Layers. Theorem 4.4 (Stable Rank Controls Jacobian Norm: Linear Layer) . For a linear layer with weight matrix \mathbf{W}\in\mathbb{R}^{m\times n} , given fixed Frobenius norm \|\mathbf{W}\|_{F}=F , the operator norm satisfies: \|\mathbf{W}\|_{2}=\frac{F}{\sqrt{\mathrm{srank}(\mathbf{W})}}. (5) The proof is provided in Appendix B.2 . This establishes the basic principle: as stable rank decreases while the Frobenius norm is kept fixed (or approximately fixed over short training windows, e.g., under Adam-like updates with bounded step sizes), the operator norm increases proportionally. Attention Layers. Theorem 4.5 (Jacobian Norm Bound: Attention Layer) . Consider a single-head attention layer with projections \mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V},\mathbf{W}_{O}\in\mathbb{R}^{d\times d_{k}} . Let \mathbf{H}\in\mathbb{R}^{n\times d} be the input, \mathbf{A}=\text{softmax}\left(\frac{\mathbf{H}\mathbf{W}_{Q}\mathbf{W}_{K}^{T}\mathbf{H}^{T}}{\sqrt{d_{k}}}\right) be the attention matrix. The Jacobian of the attention output \mathbf{Y}=\mathbf{A}\mathbf{H}\mathbf{W}_{V}\mathbf{W}_{O} satisfies: \displaystyle\left\|\frac{\partial\mathbf{Y}}{\partial\mathbf{H}}\right\|_{2}\leq \displaystyle\|\mathbf{A}\|_{2}\|\mathbf{W}_{V}\|_{2}\|\mathbf{W}_{O}\|_{2} (6) \displaystyle+\left\|\frac{\partial\mathbf{A}}{\partial\mathbf{H}}\right\|_{2}\|\mathbf{H}\|_{2}\|\mathbf{W}_{V}\|_{2}\|\mathbf{W}_{O}\|_{2}. Defining the logit margin \gamma_{\min}=\min_{i}(\max_{j}\mathbf{S}_{i,j}-\text{second\_max}_{j}\mathbf{S}_{i,j}) where \mathbf{S}=\mathbf{H}\mathbf{W}_{Q}\mathbf{W}_{K}^{T}\mathbf{H}^{T} , the attention gradient pathway is bounded by: \left\|\frac{\partial\mathbf{A}}{\partial\mathbf{H}}\right\|_{2}\leq\frac{4\min((n-1)e^{-\gamma_{\min}},1)}{\sqrt{d_{k}}}\|\mathbf{W}_{Q}\|_{2}\|\mathbf{W}_{K}\|_{2}. (7) The proof is provided in Appendix B.3 . Substituting \|\mathbf{W}_{i}\|_{2}=\|\mathbf{W}_{i}\|_{F}/\sqrt{\mathrm{srank}(\mathbf{W}_{i})} shows that low stable rank in any projection matrix amplifies the Jacobian norm. MLP Layers. Theorem 4.6 (Jacobian Norm Bound: MLP Layer) . Consider a two-layer MLP with weights \mathbf{W}_{1}\in\mathbb{R}^{d\times d_{\text{ff}}} , \mathbf{W}_{2}\in\mathbb{R}^{d_{\text{ff}}\times d} , and activation \phi . The Jacobian satisfies: \left\|\mathbf{J}_{\text{MLP}}\right\|_{2}\leq\frac{L_{\phi}\|\mathbf{W}_{1}\|_{F}\|\mathbf{W}_{2}\|_{F}}{\sqrt{\mathrm{srank}(\mathbf{W}_{1})\cdot\mathrm{srank}(\mathbf{W}_{2})}}, (8) where L_{\phi} is the Lipschitz constant of \phi ( L_{\phi}\approx 1.13 for GELU, L_{\phi}\approx 1.1 for SiLU). The proof is provided in Appendix B.4 . Across all layer types, the Jacobian norm is inversely related to the square root of stable rank, creating conditions for exponential gradient growth when combined with Jacobian alignment (Theorem 4.2 ). 4.1.3 High Total Jacobian Norm \Rightarrow Large Weight Gradient Norm We now show that high total Jacobian norms translate into large weight gradients. By the chain rule, the gradient with respect to weight vector \hat{\mathbf{v}}_{out}^{(i)} at layer i decomposes as: \frac{\partial L}{\partial\hat{\mathbf{v}}_{out}^{(i)}}=\left(\frac{\partial\mathbf{h}^{(i)}}{\partial\hat{\mathbf{v}}_{out}^{(i)}}\right)^{T}\left(\frac{\partial\mathbf{h}^{(L)}}{\partial\mathbf{h}^{(i)}}\right)^{T}\frac{\partial L}{\partial\mathbf{h}^{(L)}}. (9) Assumption 4.7 (Gradient Alignment Conditions (Informal)) . We assume: (1) uniform local gradient lower bound \gamma>0 ; (2) local-Jacobian alignment: \frac{\partial\mathbf{h}^{(i)}}{\partial\hat{\mathbf{v}}_{out}^{(i)}} is aligned with the top right singular direction of the local layer Jacobian \mathbf{J}^{(i+1)} , with alignment \geq a ; (3) terminal alignment: the last layer Jacobian \mathbf{J}^{(L)} ’s top left singular direction is aligned with the loss gradient \frac{\partial L}{\partial\mathbf{h}^{(L)}} , with alignment \geq a . The formal statement is provided in Assumption B.2 in the Appendix. Theorem 4.8 (Weight Gradient Norm Lower Bound) . Under Assumption 4.7 , combined with Theorem 4.2 when \|\mathbf{J}^{(\ell)}\|_{2}\geq M>1 and alignment \geq a : \left\|\frac{\partial L}{\partial\hat{\mathbf{v}}_{out}^{(i)}}\right\|_{2}\geq a\gamma(aM)^{L-i}\cdot\left\|\frac{\partial L}{\partial\mathbf{h}^{(L)}}\right\|_{2}. (10) The proof is provided in Appendix B.5 . Figure 2 validates this bound empirically. Figure 2 : Validation of Theorem 4.2 : Jacobian product norm lower bound vs. actual gradient norm. The theoretical bound closely tracks observed gradient growth. Theorem 4.9 (Total Gradient Norm Lower Bound) . Summing over all layers: \sum_{i=1}^{L}\left\|\frac{\partial L}{\partial\mathbf{W}^{(i)}}\right\|_{F}^{2}\geq C\cdot\frac{(aM)^{2L}-1}{(aM)^{2}-1}, (11) where C=a^{2}\gamma^{2}n_{w}\left\|\frac{\partial L}{\partial\mathbf{h}^{(L)}}\right\|_{2}^{2} . When aM>1 , this exhibits exponential growth in depth. The proof is provided in Appendix B.6 . From Observations 1 and 2, stable rank collapse and Jacobian alignment together cause aM>1 , triggering exponential gradient explosion. 4.1.4 Summary: The Failure Pathway Combining the above results, we establish the complete causal chain: 1. Low stable rank ( Observation 1 ) \Rightarrow High layer Jacobian 2-norm (Theorems 4.4 , 4.5 , 4.6 ) under approximately fixed Frobenius norms. 2. Jacobian alignment ( Observation 2 ) + High layer Jacobian norm \Rightarrow High total Jacobian norm (Theorem 4.2 ). 3. High total Jacobian norm + Gradient alignment (Assumption 4.7 ) \Rightarrow Large weight gradient (training instability) (Theorems 4.8 , 4.9 ). Taken together, these ingredients provide a sufficient mechanistic explanation for how low stable rank combined with Jacobian alignment can lead to training failure in the regimes we study; we do not claim that this mechanism is necessary for all possible failure modes. 4.2 Part II: The Positive Feedback Mechanism Having established why low stable rank and Jacobian alignment lead to training failure, we now analyze why these conditions tend to intensify during training. 4.2.1 Low-Rank Hidden States Lead to Low-Rank Gradients We first establish that low-rank structure propagates through the network and affects gradient structure. Theorem 4.10 (Low-Rank Propagation in Attention Layers) . Consider an attention layer with query, key, value, and output projection matrices \mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V},\mathbf{W}_{O} . If the hidden states \mathbf{H}^{(\ell-1)} and cohidden states \tilde{\mathbf{H}}^{(\ell)} have rank at most r , then the gradients \nabla_{\mathbf{W}_{Q}}L,\nabla_{\mathbf{W}_{K}}L,\nabla_{\mathbf{W}_{V}}L,\nabla_{\mathbf{W}_{O}}L all have rank at most r . The proof is provided in Appendix B.7 . Remark 4.11 . This result extends to MLP layers due to their two-layer structure. For MLP up-projection \mathbf{W}_{1} : \nabla_{\mathbf{W}_{1}}L=\mathbf{H}^{T}\cdot(\text{something}) , so rank is bounded by \text{rank}(\mathbf{H}) . The key insight is that outer-product gradients inherit the rank of their lower-rank factor. 4.2.2 Aligned Input-Weight-Output Structure Accelerates Stable Rank Decline In general, characterizing how gradient descent affects stable rank dynamics is challenging, as the update direction depends on the complex interplay between input statistics, weight structure, and backpropagated gradients. Here we provide a simplified analysis in a stylized linear setting under strong alignment assumptions that captures the essential feedback mechanism. While these assumptions are restrictive and do not aim to exactly describe full transformer dynamics, they offer theoretical insight into why stable rank can tend to decline during training. Theorem 4.12 (Stable Rank Feedback Mechanism) . Consider a linear layer with weight matrix \mathbf{W}=\mathbf{U}\mathbf{S}\mathbf{V}^{T} where the input hidden states and output cohidden states satisfy: • Input covariance: \Sigma_{in}=\mathbb{E}[\mathbf{h}_{in}\mathbf{h}_{in}^{T}]=\mathbf{V}\Lambda_{in}\mathbf{V}^{T} (aligned with \mathbf{W} ’s right singular vectors) • Output gradient covariance: \Sigma_{out}=\mathbb{E}[\tilde{\mathbf{h}}_{out}\tilde{\mathbf{h}}_{out}^{T}]=\mathbf{U}\Lambda_{out}\mathbf{U}^{T} (aligned with \mathbf{W} ’s left singular vectors) If the correlation between input and output gradient projections \text{Cov}(\mathbf{u}_{i}^{T}\tilde{\mathbf{h}}_{out},\mathbf{v}_{i}^{T}\mathbf{h}_{in}) is negative and satisfy \frac{\text{Cov}(\mathbf{u}_{1}^{T}\tilde{\mathbf{h}}_{out},\mathbf{v}_{1}^{T}\mathbf{h}_{in})}{\text{Cov}(\mathbf{u}_{i}^{T}\tilde{\mathbf{h}}_{out},\mathbf{v}_{i}^{T}\mathbf{h}_{in})}>\frac{s_{1}}{s_{i}},\forall 1