The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen, Xueqi Cheng · 2026
Read The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training by Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen, Xueqi Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26489.txt.
Introduction
Hongtao Zhang and four colleagues identify a spectral marker for the two-phase shape of pretraining: a fast early loss drop, then slow improvement. Their Stability of Singular Distribution says the trace-normalized singular-value spectrum settles early even as weights keep changing. This settling lines up with the slow-descent regime across GPT-2 and LLaMA models, varied schedules, weight decays, and AdamW and Muon optimizers. A simplified transformer analysis proves growing weight norms force an early threshold. After that point, loss-decline speed is bounded by singular-distribution change.
Catalog record
| CREATOR | Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen, Xueqi Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74652 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26489
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen, Xueqi Cheng Abstract Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics. † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) have established themselves as the cornerstone of modern artificial intelligence ( 7 ; 1 ) , achieving unprecedented scalability and generalization by leveraging the Transformer architecture ( 39 ) as their foundational backbone. However, the optimization dynamics governing their pre-training remain enigmatic, particularly regarding the temporal evolution of the training process. A ubiquitous observation recorded across major technical reports ( 38 ; 10 ; 47 ) is the characteristic two-phase trajectory of the training loss: an initial regime of precipitous decay followed by a prolonged period of asymptotic, heavy-tailed improvement. While this “fast-then-slow” two-phase behavior is empirically taken for granted as the standard convergence pattern of Large Language Models (LLMs), the underlying theoretical mechanisms driving this transition and specifically, what mechanistic factors dictate the onset of the slow-descent phase remain fundamentally under-explored. Figure 1 : Identification of the Stability of Singular Distribution (SoSD) phenomenon on GPT-2 Small. (a) Evolution of cosine similarity between current states and final states. The singular distributions (dashed lines) stabilize significantly earlier than the parameter matrices themselves (solid lines). (b) Synchronization between Validation Loss (top) and Singular Distribution Variation (SD Variation, bottom). Vertical red dashed lines mark the approximate steps where singular value matrices stabilize, coinciding with the transition into the slow descent regime. Existing analyses often rely on restricted task formulations, such as in-context learning ( 27 ; 6 ; 48 ) or linear regression ( 46 ) , to derive tractable convergence bounds. Recently, the focus has shifted towards more granular, multi-stage interpretations of Transformer optimization ( 51 ; 44 ; 49 ) . Notably, recent work on Condensation to Rank Collapse ( 9 ) employs a gradient flow framework on linearized attention, identifying a two-stage transition from parameter condensation to asymptotic rank collapse under small initialization. Motivated by these insights, we propose to shift the lens towards the temporal dynamics of the spectral evolution. Crucially, this perspective allows us to investigate the mechanistic synchronization between stability of singular distribution and the macroscopic saturation of the loss function, addressing a fundamental question: What intrinsic mechanism governs the transition from fast learning to slow saturation, and how does the spectral evolution of the parameters dictate this regime shift? To investigate the mechanism underlying this transition, we analyze the spectral dynamics of the parameter matrices throughout the pre-training process. We identify a phenomenon termed the Stability of Singular Distribution (SoSD) , where the normalized singular value spectrum stabilizes significantly earlier than the parameter matrices themselves (Figure 1 (a)), and the onset of this stability aligns with the validation loss entering a plateau, exhibiting a tight synchronization with optimization saturation (Figure 1 (b)). Our analysis reveals that this spectral stabilization closely characterizes the shift from the fast to the slow descent phase. Our specific contributions are as follows: 1. Identification of SoSD : We identify the Stability of Singular Distribution (SoSD) phenomenon, observing that the singular value distribution enters a stable state significantly prior to the stability of parameter matrices(Figure 1 (a)). Across GPT-2 and LLaMA families, we report a synchronization that the onset of SoSD aligns with the loss function’s transition to the slow descent regime(Figure 1 (b)). 2. Theoretical Analysis of SoSD : We establish a theoretical framework to clarify the mechanism of SoSD. We first establish the emergence of SoSD (Theorem 4.5 ) contingent upon the Non-Degeneracy of parameters and Gradient Boundedness (Assumption 4.3 ). Building on this, we incorporate Smoothness and Margin Conditions (Assumption 4.7 ) to demonstrate that loss descent is dynamically coupled with singular distribution variation: while large variation is associated with rapid loss decay, the onset of SoSD strictly bounds the subsequent loss reduction (Theorem 4.9 ). 3. Interpreting Pre-training Strategies via SoSD : We interpret pre-training strategies through the lens of SoSD dynamics and the derived stability bound \varepsilon\propto\eta/\|W\| , where \eta and \|W\| denote the learning rate and the norm of parameters, respectively. We demonstrate that learning rate schedules (step-wise decay and continuous annealing) facilitate optimization by tightening this bound, thereby mitigating the SoSD-related constraint to enable further loss minimization. Conversely, we find that Weight Decay facilitates loss descent by suppressing the growth of the weight norm; this mechanism relaxes the stability constraint, thereby permitting larger updates to the singular distribution. Finally, we validate SoSD with the Muon optimizer, observing that the SoSD phenomenon persists alongside its superior training efficiency. Conflict of Interest Disclosure. The authors declare no financial conflicts of interest related to this work. Figure 2 : Evolution of cosine similarity between current parameters( t ) and their final states( T ) during pre-training. The plots analyze cosine similarity \mathrm{cos}\langle W_{t},W_{T}\rangle and \mathrm{cos}\langle\Sigma_{t},\Sigma_{T}\rangle across four models: (a) GPT-2 Small, (b) GPT-2 Medium, (c) LLaMA 0.5B, and (d) LLaMA 2B. Solid lines represent the weight matrices ( W ), while dashed lines represent the singular value matrices ( \Sigma ). Colors denote different projection layers. The vertical red dashed lines mark the approximate steps where the singular value matrices stabilize. 2 Related Work Training Dynamics of Transformers Understanding the optimization trajectory of Transformers remains a formidable challenge given the interplay between non-convex objectives and massive parameter scales. Early theoretical efforts primarily dissected the dynamics of simplified, single-layer attention mechanisms. For instance, 37 and 33 characterized how gradient descent drives attention heads to capture co-occurrence patterns or mimic Seq2Seq algorithms, while 22 demonstrated the learnability of topic models within a BERT-like framework ( 13 ) . Building on these foundational settings, recent studies have extended dynamic analysis to more complex distributions. For instance, the staged learning phases in two-mixture linear classification have been detailed ( 42 ) . Similarly, in the realm of logical reasoning, theoretical proofs have established how attention and linear layers evolve to solve regular language tasks via Chain-of-Thought ( 19 ) . More recently, the focus has shifted towards In-Context Learning (ICL) as a testbed for understanding dynamics. A substantial body of work has adopted the linear regression setting to theoretically analyze how Transformers implement gradient descent-like algorithms during inference ( 3 ; 40 ; 24 ; 46 ) . Refining this perspective, recent research demonstrates that Transformers can transcend simple algorithms to learn latent representations, implicitly performing