Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao · 2026
Read Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering by Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03899.txt.
Introduction
Li, Zhang, Liu, and Bao explain why momentum helps the Muon optimizer by acting as a spectral filter. Muon trains large language models well in practice. What momentum does inside it stayed unclear. Past analyses dropped momentum or kept it unexplained. A structured signal-plus-perturbation gradient model frames the answer.
Momentum suppresses the perturbations while keeping the dominant signal. The spectral gap between them widens as a result. That wider gap steadies the singular subspaces fed to orthogonalization. The update that follows is more reliable. Running momentum before orthogonalization provably aligns better with the signal than the reverse order. Dropping momentum is worse still. Experiments, including language model pretraining, back the theory. The frame may extend to other matrix-based optimizers. Trainer teams get a reasoned basis for Muon's recipe, not just folklore.
Catalog record
| CREATOR | Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 138 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao Abstract Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon’s orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers. Project page: yinleung.com/denoise-ortho 1 Introduction Modern neural networks, especially large language models (LLMs), typically consist of billions of parameters and require substantial computational resources for training. As the model scale continues to grow, optimizer design has become increasingly important for efficient large-scale learning. In LLM training, adaptive first-order, coordinate-wise methods such as Adam [ 26 ] and AdamW [ 32 ] remain the de facto choice. More recently, matrix-based optimizers such as Shampoo [ 20 ] , SOAP [ 56 ] , and Scion [ 37 ] have received growing interest as alternatives that exploit matrix structure in gradient updates. Among these optimizers, Muon [ 24 ] , which approximately orthogonalizes the momentum matrix using a Newton–Schulz iteration, has shown promising empirical performance in LLM pretraining [ 30 ] , and recent optimizer benchmarks further identify Muon, along with matrix-based methods more broadly, as a strong alternative to AdamW [ 44 , 60 ] . A second long-standing thread in optimizer design is the role of momentum. Classical analyses establish accelerated rates for momentum on smooth and strongly convex problems [ 38 , 35 , 51 ] . 11 prove that momentum eliminates the need for large batches in normalized SGD, which is the vector-side analog of Muon’s polar update in the sense that both retain only the direction of the gradient. The practical picture in deep learning is, however, more nuanced. 57 show that at small learning rates and high gradient noise, momentum offers only marginal benefit, suggesting that its provable acceleration sits in the deterministic (or large-learning-rate) regime rather than in stochastic noise reduction per se. More recent work [ 29 ] reframes momentum in the frequency domain as a low-pass filter on the gradient stream, suggesting that the relevant mechanism may be filtering rather than acceleration. Despite these developments, neither Muon’s strong empirical performance nor the role momentum plays inside it is well understood. In practice, Muon’s pretraining performance is closely tied to momentum — see figure 1 , where we see a clear trend that Polar-only pipeline (i.e., Muon without momentum) underperforms in pretraining — yet existing theory has not successfully characterized the role of momentum inside Muon. Substantial recent work analyzes Muon-style polar updates from the aspects of its spectral property, implicit bias, and feature learning, but without momentum [ 33 , 50 , 12 , 28 , 58 , 25 ] . By construction, these analyses cannot explain momentum’s benefit. A thorough review of these works is deferred to Appendix A . More recently, 9 establish that Muon converges to the spectrally-constrained minimizer, though its convergence rate deteriorates with heavier momentum. 27 shows convergence of Muon to a stationary point but heavier momentum brings no further improvement. 46 analyze Muon under an inexact LMO that models the Newton–Schulz approximation error, yet its theoretical bound deteriorates in the heavy-momentum regime. 15 establish implicit bias of Muon with momentum, but still its convergence rate blows up with too heavy momentum. On the whole, theory tells us that heavy momentum brings no benefit — and may even degrade convergence — inside Muon, but this obviously contradicts what we observe in practice. (a) NanoGPT. (b) LLaMA 350M. Figure 1 : End-to-end validation loss comparisons across (a) NanoGPT training and (b) LLaMA 350M training. The Muon Pre-polar pipeline outperforms Post-polar and Polar-only pipelines. The full experimental settings are in Section F.4 . The relationship between Muon’s polar update and momentum is more nuanced. Whereas Orthogonal-SGDM [ 53 ] , proposed prior to Muon, orthogonalizes each per-step gradient before momentum smoothing, Muon applies the polar update after momentum smoothing, significantly outperforming particularly on language model pretraining [ 24 ] . Therefore, Muon’s success encourages us to closely look at (i) why momentum smoothing matters practically, and (ii) why the polar factor should be put after momentum smoothing. To this end, we compare three Muon pipelines for the weight update \Delta W_{t} at iteration t , each combining a momentum buffer M_{t}=\beta M_{t-1}+(1-\beta)G_{t} and the polar factor \cO(X)=UV^{\top} (from the thin SVD X=U\Sigma V^{\top} ) on the gradient stream \{G_{t}\} in different orders: Pre-polar: \displaystyle\quad\Delta W_{t}\propto\cO(M_{t}), (1) Post-polar: \displaystyle\quad\widetilde{M}_{t}=\beta\widetilde{M}_{t-1}+(1-\beta)\cO(G_{t}),\quad\Delta W_{t}\propto\widetilde{M}_{t},\quad\text{and} (2) Polar-only: \displaystyle\quad\Delta W_{t}\propto\cO(G_{t}). (3) Pre-polar update equation 1 is the standard Muon update [ 24 ] . Post-polar update equation 2 formalizes the Orthogonal-SGDM variant. Polar-only update equation 3 removes momentum entirely as a no-momentum baseline. Figure 1 shows that Pre-polar dominates both Post-polar and Polar-only throughout training. Two conclusions follow. First, Pre-polar and Post-polar pipelines use the same two operations in opposite orders, so any gap between them must come from the ordering itself, because momentum and orthogonalization do not commute as numerical operations. Second, the gap from Pre-polar to Polar-only shows that momentum plays a role that orthogonalization alone cannot fulfill: momentum is an essential component of Muon, not a cosmetic acceleration trick. This raises the central question of our interest: What role does momentum play on the gradient stream, and why does placing it before orthogonalization matter inside Muon? This paper supplies the missing puzzle of Muon theory. Momentum first acts as a spectral filter on the gradient stream. Inside Muon, this filtering must happen before orthogonalization. The polar factor preserves singular directions but replaces all singular values with ones, so when it acts directly on a noisy gradient it erases the amplitude gap between signal and noise directions, and we shall never distinguish signal and noise even after undergoing the subsequent momentum. When momentum acts first, it filters the gradient stream to preserve the persistent signal component and attenuate the perturbation, so the polar factor operates on a matrix whose singular-value structure already separates signal from noise. The guiding principle is simple: denoise first, then orthogonalize . Contributions. 1. Spectral gap and singular-subspace reliability ( Theorem 1 and Corollary 1 ). Under a structured signal-plus-perturbation gradient model with a coherent signal, a bounded variance mean-zero orthogonal (BVMZOS) perturbation, and the signal persistence assumption, momentum solely opens a spectral gap between the r -th and (r{+}1) -th singular values that grows with the momentum window size T , and aligns the top- r singular subspaces