The Newton–Muon Optimizer
Zhehang Du, Weijie Su · 2026
Read The Newton–Muon Optimizer by Zhehang Du, Weijie Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01472.txt.
Introduction
Du and Su introduce Newton-Muon, an optimizer derived from a surrogate approximating the loss as a quadratic function of the perturbation to a weight matrix. Minimizing this surrogate in one step under an isotropic weights assumption yields an update rule. Standard Muon can be read as a Newton-type method neglecting the right preconditioning from the input second moment. On a Modded-NanoGPT speedrun reproduction using Muon for GPT-2 pretraining, Newton-Muon reaches the loss in 6 percent fewer iteration steps. It reduces wall-clock training time by 4 percent.
Catalog record
| CREATOR | Zhehang Du, Weijie Su |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101545 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Newton–Muon Optimizer Zhehang Du, Weijie Su Abstract The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains largely elusive. In this paper, we introduce a surrogate model that not only sheds new light on the design of Muon, but more importantly leads to a new optimizer. In the same spirit as the derivation of Newton’s method, the surrogate approximates the loss as a quadratic function of the perturbation to a weight matrix W W using only three matrices: the gradient G G , an output-space curvature matrix H H , and the data matrix Z Z that stacks the layer inputs. By minimizing this surrogate in one step and adopting a certain isotropic assumption on the weights, we obtain the closed-form update rule (up to momentum and weight decay) W ← W − η ⋅ msgn ( G ( Z Z ⊤ ) − 1 ) , W\leftarrow W-\eta\cdot\mathrm{msgn}(G(ZZ^{\top})^{-1}), where η \eta is the learning rate and msgn ( X ) = U V ⊤ \mathrm{msgn}(X)=UV^{\top} if X = U S V ⊤ X=USV^{\top} is a compact singular value decomposition. This new optimization method, which we refer to as Newton–Muon, shows that standard Muon can be interpreted as an implicit Newton-type method that neglects the right preconditioning induced by the input second moment. Empirically, on a reproduction of the earliest publicly released Modded-NanoGPT speedrun configuration using Muon for GPT-2 pretraining, Newton–Muon reaches the target validation loss in 6% fewer iteration steps and reduces wall-clock training time by about 4%. \NoHyper † † footnotetext: Emails: {duz,suw}@wharton.upenn.edu . \endNoHyper \NoHyper † † footnotetext: Code is available at https://github.com/zhehangdu/Newton-Muon . \endNoHyper 1 Introduction Since early 2025, there has been a surge of interest in matrix-structured optimization methods for training deep neural networks and large language models (LLMs). A prominent optimizer at the center of this flurry of research activity is Muon ( 23 ) , closely related to spectral gradient descent ( 6 ) . Specifically, letting f({W}) be the loss function and {W} be a layer weight matrix, we consider the optimization problem \min_{W\in\mathbb{R}^{m\times n}}f(W). Let {G}\coloneqq\nabla_{{W}}f({W})\in\mathbb{R}^{m\times n} be the gradient matrix or, in practice, an approximation computed from a mini-batch. Writing the matrix sign \mathrm{msgn}({G})\coloneqq{U}{V}^{\top} if {G}={U}{S}{V}^{\top} is a compact singular value decomposition (SVD), Muon updates the weight matrix as 1 1 1 In practice, Muon approximates \mathrm{msgn}({G}) using Newton–Schulz iterations and momentum is applied to the gradient. W\leftarrow W-\eta\cdot\mathrm{msgn}({G}) for some learning rate \eta . Compared to AdamW ( 24 ; 30 ) , Muon has been reported to offer a faster convergence rate and lower wall-clock time to reach the same level of loss across a broad range of model sizes ( 29 ; 41 ; 49 ) . Furthermore, it has been used to train state-of-the-art open-source models ( 46 ; 55 ) , and many extensions of Muon have been proposed ( 28 ; 37 ; 2 ; 21 ; 51 ; 38 ; 19 ) . While matrix-based optimizers have demonstrated highly effective empirical performance, the theoretical mechanisms underlying Muon remain largely mysterious. For instance, it is natural to ask why preserving the matrix structure of the gradient is beneficial and why simply discarding its singular values is empirically effective. This is, however, not entirely surprising, as the nonconvex nature of deep learning optimization makes it notoriously difficult to analyze. Given that a rigorous theoretical understanding remains out of reach, a practical approach is to establish an intuitive yet principled framework for designing deep learning optimizers ( 4 ; 37 ; 26 ; 17 ) . Among these, 45 introduced the isotropic curvature model as a surrogate for approximating the loss function by assuming isotropic curvature for preconditioning and isotropic input activations. A one-step descent analysis applied to this model suggests that the optimal update direction naturally preserves the matrix subspace of the gradient, thereby partially justifying gradient orthogonalization. However, the unspecified curvature function in the isotropic curvature model limits its utility for deriving practical optimization methods for LLM training. In this paper, we address this challenge by introducing a more tractable surrogate model for understanding these optimizers and, more importantly, for proposing a new method. Taking Q\in\mathbb{R}^{m\times n} to be a potential update direction and letting H denote a curvature matrix, 2 2 2 Here, the curvature matrix H is not the full parameter-space Hessian with respect to the vectorized weights. In particular, the curvature matrix H has size m\times m . this paper introduces a surrogate model of the form f(W-Q)-f(W)\approx-\mathrm{tr}({Q}{G}^{\top})+\frac{1}{2N}\mathrm{tr}\Big({H}{Q}({Z}{Z}^{\top}){Q}^{\top}\Big), (1) where Z=[\boldsymbol{z}_{1},\ldots,\boldsymbol{z}_{N}] denotes the collection of all activation inputs from the N training data points to the layer W . Because it involves three components—namely G , H , and Z —this model is referred to as the triplet quadratic surrogate model . By design, the gradient matrix G and the update direction Q naturally retain their matrix forms. The linear term -\mathrm{tr}({Q}{G}^{\top}) captures the first-order approximation, as in the isotropic curvature model, while the quadratic term \frac{1}{2N}\mathrm{tr}\Big({H}{Q}({Z}{Z}^{\top}){Q}^{\top}\Big) is considerably simpler than the curvature function in the isotropic curvature model. Notably, this quadratic term is the matrix-form expression induced by the Kronecker-factored curvature approximation in K-FAC ( 32 ) . This triplet quadratic surrogate for approximating f(W-Q)-f(W) is essentially of the form “ -\textit{linear gradient term}+\textit{quadratic curvature term} ”, which closely resembles the derivation of Newton’s method. As such, this surrogate model serves as a means of deriving a Newton-type method in a manner that fully leverages the matrix structure of the gradient via a one-step descent analysis; that is, by minimizing the right-hand side of ( 1 ) over Q . In fact, although second-order methods such as Newton’s method are rarely used in training large-scale neural networks, recent work demonstrates a 3 – 5\times iteration speedup when using the Gauss–Newton method compared to common optimizers in deep learning ( 1 ) . Thus, there is strong motivation to develop an implicit Newton-type method for LLM training, provided that the per-iteration computational cost remains roughly comparable to that of AdamW or Muon. Despite the ease of minimizing the triplet quadratic surrogate model, a key difficulty arises because the optimal solution involves the unknown curvature matrix H . To circumvent this, we make an assumption on the displacement W-W^{\star} , which is the difference between the current weight matrix W and an optimal weight matrix W^{\star} . 3 3 3 Since there are generally exponentially many optimal weight matrices, we simply consider one that is locally closest to W . We assume that this displacement is isotropic, in the sense that no direction is favored over another on average. Surprisingly, under this least-informative assumption, we obtain a closed-form expression for the Newton-type update: Q^{\star}\propto\mathrm{msgn}(G(ZZ^{\top})^{-1}) . Accordingly, our new method updates the weight matrix according to the rule 4 4 4 In practice, momentum and other common implementation tricks are also used. W\leftarrow W-\eta\cdot\mathrm{msgn}(G(ZZ^{\top})^{-1}). This method differs from standard Muon in that the gradient matrix G is right-preconditioned by the inverse second moment of the input data. This can be interpreted as incorporating the geometry of the data distribution to