Beyond the Ideal: Analyzing the Inexact Muon Update

Egor Shulgin Sultan Alrashed Francesco Orabona Peter Richtárik · 2025

Read Beyond the Ideal: Analyzing the Inexact Muon Update by Egor Shulgin Sultan Alrashed Francesco Orabona Peter Richtárik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19933.txt.

Introduction

Egor Shulgin and three co-authors supply the missing theory for the Muon optimizer as it is actually used in training large neural networks. Muon's update depends on orthogonalizing a matrix, which practical implementations approximate quickly, for instance with a few Newton-Schulz steps; all earlier theory assumed the exact, intractable version. The authors analyze the inexact update inside a general oracle-based optimization framework.

Their error model gives explicit bounds on how performance degrades as the orthogonalization gets sloppier. A practical coupling emerges: lower precision demands a smaller step size but larger momentum, so the approximation budget must be tuned together with the learning schedule. Experiments training a small GPT model confirm the predicted shifts in optimal learning rate.

Catalog record

CREATOREgor Shulgin Sultan Alrashed Francesco Orabona Peter Richtárik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Ideal: Analyzing the Inexact Muon Update Egor Shulgin Sultan Alrashed Francesco Orabona Peter Richtárik Abstract The Muon optimizer has rapidly emerged as a powerful, geometry-aware alternative to AdamW , demonstrating strong performance in large-scale training of neural networks. However, a critical theory-practice disconnect exists: Muon ’s efficiency relies on fast, approximate orthogonalization, yet all prior theoretical work analyzes an idealized, computationally intractable version assuming exact SVD-based updates. This work moves beyond the ideal by providing the first analysis of the inexact orthogonalized update at Muon ’s core. We develop our analysis within the general framework of Linear Minimization Oracle (LMO)-based optimization, introducing a realistic additive error model to capture the inexactness of practical approximation schemes. Our analysis yields explicit bounds that quantify performance degradation as a function of the LMO inexactness/error, δ \delta . We reveal a fundamental coupling between this inexactness and the optimal step size and momentum: lower oracle precision requires a smaller step size but larger momentum parameter. These findings elevate the approximation procedure (e.g., the number of Newton-Schulz steps) from an implementation detail to a critical parameter that must be co-tuned with the learning schedule. NanoGPT experiments directly confirm the predicted coupling, with optimal learning rates clearly shifting as approximation precision changes. 1 Introduction For over a decade, the landscape of deep learning optimization has been dominated by adaptive first-order methods, with AdamW ( Kingma and Ba 2015 ; Loshchilov and Hutter 2019 ) serving as the de facto standard for training large and complex neural networks. Its robustness and general effectiveness have powered progress across numerous domains. Yet a new class of geometry-aware optimizers has recently emerged, challenging this paradigm. Among them, Muon ( Jordan et al. 2024b ) has quickly gained prominence as a successor to AdamW . By leveraging matrix structure in neural network parameters, Muon has demonstrated superior performance and scalability, setting new training speed records for models like nanoGPT ( Jordan et al. 2024a ) and enabling the efficient training of state-of-the-art Large Language Models (LLMs), such as Kimi from Moonshot AI ( Bai et al. 2025 ) . Benchmarking studies consistently show that Muon can be significantly more computationally efficient than AdamW ( Liu et al. 2025 ; Shah et al. 2025 ; Wen et al. 2025 ) . However, a critical disconnect lies at the heart of Muon ’s success. The practical efficiency of the optimizer is entirely predicated on its use of fast, approximate orthogonalization methods, most notably the Newton-Schulz iteration ( Jordan et al. 2024b ) . This iterative matrix-multiplication-based procedure provides a computationally cheap way to approximate the orthogonal factor of a matrix’s polar decomposition, avoiding a full Singular Value Decomposition (SVD) which would be prohibitively expensive ( Grishina et al. 2025 ) . Yet, a significant gap exists between this practical implementation and its theoretical understanding. All prior theoretical analyses of Muon ( Li and Hong 2025 ; Shen et al. 2025 ; Chen et al. 2025 , e.g.,) have studied an idealized , computationally intractable version of the algorithm. In fact, these studies assume access to an exact orthogonalization oracle that computes the perfect SVD-based update. Hence, the existing theory describes an algorithm that is never actually used in practice, leaving the real-world performance of Muon potentially unexplained. This work moves beyond the ideal to provide the first theoretical analysis of the inexact Muon update. We situate our analysis within the general framework of Linear Minimization Oracle (LMO)-based optimization ( Pethick et al. 2025b ) , also known as the Frank-Wolfe framework ( Frank and Wolfe 1956 ; Hazan 2008 ; Clarkson 2010 ; Jaggi 2013 ) . This perspective frames the core Muon operation as an LMO call over the unit ball with respect to the spectral norm. To account for the realities of practical computation, we introduce a realistic additive error model. This model is directly motivated by the behavior of practical approximation schemes like Newton-Schulz , allowing us to capture the inexactness inherent in any efficient implementation. By analyzing the algorithm under this inexact LMO, we bridge the critical gap between theory and practice. 1.1 Contributions Our main contributions to the theory and practice of LMO-based optimizers are: \bullet The first analysis of the practical (inexact) Muon update. We present the first formal convergence analysis for LMO-based methods with an inexact oracle under a realistic additive error model (Assumption 1 ). Our general framework provides the first theoretical guarantees for the implemented Muon update, moving beyond prior works that exclusively analyzed an idealized, intractable version of the algorithm. \bullet Uncovering a fundamental hyperparameter coupling. Our analysis reveals that the LMO inexactness, \delta , is a critical parameter that alters the optimization dynamics. We derive explicit formulas for the step size ( \gamma^{*}\propto(1+\delta)^{-1/4} ) and momentum ( \alpha^{*}\propto\sqrt{1+\delta} ) in the stochastic setting (Corollary 3 ), uncovering a crucial coupling: a less precise LMO (larger \delta ) requires a smaller learning rate but a larger momentum parameter. This elevates the approximation quality from an implementation detail to a core hyperparameter. \bullet Comprehensive theoretical framework. We establish convergence rates for both deterministic and stochastic settings, with extensions to the (L^{0},L^{1}) -smoothness model ( Zhang et al. 2020 ) and a layer-wise setting ( Riabinin et al. 2025 ) . These results tightly generalize prior work, exactly recovering the rates for exact LMOs when the inexactness is set to zero. \bullet Empirical validation. We validate our key theoretical predictions with experiments on CIFAR-10 and NanoGPT. We confirm that performance degrades as LMO precision decreases and, crucially, our experiments with NanoGPT (Figure 1(b) ) provide clear empirical evidence for the predicted hyperparameter coupling, showing that the optimal learning rate shifts to a lower value when a less precise LMO is used. 1.2 Related work We survey the key developments toward a theoretical understanding of Muon . The conceptual basis for Muon ’s orthogonalized update was introduced by Jordan et al. 2024b and theoretically motivated by Bernstein and Newhouse 2024 , who showed that the preconditioned update of a simplified Shampoo optimizer is equivalent to steepest descent under the spectral norm. The first formal convergence analyses for Muon sought to connect it to existing theoretical frameworks. Notably, Li and Hong 2025 provided an initial convergence guarantee by viewing the Muon update as a matrix-based generalization of normalized SGD with momentum, building upon the analysis of Cutkosky and Mehta 2020 . Following these initial results, two powerful and unifying frameworks emerged concurrently, providing a more general perspective. Pethick et al. 2025b introduced the Scion framework, which situates Muon within the broader class of methods based on the Linear Minimization Oracle (LMO), a core component of the Frank-Wolfe algorithm ( Frank and Wolfe 1956 ) . Independently, Kovalev 2025 developed a non-Euclidean trust-region interpretation, also for arbitrary norms. Both frameworks successfully recover the idealized Muon update as a special case when the spectral norm is chosen, and provided the first general convergence guarantees for this class of optimizers. An idealized trust-region method, called the ball-proximal (“broximal”) point method ( BPM ) was concurrently developed by Gruntkowska et al. 2025b . BPM has remarkable theoretical guarantees for convex optimization—it converges in finitely many steps, and has linear

Read the complete text