Practical Efficiency of Muon for Pretraining
Essential AI · 2025
Read Practical Efficiency of Muon for Pretraining by Essential AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02222.txt.
Introduction
Researchers at Essential AI test a leaner training optimizer for large models in this 2025 paper. Muon is a simple second-order method meant to rival the standard AdamW. Training giant models is costly, so each gain in efficiency matters. The team compared the two across model sizes up to four billion parameters.
Muon stretched the frontier of quality against training time. It kept its data efficiency at batch sizes far beyond where AdamW fades. Paired with a scaling rule for settings, it let tuning transfer across model sizes. A telescoping fix handled the rule's error sources at small extra cost. Data and architecture checks backed the main findings.
Catalog record
| CREATOR | Essential AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02222
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Practical Efficiency of Muon for Pretraining Essential AI Abstract We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon is more effective than AdamW in retaining data efficiency at large batch sizes, far beyond the so-called critical batch size, while remaining computationally efficient, thus enabling more economical training. We study the combination of Muon and the maximal update parameterization (muP) for efficient hyperparameter transfer and present a simple telescoping algorithm that accounts for all sources of error in muP while introducing only a modest overhead in resources. We validate our findings through extensive experiments with model sizes up to four billion parameters and ablations on the data distribution and architecture. Figure 1: Muon ( Jordan et al., 2024 ) explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff by retaining data efficiency at large batch sizes. 1 Introduction Adam ( Kingma and Ba, 2014 ) and AdamW ( Loshchilov and Hutter, 2017 ) have empirically dominated the landscape of neural network optimizers in recent years. Superior generalization from the same training data (data-efficiency) at marginal increase in compute cost enabled them to gradually replace other first-order incumbents. By exploiting the statistical dependencies between model parameters, second-order optimizers have the potential to challenge the status quo. Recent second-order optimizers ( Jordan et al., 2024 ; Vyas et al., 2024 ) indeed promise data-efficiency over AdamW for pretraining, without compromising the overall computational advantage. However, these efforts rely on wall-clock time ( MLCommons, 2025 ) or FLOPs to abstract away from algorithmic complexity and implementation details. But, this is insufficient to conclude their advantage over AdamW. Pretraining workloads come in various model shapes, data sizes, and resource affordances. To fully explain the practicality of an optimizer one needs to capture the fundamental tradeoff between compute and time resources—the ability to reduce total training time by using more devices. Since this trade-off is what makes large-scale pretraining feasible, it is crucial to characterize how it changes when comparing optimizers for pretraining. A key element in the compute-time tradeoff is the batch size. We may process data faster by increasing the batch size (i.e., by increasing the number of devices in distributed training) but likely at the cost of data efficiency - the loss reduction the model achieves per token. A “good” optimizer should therefore support large batch sizes to facilitate faster training while remaining persistently data-efficient. Simultaneously, it must remain computationally efficient by maintaining or improving FLOP efficiency and hardware utilization, ideally with any reasonable implementation. How can we take into account all these considerations to show that a second-order optimizer yields a more favorable compute-time tradeoff over AdamW? To answer this question, we directly plot the compute and time resources required to achieve the same target loss using different optimizers. On this two-dimensional plane, optimizers are represented as iso-loss frontiers that marginalize over all other variables affecting the final tradeoff, thus admitting effective comparison. To study the tradeoff, we focus on Muon ( Jordan et al., 2024 ) , the simplest instantiation of a second-order optimizer. We build on the recent work by Liu et al. (2025) , which shows that Muon is more FLOP-efficient than AdamW, and make the stronger claim that Muon expands the Pareto frontier on the compute-time tradeoff with variable batch sizes, thereby strictly increasing the practitioner’s flexibility in resource allocation (Figure 1 ). To analyze this behavior, we present a study of the relative data efficiency of Muon over AdamW in the large batch size regime. In the second part of the paper, we address another core challenge in optimization for pretraining: how to choose optimal hyperparameters. We focus on the maximal update parameterization (muP) ( Yang et al., 2022 ) , a principled approach to hyperparameter transfer. It stipulates certain weight initialization and learning rate scaling such that optimal hyperparameters identified on a small model remain effective on a large one, reducing the burden of hyperparameter search at a large model scale. We demonstrate successful hyperparameter transfer with muP under Muon, extending known results under AdamW. While muP is asymptotically correct in the limit on the network width and search granularity, in practice it suffers from estimation errors and requires multi-scale tuning over different model sizes. We develop a simple “telescoping” algorithm for training a final model of cost C and width N that adjusts the search grid across model scales to account for these estimation errors with an additional modest O(C\log{(N)}) compute cost (Figure 6 ). Our paper serves as a practitioner’s guide on optimization for pretraining. Our final recommendation is to choose Muon over AdamW because it increases flexibility in resource allocation by remaining data-efficient with large batch sizes, and to combine it with muP for compute-efficient hyperparameter search at scale. We back our findings through extensive experiments, varying the model size up to 4 billion parameters and batch size up to 16 million tokens, ablating the effect of the data distribution and architecture. 1 1 1 All our experimental artifacts will be released at: https://huggingface.co/EssentialAI . 2 Muon Improves the Compute-Time Tradeoff 2.1 Review of Muon A transformer language model primarily consists of matrix weights W\in\mathbb{R}^{m\times n} parameterizing the feedforward and attention layers. We assume m\leq n without loss of generality. A weight update takes the form of W_{t+1}=W_{t}-\eta_{t}O_{t} where O_{t}\in\mathbb{R}^{m\times n} is some transformation of the gradient G_{t}\in\mathbb{R}^{m\times n} on the t -th batch. Muon can be seen as matrix-structured steepest descent with spectral norm regularization: \displaystyle O_{t}=\argmin_{O\in\mathbb{R}^{m\times n}:\;\left|\left|O\right|\right|_{2}\leq 1}\mathrm{tr}\left(G_{t}^{\top}O\right) It is easy to show that O_{t}=UV^{\top} is an optimal solution where G_{t}=U\Sigma V^{\top} is a singular value decomposition (SVD) ( Bernstein and Newhouse, 2024 ) . Muon avoids explicit computation of SVD by Newton-Schulz iteration ( Kovarik, 1970 ; Björck and Bowie, 1971 ) . In practice, we combine Muon with Nesterov momentum, learning rate scaling, and (coupled) weight decay. Thus the only state Muon maintains is the first moment M_{t}\in\mathbb{R}^{m\times n} initialized to zero. In the t -th update, given the gradient G_{t} we compute \displaystyle M_{t} \displaystyle=G_{t}+\beta M_{t-1} \displaystyle O_{t} \displaystyle=\textbf{NewtonSchulz}(G_{t}+\beta M_{t}) \displaystyle W_{t+1} \displaystyle=W_{t}-\eta_{t}((0.2\sqrt{n})O_{t}+\lambda W_{t}) (1) where \beta,\lambda\in\mathbb{R} are momentum and weight decay hyperparameters. The constant 0.2\sqrt{n} , proposed by Liu et al. (2025) , normalizes the Muon update to have a similar RMS value as the AdamW update. 2 2 2 0.2 is the empirically observed RMS of a typical AdamW update. 1/\sqrt{n} is the RMS of a perfectly normalized O_{t} , which follows immediately from the fact that the squared Frobenius norm is the sum of squared singular values. This ensures that the same learning rate schedule and weight decay work well for both Muon and AdamW, which is useful because AdamW is used to update embedding and normalization parameters. Jordan et al. (2024) provide well-tuned hyperparameters for Newton-Schulz that rapidly flatten all singular values to range (0.7, 1.3). 2.1.1 Why we focus on Muon While Muon is not the only viable second-order optimizer, we limit our scope to Muon for the following reasons. First, it has the simplest derivation and implementation, allowing us to focus on characterizing the practical benefits of second-order methods. Second, we can reduce