AMUSE: Anytime Muon with Stable Gradient Evaluation
Jueun Kim, Baekrok Shin · 2026
Read AMUSE: Anytime Muon with Stable Gradient Evaluation by Jueun Kim, Baekrok Shin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22432.txt.
Introduction
Kim and colleagues study Muon through the river-valley landscape, where progress runs along a flat river subspace while steep valley walls cause oscillations. Muon's orthogonalisation speeds river progress but amplifies dominant-direction noise. Their AMUSE blends Muon's bulk speed with Schedule-Free averaging for stability. A shifting coefficient starts near the fast Muon sequence. It then shifts toward the stable averaged sequence to damp oscillations. AMUSE needs no schedules and supports anytime training. Across vision and language pretraining, it beats Schedule-Free AdamW and Muon.
Catalog record
| CREATOR | Jueun Kim, Baekrok Shin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82346 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22432
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AMUSE: Anytime Muon with Stable Gradient Evaluation Jueun Kim, Baekrok Shin Abstract Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters. Despite Muon’s strong empirical performance, its underlying mechanism remains partially understood. We study Muon through the river-valley loss landscape, where useful training progress occurs along a flat, low-curvature bulk subspace (the river), while high-curvature dominant directions form steep valley walls that induce oscillations. We empirically show that while Muon’s orthogonalization accelerates river progress by increasing the bulk component, it also amplifies dominant-direction noise, causing oscillatory trajectories. Building on this, we propose Anytime MUon with Stable gradient Evaluation (AMUSE), which integrates Muon’s rapid bulk progress with the stabilizing effect of Schedule-Free averaging. AMUSE uses a time-varying interpolation coefficient that initially evaluates gradients near the fast Muon sequence for rapid adaptation, then gradually shifts toward the stable averaged sequence to suppress valley-wall oscillations. As a result, AMUSE requires no learning rate schedules and supports anytime training. Across vision tasks and large language model pretraining, AMUSE consistently improves the performance-iteration Pareto frontier over (Schedule-Free) AdamW and Muon. https://github.com/kjeiun/amuse 1 Introduction Optimization remains a primary lever for improving training efficiency in deep learning. For many modern architectures, especially Transformer-based language models, the standard recipe has long combined the Adam(W) optimizer ( 22 ; 27 ) with warmup and a prescribed learning rate decay schedule such as cosine decay ( 26 ) . Recent work challenges both components of this recipe. Schedule-Free AdamW ( 10 ) removes the need for explicit learning rate schedules by evaluating gradients at interpolated points between the current and averaged iterates, while using the averaged iterate for inference. It has shown strong empirical performance relative to schedule-based baselines, including winning the Self-Tuning track of the 2024 AlgoPerf Challenge ( 7 ; 20 ) . A complementary advance changes the underlying geometry of the update itself. Muon ( 18 ) exploits matrix structure by applying momentum to matrix-valued parameters and orthogonalizing the resulting update direction. Muon and its variants have been reported to outperform AdamW in language model pretraining ( 24 ) , and have recently been adopted in large-scale open-model training recipes, including Kimi-K2 ( 21 ) , GLM-5 ( 13 ) , and DeepSeek-V4 ( 9 ) . Despite this empirical success, why orthogonalized momentum improves optimization remains only partially understood. We take this question as the first step toward optimizer design, and analyze Muon from the perspective of loss landscape geometry. Deep learning optimization takes place in high-dimensional parameter spaces whose loss landscapes are highly anisotropic. Empirically, the Hessian spectrum often contains a small number of large outlier eigenvalues and a much larger bulk of relatively small eigenvalues. This spectral structure induces a local decomposition of the parameter space into a low-dimensional, high-curvature dominant subspace and a high-dimensional, low-curvature bulk subspace ( 42 ; 43 ; 33 ; 34 ; 35 ; 12 ) . Recent work casts this anisotropy as the river-valley landscape ( 46 ; 52 ) : the dominant subspace forms steep valley walls, while the bulk subspace forms a relatively flat river. In this picture, training progress occurs mainly along the river, whereas dominant subspace components cause valley-wall oscillations. This suggests two core principles for optimizer design: (i) take large, consistent steps along the river and (ii) suppress unnecessary updates across the valley. We find that Muon naturally promotes the first part of this principle. By orthogonalizing the momentum matrix, Muon prevents the update from being dominated by a few large singular components, producing a more balanced matrix-valued step. Empirically, Muon updates contain substantially larger bulk components than SGD or Adam. Moreover, orthogonalization itself explicitly increases this relative bulk component. This provides a geometric explanation for Muon’s efficiency: orthogonalized momentum accelerates progress along the river directions . However, this benefit comes with a tradeoff. Because orthogonalization is not selective, it can amplify noisy components as well as useful ones. In particular, small noise aligned with high-curvature dominant directions can become oscillations across the valley walls after orthogonalization, leading to less stable trajectories ( 38 ; 56 ; 16 ) . Schedule-Free (SF) optimization offers a complementary stabilizing mechanism by evaluating gradients at an interpolation between the fast optimizer sequence and an averaged inference sequence. 47 show that this mechanism can keep the inference trajectory close to the river without requiring learning rate decay. We find this particularly beneficial for Muon: evaluating gradients closer to the averaged sequence reduces the high-curvature component before orthogonalization, thereby suppressing oscillations between the valley walls and stabilizing the resulting trajectory. Motivated by this complementarity, we propose AMUSE (Anytime MUon with Stable gradient Evaluation). AMUSE combines Muon’s orthogonalized matrix updates with the SF formulation, using a time-varying interpolation coefficient. AMUSE initially evaluates gradients near the fast Muon sequence for rapid adaptation, then gradually shifts toward the averaged sequence, where gradients are less dominated by high-curvature components. Because this schedule is independent of the total training horizon and requires no learning rate decay, AMUSE naturally supports anytime training. We evaluate AMUSE across a diverse set of benchmarks spanning image classification, segmentation, and large language model pretraining. The results show that AMUSE consistently improves the performance-iteration Pareto frontier over (Schedule-Free) AdamW and Muon. Notably, AMUSE reaches Muon’s final performance with 1.51\times , 1.12\times , and 3.08\times fewer steps on 720M LLM pretraining, ImageNet with ResNet-50, and ImageNet with ViT fine-tuning, respectively. 2 Related Work 2.1 River-Valley Loss Landscape Previous studies have characterized neural network training through a low-rank Hessian structure, consisting of a few high-curvature directions and a much broader bulk component ( 12 ; 33 ; 34 ; 35 ; 42 ; 43 ; 54 ) . Recent work further interprets this geometry through the river-valley landscape (see Figure 1 for illustration), where dominant directions form steep valley walls while the bulk defines the flatter river floor where most useful progress occurs ( 46 ; 52 ; 2 ; 5 ; 58 ; 51 ; 50 ; 11 ) . 52 use this geometry to explain the success of Warmup-Stable-Decay (WSD) schedules in large language model pretraining: the stable phase facilitates rapid traversal along the river, while the decay phase helps the iterate settle onto the valley floor. Extending this perspective, 2 show that a similar geometry also arises in convolutional networks trained on image classification tasks. 46 show that projecting updates only onto the dominant subspace is insufficient to reduce training loss, while updates projected onto the bulk subspace remain effective. Recently, several works have leveraged the river-valley landscape to design optimizers that either amplify updates along river directions or stabilize oscillations along valley directions ( 59 ; 57 ; 25 ; 28 ) . 2.2 Schedule-Free Optimizer 10 introduce the schedule-free (SF) optimizer as an interpolation between Polyak–Ruppert (PR) averaging ( 40 ; 37 ) and primal averaging ( 31 ; 48 ) . Its update is given by \displaystyle{\bm{y}}_{t} \displaystyle=(1-\beta){\bm{z}}_{t}+\beta{\bm{x}}_{t}, (1) \displaystyle{\bm{z}}_{t+1} \displaystyle={\bm{z}}_{t}-\eta\Delta_{t}, \displaystyle{\bm{x}}_{t+1} \displaystyle=\left(1-c_{t+1}\right){\bm{x}}_{t}+c_{t+1}{\bm{z}}_{t+1}, where \eta is the learning rate, \Delta_{t} is the base optimizer update computed from the gradient at {\bm{y}}_{t} , c_{t+1}=1/(t+1) , and the