Ringmaster LMO : Asynchronous Linear Minimization Oracle Momentum Method

Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, Peter Richtárik · 2026

Read Ringmaster LMO : Asynchronous Linear Minimization Oracle Momentum Method by Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, Peter Richtárik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18174.txt.

Introduction

Abdurakhmon Sadiev's 2026 paper presents Ringmaster LMO, an asynchronous momentum method built on linear minimization oracles for neural network training. Muon has emerged as a strong AdamW alternative, but oracle methods usually run in lockstep, forcing fast workers to wait for slow ones in heterogeneous clusters. Ringmaster LMO borrows delay thresholding from Ringmaster ASGD, discarding overly stale gradients to keep time complexity optimal. The authors prove convergence under generalized smoothness and add a parameter-agnostic variant with shrinking steps and adaptive delay limits. Its edge grows with system heterogeneity.

Catalog record

CREATORAbdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, Peter Richtárik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117314
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ringmaster LMO : Asynchronous Linear Minimization Oracle Momentum Method Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, Peter Richtárik Abstract Muon has recently emerged as a strong alternative to AdamW for training neural networks, with encouraging large-scale pretraining results and growing evidence that matrix-structured updates can be faster in practice. Yet Muon, and more generally Linear Minimization Oracle (LMO) based methods, are typically used synchronously. This is problematic in heterogeneous distributed systems, where workers complete gradient computations at different speeds and synchronous training must repeatedly wait for slower workers. In this work, we introduce Ringmaster LMO , an asynchronous LMO-based momentum method for unconstrained stochastic nonconvex optimization. Our method builds on the delay-thresholding idea of Ringmaster ASGD . For SGD -type methods, Ringmaster ASGD achieves optimal time complexity by discarding overly stale gradients. Ringmaster LMO extends this mechanism to general LMO-based updates. We establish convergence guarantees under generalized ( L 0 , L 1 ) (L_{0},L_{1}) -smoothness and further develop a parameter-agnostic variant with decreasing stepsizes and adaptive delay thresholds. Finally, we translate our iteration guarantees into time complexity bounds under heterogeneous worker computation times. In the classical Euclidean smooth setting, these bounds recover the optimal time complexity of Ringmaster ASGD . Experiments on stochastic quadratic problems and NanoChat language-model pretraining show that the advantages of Ringmaster LMO grow with system heterogeneity and that the method outperforms strong synchronous and asynchronous baselines. theorem]Theorem theorem]Lemma theorem]Definition theorem]Assumption † † footnotetext: Code: github.com/vectozavr/ringmaster-lmo . 1 Introduction For more than a decade, Adam ( 25 ) has been the default choice for training large neural networks, with its decoupled weight-decay variant AdamW ( 31 ) more recently becoming the default variant. For matrix-valued layers, however, these methods treat each weight matrix as a vector, rather than explicitly using its matrix structure. Recently, 22 proposed the Muon optimizer, showing that using the matrix structure of hidden-layer weights can be beneficial. Muon forms the update direction from the matrix gradient via approximate orthogonalization. This matrix-structured update has led to strong empirical results, including the nanoGPT speedrun where Muon improves on the AdamW baseline ( 21 ) . Recent pretraining benchmarks also report that Muon can be substantially more compute-efficient than AdamW ( 44 ; 30 ; 51 ) . More generally, Muon fits into the framework of linear minimization oracle (LMO)-based optimizers introduced by 40 . This viewpoint makes the geometry of the update explicit: the choice of norm determines the update direction, and for matrix parameters the spectral-norm ball recovers the orthogonalized direction used by Muon . Despite this progress, LMO-based optimizers are still typically used within synchronous training pipelines. When several workers compute gradients in parallel, the usual synchronous implementation must wait for the slowest worker before forming the next update. Asynchronous methods, going back at least to HOGWILD! ( 42 ) , avoid this synchronization barrier by updating the model whenever a worker returns a gradient. The price is delay: a gradient arriving at iteration k may have been computed at an older iterate with delay \delta_{k}\geq 0 . Recent analyses have clarified the effect of such delays for asynchronous SGD , and delay-thresholding variants can achieve optimal time complexity under heterogeneous worker speeds ( 49 ; 37 ; 34 ) . These results, however, are built for SGD -type updates in Euclidean setting, and do not directly apply to LMO-based optimizers. This leaves the central question of this paper: Can asynchronous LMO-based methods recover the optimal time complexity known for Euclidean setting. We show that this can be done. Building on the delay-thresholding mechanism of Ringmaster ASGD ( 37 ) and the LMO-based momentum framework of 40 , we introduce an asynchronous LMO-based method for unconstrained stochastic nonconvex optimization. To the best of our knowledge, this is the first asynchronous LMO-based method with time complexity guarantees under heterogeneous worker computation times. In the classical Euclidean smooth setting, the bound recovers the optimal time complexity of Ringmaster ASGD . Table 1: High-level comparison with the closest theoretical works. All listed methods can be viewed through the LMO-based update perspective; the norm column reports the geometry covered by the available analysis, and ‘Euclidean’ indicates results specialized to normalized SGD . The asynchronous column indicates whether the method is analyzed in the delayed-gradient asynchronous setting, the parameter-agnostic column indicates whether the method avoids tuning the main schedule from problem parameters, and the generalized smoothness column indicates whether the theory covers our generalized (L_{0},L_{1}) -smoothness model ( 2.3 ). Method Norm Asynchronous Parameter- agnostic Generalized (L_{0},L_{1}) - smoothness Existing LMO-based methods ( 40 ; 41 ) Any norm ✘ ✘ ✔ NSGD-M ( 19 ) Euclidean ✘ ✔ ✔ Ringmaster Asynchronous NSGD-M ( 53 ) Euclidean ✔ ✘ ✘ Ringmaster LMO [new] ( Algorithm 1 ) Any norm ✔ ✔ ✔ 1.1 Contributions Our contributions are the following: • First asynchronous LMO-based method. We propose Ringmaster LMO ( Algorithm 1 ), an asynchronous LMO-based method that combines momentum with delay-thresholding. To the best of our knowledge, this is the first asynchronous LMO-based method with time complexity guarantees under heterogeneous worker computation times. The method extends the Ringmaster ASGD delay-control principle from SGD -type updates to LMO-based updates; see Section 3 . • Convergence under generalized smoothness. The analysis in Sections 4.1 and 4.2 establishes convergence under generalized (L_{0},L_{1}) -smoothness ( 55 ; 54 ) , which includes the classical smooth case as L_{1}=0 . The resulting rates recover the standard stochastic LMO-type dependence while allowing delayed gradients generated by asynchronous workers; see Theorems 4.1 and 4.4 . • Parameter-agnostic scheduling. Section 4.2 gives a variant with decreasing stepsizes and adaptive delay thresholds. This removes the need to tune the main schedule using problem parameters and, in particular, provides a parameter-agnostic extension that was not present in the original Ringmaster ASGD analysis; see Theorem 4.4 . • time complexity guarantees. Theorems 4.3 , 4.6 and E.6 translate the iteration complexity into time complexity under both the fixed computation model and the universal time-varying computation model. In the standard smooth Euclidean setting, these bounds recover the same optimal time complexity as Ringmaster ASGD . • Experiments. Section 5 compares the proposed method against synchronous and asynchronous baselines. On both stochastic quadratic problems and NanoChat pretraining, the proposed method is competitive in mild heterogeneity and its advantage grows as worker-speed heterogeneity increases. Table 1 places our method in the context of the closest theoretical works. 1.2 Related work We review the prior work most closely related to our setting. LMO-based and matrix-structured optimizers. The empirical success of Muon ( 22 ; 21 ) has motivated a line of work seeking to explain and generalize its matrix-structured update. 4 relate Muon to older adaptive and preconditioned methods, including Shampoo ( 16 ) , by emphasizing the role of the underlying norm. 40 give a broader optimization framework in which the update direction is chosen by a linear minimization oracle over a norm ball. LMOs are classical in Frank-Wolfe and conditional-gradient methods ( 14 ; 18 ; 9 ; 20 ) , but the Scion framework uses them for unconstrained stochastic optimization and recovers several normalized or structured-gradient methods through the choice of norm. In parallel, 27 interprets gradient orthogonalization through non-Euclidean

Read the complete text