DMuon: Efficient Distributed Muon Training with Near-Adam Overhead

X SQUARE ROBOT TEAM · 2026

Read DMuon: Efficient Distributed Muon Training with Near-Adam Overhead by X SQUARE ROBOT TEAM (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27153.txt.

Introduction

The X Square Robot Team presents DMuon, a distributed version of the Muon optimiser used to train large neural networks. Muon treats each weight matrix as a whole and orthogonalises it, rather than nudging entries one by one like AdamW. Its Newton-Schulz iterations can cost more than twice the normal forward and backward passes. In 2026, DMuon drops into existing pipelines with no framework changes.

Tests span embodied foundation models and large language models. End-to-end step time speeds up by roughly one and a half to three times across those workloads. The optimiser step alone runs many times faster, in the best cases over a hundred times faster. Per-step delay lands near AdamW levels, removing the old penalty for matrix-aware training. Efficient distributed Muon training becomes practical at scale.

Catalog record

CREATORX SQUARE ROBOT TEAM
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27153

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DMuon: Efficient Distributed Muon Training with Near-Adam Overhead X SQUARE ROBOT TEAM Abstract Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-aware updates offer a compelling alternative to conventional element-wise optimization, particularly as model architectures continue to grow in scale and heterogeneity. Yet contemporary distributed training infrastructure built around the assumption of element-wise optimizers is poorly matched to matrix-level optimizers such as Muon, whose updates couple entire weight matrices and require costly Newton-Schulz iterations. Vanilla Muon implementations incur more than 𝟐 × \mathbf{2\times} the cost of forward and backward passes. To close this gap, we present DMuon, an open-source distributed Muon implementation that integrates into existing training pipelines as a drop-in module, with no framework-level modifications. Across both embodied foundation model and large language model (LLM) training workloads, DMuon achieves a 1.48 × – 3.01 × \mathbf{1.48\times\!\text{--}\!3.01\times} speedup in end-to-end step time and a 6.85 × – 163.00 × \mathbf{6.85\times\!\text{--}\!163.00\times} speedup in optimizer-step time, bringing per-step latency to near-AdamW levels and enabling efficient scaling in our model training. \metadata [Code] https://github.com/X-Square-Robot/dmuon 1 Introduction ⬇ import dmuon dmuon . dedicate_params ( model , mesh ) opt = dmuon . Muon ( model ) (a) Programming interface (b) Negligible per-step overhead (c) Faster wall-clock convergence Figure 1 : (a) Three lines of code replace AdamW with DMuon, at a (b) negligible per-step overhead, yielding (c) faster wall-clock convergence. The 3.1 \times speedup is averaged across the four models in Section 5.1 , and the loss curves are captured from our real training run of Wall-OSS-0.5 . Large-model training has begun to move beyond the element-wise optimization paradigm that has dominated since AdamW ( 16 ) . Matrix-aware optimizers, most prominently Muon ( 11 ) , apply a Newton–Schulz iteration to the momentum-aggregated gradient of each weight matrix, producing updates whose singular values are driven toward a near-uniform spectrum. Recent works suggest that this update geometry translates into meaningful practical benefits, Moonlight ( 15 ) reports approximately 2\times higher compute efficiency than AdamW under compute-optimal pretraining, and Kimi-K2, DeepSeek-V4 ( 17 , 7 ) have adopted Muon in production-scale training, providing further evidence that matrix-aware optimization is viable at frontier-model scale. The algorithmic promise of matrix-aware optimizers is therefore increasingly well established. Their deployment cost, however, remains a significant practical obstacle. Modern distributed training stacks execute the optimizer step on shards of each parameter, one shard per rank. The implicit contract is that the optimizer’s update rule is element-wise: AdamW’s updates apply to a shard with no knowledge of any other shard. Muon breaks this contract. Its core computation, the Newton–Schulz iteration, operates on the full weight matrix rather than individual shards. Before a Muon update can be computed, the matrix must therefore be reconstructed, introducing communication that is absent from conventional element-wise optimizers. This additional communication is not a minor overhead. It appears at every optimizer step, for every matrix parameter, and scales with model size and distributed width. In practice, the cost of distributed Muon can rival or even exceed the combined wall-clock time of the forward and backward passes. The extra per-step cost erodes Muon’s FLOP-level advantage and weakens the wall-clock incentive to deploy it at scale. The penalty is particularly pronounced for embodied-model training. Compared to contemporary LLM and VLM pretraining ( 18 , 23 , 24 , 3 ) , whose forward–backward passes typically involve substantially more computation per optimizer step, VLA training ( 4 , 10 , 12 , 33 , 32 , 31 ) typically uses much shorter temporal contexts. As a result, the forward–backward phase occupies a much smaller fraction of the step time, making optimizer overhead significantly harder to amortize. Therefore, we present DMuon, a distributed Muon that closes the per-step performance gap to AdamW. Concretely, this work makes the following contributions: • Fine-grained communication optimization. We develop an owner-centric strategy that minimizes the communication overhead of distributed Muon execution while preserving exact optimizer semantics. • Shape-adaptive execution stack. We develop a high-performance execution engine that combines batched Gram Newton–Schulz execution, symmetry-aware kernels, and DSL-driven autotuning, enabling efficient orthogonalization across heterogeneous matrix workloads. • Computation-aware load balancing. We formulate owner assignment as a measured-cost optimization problem, leveraging runtime profiling to balance work across owners under batching, kernel-selection, and autotuning effects. • Drop-in distributed module. We provide DMuon as a drop-in module that remains orthogonal to the host training framework and parallelism strategy. Existing training pipelines can benefit from distributed Muon with minimal integration effort while preserving exact optimizer semantics. • Production-scale validation. We validate DMuon on both robotics and language-model training workloads, including the production training of WALL-OSS-0.5 and WALL-WM, demonstrating near-AdamW optimizer overhead while retaining Muon’s convergence benefits. In sum, as shown in Figure 1 , DMuon enables Muon’s convergence benefits to translate into practical training speedups. The following sections describe how DMuon combines communication, scheduling, and kernel optimizations to achieve these gains. 2 Background and Motivation This section establishes the three pieces of context needed to motivate DMuon’s design: the Muon optimizer and its Newton-Schulz iteration (§ 2.1 ), the sharded-training abstractions DMuon operates under (§ 2.2 ), and the structural mismatch between the two (§ 2.3 ) that drives the system cost we set out to eliminate. 2.1 Muon and the Newton–Schulz Iteration Muon ( 11 ) is a matrix-aware optimizer designed for two-dimensional parameters such as the weight matrices of linear and attention projections. For a weight matrix W\in\mathbb{R}^{m\times n} with momentum-smoothed gradient M_{t} , the core update rule is W_{t+1}=W_{t}-\eta\cdot\mathrm{NS}_{k}(M_{t}), (1) where \mathrm{NS}_{k}(\cdot) denotes k steps of a Newton–Schulz iteration that approximates the matrix sign function, equivalently the orthogonal factor UV^{\top} of the SVD M_{t}=U\Sigma V^{\top} . Each Newton–Schulz step has the form X_{i+1}=aX_{i}+b\,X_{i}X_{i}^{\top}X_{i}+c\,(X_{i}X_{i}^{\top})^{2}X_{i}, (2) where the coefficients (a,b,c) are chosen to drive the singular values of X_{i} toward unity. In practice, k=5 iterations are typically sufficient, and the computation is performed in reduced precision since only the singular subspace, rather than the singular values themselves, must be preserved. The practical cost of Muon is dominated by the Newton–Schulz iteration. A single optimizer step requires multiple Newton–Schulz updates, each consisting primarily of large matrix multiplications. Consequently, the cost of orthogonalization grows rapidly with matrix dimensions and can become a substantial component of the overall training step. As model sizes continue to increase, efficiently executing the Newton–Schulz iteration becomes critical for practical deployment. The remainder of this paper focuses on reducing the systems overhead associated with this computation. 2.2 Sharded Training Abstractions Distributed training systems partition weight matrices in different ways. The location of shard boundaries determines how a matrix is stored, communicated, and reconstructed during training. We briefly review the three sharding schemes most relevant to matrix-aware optimization: ZeRO-style sharding, FSDP-style sharding, and tensor parallelism. ZeRO-style: flatten then shard. ZeRO ( 19 ) (as realized by DeepSpeed ( 20 ) ) first flattens a group of parameters into a single contiguous buffer and then partitions that buffer across data-parallel ranks. Shard boundaries are defined by storage layout rather than tensor structure, so a single matrix may span multiple ranks and share a shard with unrelated parameters. FSDP: per-tensor sharding. FSDP2 ( 35 ) represents each parameter as

Read the complete text