Dion: Distributed Orthonormalized Updates

Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford · 2025

Read Dion: Distributed Orthonormalized Updates by Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05295.txt.

Introduction

Kwangjun Ahn and co-authors present Dion, a training rule for large language models that keeps the gains of a recent method called Muon while fitting how big models are stored. Muon uses heavy matrix math that clashes with weights split across many machines. Dion swaps that math for a cheaper repeated step on a stored momentum buffer.

A rank setting with error feedback lets Dion use partial updates that cost far less. Tests on models from 160 million to 3 billion weights show the same training gains as the older method. Wall-clock training time drops sharply at large scale, which makes the rule practical for the next wave of models.

Catalog record

CREATORKwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dion: Distributed Orthonormalized Updates Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford Abstract Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash with sharded weights in large-scale LLM training, causing high compute and communication cost. We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule that replaces Newton–Schulz iteration with amortized power iteration on a momentum buffer, avoiding full-matrix reconstruction and integrating cleanly with weight sharding. The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings. On language models from 160M to 3B parameters, Dion retains the benefits of orthonormalized updates, while markedly reducing wall-clock time at scale, making it a practical optimizer for next-generation foundation models. Code is available at: https://github.com/microsoft/dion/ \displaystyle\boxed{\text{Code is available at: \url{https://github.com/microsoft/dion/}}} Figure 1 : Wall-clock time per training step comparison of Muon and Dion optimizers on H100 GPUs (simulated on a single GPU without communication costs). ( Left ) Benchmark across matrix parameters of sizes 2048 2 -32768 2 , showing low-rank Dion achieve faster orthonormal updates at larger scales. ( Right ) Time per step on Llama-3 405B model ( Grattafiori et al., 2024 ) , a challenging configuration for Muon ( Essential-AI, 2025 ) . 1 Introduction Training state-of-the-art AI models consumes millions of GPU-hours, making improvements to the optimizer’s update rule critical for reducing costs. In this work, we propose a new update rule for matrix-valued parameters, which constitute the bulk of modern neural network weights. Orthonormalized update rules , exemplified by the Muon optimizer ( Jordan et al., 2024b ) , has recently shown strong promise. By conditioning weight updates to induce consistent changes in hidden states ( Bernstein, 2025 ) , they deliver faster convergence, improved training stability, and robust hyperparameter transfer across scales ( Bernstein and Newhouse, 2025 ; Large et al., 2024 ; Pethick et al., 2025 ) . Large-scale studies report that Muon nearly doubles the efficiency of AdamW ( Liu et al., 2025a ) , scales well with batch size ( Shah et al., 2025 ) , and has been deployed in foundation models such as Kimi-K2 ( Kimi-AI et al., 2025 ) and GLM-4.5 ( Zeng et al., 2025 ) . Yet scaling orthonormalized updates exposes a critical systems bottleneck. Muon’s reliance on Newton–Schulz iteration ( Bernstein and Newhouse, 2025 ; Jordan et al., 2024b ) requires dense matrix multiplications that conflict with how large-scale LLM training shards parameters across devices. For instance, ( Essential-AI, 2025 ) reports that Muon’s overhead can become either compute- or communication-bound when training Llama-3 405B. This motivates a central question: Can we devise a compute- and communication-efficient orthonormalized update rule? We introduce Dion (Distributed Orthonormalization), a scalable update rule that rethinks the underlying linear algebra. Instead of applying Newton–Schulz to the full update matrix, Dion performs amortized power iteration on a slowly evolving momentum buffer, avoiding full-matrix reconstruction and integrating seamlessly with weight sharding. Inspired by PowerSGD ( Vogels et al., 2019 ) , Dion decouples the momentum buffer to enable compressed data-parallel synchronization while preserving equivalence to the centralized formulation. A key scalability lever is the rank-fraction parameter, which allows low-rank updates that significantly reduce compute and communication costs. To enhance update quality, Dion uses error feedback , adding the low-rank approximation error in the momentum buffer, allowing persistent signals to accumulate and be expressed in future steps. Empirically, we observe that the rank required for strong performance grows slowly with model size, and larger models are more robust to rank reduction. We evaluate Dion on language models from 160M to 3B parameters, analyzing scaling with both model and batch size. Across all settings, Dion preserves the advantages of orthonormalized updates, while benchmarks ( Figure 1 ) show it delivers substantially lower wall-clock time at the largest scales. Together, these results indicate that Dion retains Muon’s optimization benefits while overcoming its system bottlenecks, making it a practical optimizer for training next-generation foundation models. 1.1 Why Dion? Advantages over Muon Given Muon’s success in large-scale AI training ( Kimi-AI et al., 2025 ; Zeng et al., 2025 ) , one may ask: what does Dion add? Its key advantages are: Preserved benefits. Being an orthonormalized update, Dion retains all of Muon’s practical strengths—fast convergence, tolerance of large batch sizes, robust hyperparameter transfer—as verified in Section 6 . Enhanced scalability. Dion introduces a new lever—the rank-fraction parameter—which trades a small amount of update quality for substantial efficiency gains. As shown in Figure 1 , low-rank variants remain practical even at very large matrix sizes, making Dion viable at extreme scales such as Llama-3 405B, where Muon encounters bottlenecks ( Essential-AI, 2025 ) . Seamless weight sharding. Dion supports efficient implementations under both 1D sharding ( Algorithm 3 ) and 2D sharding ( Algorithm 4 ), ensuring compatibility with distributed training. Algorithmic flexibility. By rethinking the underlying linear algebra—replacing Newton–Schulz iterations with amortized power iteration—Dion enables new forms of optimization, as demonstrated in Section 5 . For instance, combining lazy updates with CPU offloading yields orthonormalized updates with only minimal wall-clock overhead ( Section 5.1 ). 2 Notation Conventions Algorithm 1 A single step of Dion optimizer on a (sharded) matrix parameter X\in\mathbb{R}^{m\times n} . 1: Hyperparams: Rank factor r , learning rate \eta . 2: Optimizer states: momentum M\in\mathbb{R}^{m\times n} , right vectors V\in\mathbb{R}^{n\times r} . 3: Receive gradient G\in\mathbb{R}^{m\times n} 4: O,\,M_{\mathrm{new}},\,V_{\mathrm{new}}\leftarrow\begin{cases}\texttt{{Dion}}^{0\mathrm{D}}(G,M,V)&(\text{\autoref{alg:un-dion}})\quad\text{if }X\text{ is unsharded},\\ \texttt{{Dion}}^{1\mathrm{D}}(G,M,V)&(\text{\autoref{alg:1d-dion}})\quad\text{if }X\text{ is 1D-sharded},\\ \texttt{{Dion}}^{2\mathrm{D}}(G,M,V)&(\text{\autoref{alg:2d-dion}})\quad\text{if }X\text{ is 2D-sharded},\\ \begin{cases}\texttt{{Lazy-Dion}}_{k}(G,M,V)\\ \texttt{{CPU-Dion}}(G,M,V)\end{cases}&(\text{\autoref{alg:lazy-dion}})\quad\text{for faster variants.}\end{cases} 5: X_{\mathrm{new}}\leftarrow X-\eta\,O We adopt the sharded array and communication notation of “How To Scale Your Model” ( Austin et al., 2025 , Ch. 3) , which expresses parallelism and sharding directly in tensor indices. Below we give a concise overview of the notation, while referring readers to the reference for full details. A device mesh is written \text{Mesh}(\{\texttt{X}:a,\texttt{Y}:b,\texttt{Z}:c\}) , where each symbol denotes a named mesh axis. An array A[I,J,\ldots] is annotated as A[I_{\texttt{X}},J_{\texttt{Y}},\ldots] to indicate that dimension I is sharded across mesh axis X , J across Y , etc. Dimensions without subscripts are replicated. For example, X[I,J_{\texttt{X}}] denotes a matrix of shape (|I|,|J|) column-sharded across X . Low-rank factorizations introduce a dedicated rank axis R with size |R|=r . Matrix multiplications are written with explicit contracted indices: C[I,K]\leftarrow A[I,J]\cdot_{J}B[J,K], which makes clear which logical axis is eliminated. If J is sharded across X (e.g. A[I,J_{\texttt{X}}] and B[J_{\texttt{X}},R] ), the local computation produces a partial result along X , written with an unreduced tag: C[I,K]\{U_{\texttt{X}}\} . In other words, \{U_{\texttt{X}}\} means “this quantity still needs a reduction over the X mesh axis.” Applying a collective that reduces X clears the tag; for instance, \mathbf{AllReduce}_{\texttt{X}}\!\big(C[I,K]\{U_{\texttt{X}}\}\big). Collectives are written as explicit shape transforms: • \mathbf{AllGather}_{\texttt{X}} : removes a subscript {}_{\texttt{X}} . • \mathbf{ReduceScatter}_{\texttt{X},K} : reduces along X and shards the result across logical axis K . • \mathbf{AllReduce}_{\texttt{X}} : equivalent to \mathbf{ReduceScatter}+\mathbf{AllGather} . • \mathbf{AllToAll}_{\texttt{X},K} : moves a subscript {}_{\texttt{X}} from one logical axis to another K . We use the following axis notation throughout the paper: \texttt{X}=\text{Sharding Axis 1},\quad\texttt{Y}=\text{Sharding Axis 2},\quad\texttt{Z}=\text{Replicated Data Parallel}. For example, standard DDP corresponds to replication along

Read the complete text