Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, Trung Le · 2026

Read Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence by Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, Trung Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13079.txt.

Description

Nguyen and five co-authors explain why Muon, which orthogonalises the momentum buffer before each update, tolerates far larger learning rates than other optimizers. The mechanism is spectral flattening: Muon's largest stable step scales with the average singular value of the gradient, not the largest, which bottlenecks standard gradient descent. Recast as a preconditioned gradient method, it improves the effective convergence factor under a Kronecker-factored curvature model. Tests confirm Muon stays stable where SGD diverges early and hits accuracy milestones epochs sooner.

Catalog record

CREATORTien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, Trung Le
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57979
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13079

Browse the full library · About the Tower of Babel