OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling

Yuxuan Lou, Yang You · 2026

Read OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling by Yuxuan Lou, Yang You (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07815.txt.

Description

Yuxuan Lou and Yang You improve Muon, an optimizer that sets each matrix update's direction by orthogonalizing its momentum. Muon fixes direction well, but each layer's step size gets only a static shape-based correction, and the standard recipe for magnitudes was tuned for a different optimizer. The authors ask what the right per-layer scale should be.

Their method, OrScale, adapts the trust-ratio idea from LARS and LAMB to the orthogonalized setting, using the norm of the update actually applied and calibrating each layer's ratio to one. The Moonlight recipe then transfers with no new tuning sweep. In pretraining runs from small dense models to a large mixture-of-experts model, OrScale matched or beat the tuned baseline at equal wall-clock cost. The gap widened on the heterogeneous model, as theory predicted.

Catalog record

CREATORYuxuan Lou, Yang You
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07815

Browse the full library · About the Tower of Babel