OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
Yuxuan Lou, Yang You · 2026
Read OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling by Yuxuan Lou, Yang You (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07815.txt.
Description
Yuxuan Lou and Yang You improve Muon, an optimizer that sets each matrix update's direction by orthogonalizing its momentum. Muon fixes direction well, but each layer's step size gets only a static shape-based correction, and the standard recipe for magnitudes was tuned for a different optimizer. The authors ask what the right per-layer scale should be.
Their method, OrScale, adapts the trust-ratio idea from LARS and LAMB to the orthogonalized setting, using the norm of the update actually applied and calibrating each layer's ratio to one. The Moonlight recipe then transfers with no new tuning sweep. In pretraining runs from small dense models to a large mixture-of-experts model, OrScale matched or beat the tuned baseline at equal wall-clock cost. The gap widened on the heterogeneous model, as theory predicted.
Catalog record
| CREATOR | Yuxuan Lou, Yang You |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07815