PolarGrad : A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
Tim Tsz-Kit Lau, Qi Long, Weijie Su · 2025
Read PolarGrad : A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective by Tim Tsz-Kit Lau, Qi Long, Weijie Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21799.txt.
Introduction
Tim Lau and co-authors frame a family of training methods for neural networks around matrix shape, in a 2025 paper. Optimizers steer how a network's weights shift during training. Classic tools treat the weights as one flat list. Newer ones, like Muon, respect that each layer's weights form a matrix, and they train faster for it.
The authors split two jobs apart: fixing bend across traits, and fixing bend across the matrix's rows and columns. That split explains quirks seen in training, from early wobbles in Adam to why warmup steps help. On that base they build PolarGrad, a class of methods drawn from the polar split of the gradient matrix. One member of the class is Muon itself, with a fresh scaling. Tests on matrix tasks and language model training show PolarGrad beating both Adam and Muon.
Catalog record
| CREATOR | Tim Tsz-Kit Lau, Qi Long, Weijie Su |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 213 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21799