PolarGrad : A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective

Tim Tsz-Kit Lau, Qi Long, Weijie Su · 2025

Read PolarGrad : A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective by Tim Tsz-Kit Lau, Qi Long, Weijie Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21799.txt.

Introduction

Tim Lau and co-authors frame a family of training methods for neural networks around matrix shape, in a 2025 paper. Optimizers steer how a network's weights shift during training. Classic tools treat the weights as one flat list. Newer ones, like Muon, respect that each layer's weights form a matrix, and they train faster for it.

The authors split two jobs apart: fixing bend across traits, and fixing bend across the matrix's rows and columns. That split explains quirks seen in training, from early wobbles in Adam to why warmup steps help. On that base they build PolarGrad, a class of methods drawn from the polar split of the gradient matrix. One member of the class is Muon itself, with a fresh scaling. Tests on matrix tasks and language model training show PolarGrad beating both Adam and Muon.

Catalog record

CREATORTim Tsz-Kit Lau, Qi Long, Weijie Su
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE213 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21799

Browse the full library · About the Tower of Babel