To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

Sara Dragutinović, Rajesh Ranganath · 2026

Read To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters by Sara Dragutinović, Rajesh Ranganath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00742.txt.

Description

Dragutinović, Zhang, and Ranganath probe a downside of the Muon optimizer, whose speed comes at the cost of gradient descent's simplicity bias. Theory on deep linear networks and linear attention shows Muon dodges saddle points but sacrifices the sequential growth of solution complexity that marks gradient descent learning. Experiments show Muon struggles to find shared structure across tasks and may fit spurious features instead. Faster optimisation, the paper warns, is rarely free: gains can reshape the inductive biases that govern generalisation.

Catalog record

CREATORSara Dragutinović, Rajesh Ranganath
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45197
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00742

Browse the full library · About the Tower of Babel