To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Sara Dragutinović, Rajesh Ranganath · 2026
Read To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters by Sara Dragutinović, Rajesh Ranganath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00742.txt.
Description
Dragutinović, Zhang, and Ranganath probe a downside of the Muon optimizer, whose speed comes at the cost of gradient descent's simplicity bias. Theory on deep linear networks and linear attention shows Muon dodges saddle points but sacrifices the sequential growth of solution complexity that marks gradient descent learning. Experiments show Muon struggles to find shared structure across tasks and may fit spurious features instead. Faster optimisation, the paper warns, is rarely free: gains can reshape the inductive biases that govern generalisation.
Catalog record
| CREATOR | Sara Dragutinović, Rajesh Ranganath |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45197 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00742