Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model
Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research · 2026
Read Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model by Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05065.txt.
Description
Does stochastic gradient descent prefer flat valleys or sharp pits in the loss landscape? Yizhou Xu and 3 co-authors settle part of that debate in this 2026 machine learning paper with a model simple enough to solve exactly. Folklore links flat minima to good generalization, yet evidence has pointed both ways, and this work looks for a causal answer.
In their solvable model, the training rule itself has no flatness preference at all. What it prefers is small gradient fluctuations. The data then decides the ending sharpness: flat minima win only when label noise is spread evenly across output dimensions, and with lopsided noise the model can settle into arbitrarily sharp solutions. The team reproduced the effect in controlled tests on multilayer perceptrons, recurrent networks, and transformers, suggesting the lesson reaches beyond the toy model.
Catalog record
| CREATOR | Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05065