Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model

Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research · 2026

Read Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model by Yizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05065.txt.

Description

Does stochastic gradient descent prefer flat valleys or sharp pits in the loss landscape? Yizhou Xu and 3 co-authors settle part of that debate in this 2026 machine learning paper with a model simple enough to solve exactly. Folklore links flat minima to good generalization, yet evidence has pointed both ways, and this work looks for a causal answer.

In their solvable model, the training rule itself has no flatness preference at all. What it prefers is small gradient fluctuations. The data then decides the ending sharpness: flat minima win only when label noise is spread evenly across output dimensions, and with lopsided noise the model can settle into arbitrarily sharp solutions. The team reproduced the effect in controlled tests on multilayer perceptrons, recurrent networks, and transformers, suggesting the lesson reaches beyond the toy model.

Catalog record

CREATORYizhou Xu 1 , Pierfrancesco Beneventano 2 , Isaac Chuang 2 , Liu Ziyin 2,3 1 École Polytechnique Fédérale de Lausanne 2 Massachusetts Institute of Technology 3 NTT Research
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05065

Browse the full library · About the Tower of Babel