Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

Ruinan Jin, Yingbin Liang, Shaofeng Zou · 2026

Read Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails by Ruinan Jin, Yingbin Liang, Shaofeng Zou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03099.txt.

Description

Ruinan Jin, Yingbin Liang, and Shaofeng Zou give the first proof that Adam's edge over plain stochastic gradient descent is real, in a precise sense. Adam converges faster in practice on many problems, yet theory long rated the two methods about equal. Their 2026 analysis isolates Adam's second-moment normalization, the rescaling that steadies each coordinate's step.

Using a stopping-time and martingale argument under the standard bounded-variance assumption, they separate the methods' high-probability guarantees. Adam's dependence on the confidence level is provably gentler, scaling with its square root where SGD must pay the full amount. That sharper tail behavior is a concrete, measurable advantage, not just a heuristic impression. The result helps explain a decade of empirical preference for Adam.

Catalog record

CREATORRuinan Jin, Yingbin Liang, Shaofeng Zou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE138420
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03099

Browse the full library · About the Tower of Babel