The Effect of Mini-Batch Noise on the Implicit Bias of Adam

Matias D. Cattaneo, Boris Shigida · 2026

Read The Effect of Mini-Batch Noise on the Implicit Bias of Adam by Matias D. Cattaneo, Boris Shigida (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01642.txt.

Description

Matias Cattaneo and Boris Shigida study how mini batch noise shapes Adam's implicit bias in multi epoch training, where data makes passes matter. Adam carries two momentum settings, and batch size controls noise, which steers memory toward sharper or flatter regions linked to generalization. With large batches, second momentum amplifies anti regularization, hurting generalization, but with small batches it flips, and first momentum shifts oppositely. The default pair suits small batches; large batches gain by moving first momentum closer to second, with the flip tied to batch size scale.

Catalog record

CREATORMatias D. Cattaneo, Boris Shigida
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79931
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01642

Browse the full library · About the Tower of Babel