AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm

Matia Bojović, Saverio Salzo, Massimiliano Pontil · 2026

Read AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm by Matia Bojović, Saverio Salzo, Massimiliano Pontil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13112.txt.

Description

Gradient methods live or die by stepsize choice, and adaptive methods like AdaGrad ease that tuning burden. Matia Bojovic and two co-authors propose AdaGrad-Diff, an AdaGrad-style method where adaptation follows cumulative squared norms of successive gradient differences, not gradient norms themselves. The intuition is simple: when gradients barely change, the stepsize should not shrink needlessly, while large gradient swings, signalling curvature or instability, should damp the stepsize automatically.

Numerical experiments show the new method hardier than AdaGrad in several settings that matter in practice. For practitioners tired of stepsize grids, adaptation driven by gradient change rather than gradient size offers a principled alternative: stable phases keep their pace, and turbulent phases slow down on their own.

Catalog record

CREATORMatia Bojović, Saverio Salzo, Massimiliano Pontil
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-13112

Browse the full library · About the Tower of Babel