AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm
Matia Bojović, Saverio Salzo, Massimiliano Pontil · 2026
Read AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm by Matia Bojović, Saverio Salzo, Massimiliano Pontil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13112.txt.
Description
Gradient methods live or die by stepsize choice, and adaptive methods like AdaGrad ease that tuning burden. Matia Bojovic and two co-authors propose AdaGrad-Diff, an AdaGrad-style method where adaptation follows cumulative squared norms of successive gradient differences, not gradient norms themselves. The intuition is simple: when gradients barely change, the stepsize should not shrink needlessly, while large gradient swings, signalling curvature or instability, should damp the stepsize automatically.
Numerical experiments show the new method hardier than AdaGrad in several settings that matter in practice. For practitioners tired of stepsize grids, adaptation driven by gradient change rather than gradient size offers a principled alternative: stable phases keep their pace, and turbulent phases slow down on their own.
Catalog record
| CREATOR | Matia Bojović, Saverio Salzo, Massimiliano Pontil |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-13112