Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
Depen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University · 2025
Read Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants by Depen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02431.txt.
Description
Depen Morwani and co-authors draw in 2025 explicit links between new deep learning optimizers and older accelerated gradient methods. Recent optimizers change how momentum works, while a separate line of work sped up stochastic gradient descent by decoupling the momentum coefficient from the weight on the current gradient. The paper shows how these two lines connect in theory.
Early experiments on a mid size language modeling task support the links. One optimizer that most closely matches the accelerated gradient form performed best. Building on that, the authors simplify that optimizer into a version that keeps its performance at both large and small batch sizes while needing only one momentum term instead of two.
Catalog record
| CREATOR | Depen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02431