Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

Depen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University · 2025

Read Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants by Depen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02431.txt.

Description

Depen Morwani and co-authors draw in 2025 explicit links between new deep learning optimizers and older accelerated gradient methods. Recent optimizers change how momentum works, while a separate line of work sped up stochastic gradient descent by decoupling the momentum coefficient from the weight on the current gradient. The paper shows how these two lines connect in theory.

Early experiments on a mid size language modeling task support the links. One optimizer that most closely matches the accelerated gradient form performed best. Building on that, the authors simplify that optimizer into a version that keeps its performance at both large and small batch sizes while needing only one momentum term instead of two.

Catalog record

CREATORDepen Morwani Kempner Institute, Harvard University [email protected], Nikhil Vyas Department of Computer Science, Harvard University [email protected] Hanlin ZhangDepartment of Computer Science, Harvard University [email protected] Sham KakadeKempner Institute, Harvard University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02431

Browse the full library · About the Tower of Babel