In Search of Adam’s Secret Sauce

Antonio Orvieto, Robert Gower · 2025

Read In Search of Adam’s Secret Sauce by Antonio Orvieto, Robert Gower (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21829.txt.

Description

Antonio Orvieto and Robert Gower investigate why Adam, the workhorse training algorithm for large language models, works so well. Many simpler variants of Adam have been proposed, such as methods that keep only the sign of each update. The pair ran a huge comparison, training more than fifteen hundred language models across data setups and sizes, pitting Adam against its simplified cousins with careful tuning.

They found the signed methods train faster than a basic baseline but still lag Adam even after tuning. One constraint stood out, though: forcing Adam's two momentum settings to be equal kept nearly all of its performance. That choice also unlocks a clean interpretation. In this form, Adam acts as an online method for estimating the average and spread of the update signals. The authors call this the secret sauce on top of signed momentum.

Catalog record

CREATORAntonio Orvieto, Robert Gower
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21829

Browse the full library · About the Tower of Babel