In Search of Adam’s Secret Sauce
Antonio Orvieto, Robert Gower · 2025
Read In Search of Adam’s Secret Sauce by Antonio Orvieto, Robert Gower (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21829.txt.
Description
Antonio Orvieto and Robert Gower investigate why Adam, the workhorse training algorithm for large language models, works so well. Many simpler variants of Adam have been proposed, such as methods that keep only the sign of each update. The pair ran a huge comparison, training more than fifteen hundred language models across data setups and sizes, pitting Adam against its simplified cousins with careful tuning.
They found the signed methods train faster than a basic baseline but still lag Adam even after tuning. One constraint stood out, though: forcing Adam's two momentum settings to be equal kept nearly all of its performance. That choice also unlocks a clean interpretation. In this form, Adam acts as an online method for estimating the average and spread of the update signals. The authors call this the secret sauce on top of signed momentum.
Catalog record
| CREATOR | Antonio Orvieto, Robert Gower |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21829