Pre-Training LLMs on a budget: A comparison of three optimizers
Joel Schlotthauer, Christian Kroos · 2025
Read Pre-Training LLMs on a budget: A comparison of three optimizers by Joel Schlotthauer, Christian Kroos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08472.txt.
Description
Joel Schlotthauer and co-authors race three training optimizers on a tight budget in a 2025 paper. Pre-training a language model is an optimizer's marathon: the update rule decides how fast loss falls and how good the model ends. Three contenders line up: AdamW, the default champion; Lion, a lean rule found by evolutionary search; and Sophia, a second-order method that prices curvature.
Fairness drove the design. Each optimizer got its own tuned settings through small proxy models under a modern parameterization, across two base architectures and one- and multi-epoch schedules at fixed token counts. The finishes were close. Sophia reached the lowest training and validation loss, Lion trained fastest in wall-clock terms, and AdamW produced the best scores on downstream tasks. The paper gives budget-bound trainers a map of which virtue each rule actually buys.
Catalog record
| CREATOR | Joel Schlotthauer, Christian Kroos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08472