Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov, Matteo Pagliardini, Martin Jaggi · 2025
Read Benchmarking Optimizers for Large Language Model Pretraining by Andrei Semenov, Matteo Pagliardini, Martin Jaggi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01440.txt.
Introduction
New optimizers for training large language models appear often, each claiming faster training or fewer settings to tune, but differing test setups make claims hard to compare. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi benchmark recent optimization methods under one standardized protocol for language model pretraining. They vary model size, batch size, and training duration in a systematic way.
Each method was tuned carefully before comparison, so the results reflect best use rather than lazy defaults. The study offers practitioners guidance on which optimizer suits which scenario, and flags promising directions for researchers. All code is released and every experiment is reproducible. Semenov and the co-authors aim to make future optimizer claims easier to test fairly, in a field where training runs are costly and protocol drift is common.
Catalog record
| CREATOR | Andrei Semenov, Matteo Pagliardini, Martin Jaggi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 210 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01440