Benchmarking Optimizers for Large Language Model Pretraining

Andrei Semenov, Matteo Pagliardini, Martin Jaggi · 2025

Read Benchmarking Optimizers for Large Language Model Pretraining by Andrei Semenov, Matteo Pagliardini, Martin Jaggi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01440.txt.

Introduction

New optimizers for training large language models appear often, each claiming faster training or fewer settings to tune, but differing test setups make claims hard to compare. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi benchmark recent optimization methods under one standardized protocol for language model pretraining. They vary model size, batch size, and training duration in a systematic way.

Each method was tuned carefully before comparison, so the results reflect best use rather than lazy defaults. The study offers practitioners guidance on which optimizer suits which scenario, and flags promising directions for researchers. All code is released and every experiment is reproducible. Semenov and the co-authors aim to make future optimizer claims easier to test fairly, in a field where training runs are costly and protocol drift is common.

Catalog record

CREATORAndrei Semenov, Matteo Pagliardini, Martin Jaggi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE210 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01440

Browse the full library · About the Tower of Babel