Fantastic Pretraining Optimizers and Where to Find Them

Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang · 2025

Read Fantastic Pretraining Optimizers and Where to Find Them by Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02046.txt.

Introduction

Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang benchmark pretraining optimizers fairly and find smaller speedups than often claimed. Optimizers steer model training by updating weights from gradients. AdamW has long dominated language model pretraining despite rivals claiming large speed gains. The team blames unequal tuning and narrow evaluation setups for the confusion.

Ten optimizers are tested across four model scales and varied data to model ratios with rigorous tuning. Rankings can flip mid training, so only end of training scores count. Real speedups shrink with scale, falling to a modest edge at the largest size tested. The fastest methods all use matrix preconditioners rather than per entry scaling. Careful methodology, the authors argue, is the real bottleneck.

Catalog record

CREATORKaiyue Wen, David Hall, Tengyu Ma, Percy Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE226 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fantastic Pretraining Optimizers and Where to Find Them Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang Abstract AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2 × \times speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B–1.2B parameters) and data-to-model ratios (1–8 × \times the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1 × \times for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners — multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4 × \times over AdamW for 0.1B parameter models to merely 1.1 × \times for 1.2B parameter models. 1 Introduction Figure 1: Top Left : The commonly used AdamW baseline for optimizer design is under-tuned. Up to a 2 \times speedup is achievable by tuning a single hyperparameter (learning rate) in the GPT-3 recipe Brown et al. 2020 for a 100M model (adopted in Liu et al. 2024a ; Wen et al. 2024 ; Yuan et al. 2025 ; Liang et al. 2025 ; Wang et al. 2025 ), highlighting the importance of proper hyperparameter optimization. Top Right : Fixing hyperparameters across optimizers does not guarantee fair comparison. Shared hyperparameters such as learning rate and weight decay are commonly set to a constant in previous studies. However, even conceptually similar optimizers may correspond to very different optimal hyperparameters. Bottom Left : Speedup decays with model size. While some optimizers show high (1.3-1.4 \times ) speedup over AdamW on models under 1B parameters, the speedup decays with model size to only 1.1 \times for 1.2B parameters. Bottom Right : Matrix-based optimizers consistently outperform scalar-based optimizers. The loss curves for three scalar-based optimizers (AdamW, Nesterov AdamW, Mars) and three matrix-based optimizers (Kron, Soap, Muon) trained with different Chinchilla ratios of data are shown. Matrix-based optimizers achieve a consistent speedup over scalar-based optimizers. Furthermore, the three matrix-based optimizers converge to a similar loss in an overtrained setting. Pretraining has been the most computationally expensive component in the training pipeline for large language models, accounting for over 95% of the cost in DeepSeek V3 DeepSeek-AI et al. 2025b , and the additional RL training cost in DeepSeek R1 DeepSeek-AI et al. 2025a is also comparatively much smaller. Until recently, AdamW has been the standard optimizer. Recent studies have introduced novel optimizers that claim to accelerate pretraining by 1.4 \times to 2 \times compared to AdamW Liu et al. 2024a ; Vyas et al. 2025 ; Liu et al. 2025a ; Yuan et al. 2025 ; Liang et al. 2025 ; Wang et al. 2025 ; Liu et al. 2025c ; Pethick et al. 2025 ; Ma et al. 2025 , yet these optimizers have not yet been widely adopted in real-world pretraining DeepSeek-AI et al. 2024 ; Yang et al. 2025 ; Grattafiori et al. 2024 , with the notable exception of Kimi K2 Team et al. 2025 , which uses the Muon-clip optimizer (a variant of Muon Jordan et al. 2024 ). We pinpoint two problems in optimizer evaluation in the evaluation process of these optimizers that both undermine confidence in new methods and limit their practical adoption. First, some baselines suffer from improperly tuned hyperparameters. Second, many experiments are confined to smaller-scale settings, leaving unanswered questions about how these optimizers perform in broader, more realistic scenarios. To address these issues, we benchmark eleven optimizers, including AdamW, in a rigorously controlled setup, focusing on two key questions: 1. How to ensure hyperparameter optimality? Previous works typically rely on manual hyperparameter selection and keep common hyperparameters such as learning rate and weight decay fixed across optimizers (e.g. Liu et al. 2025a ). This tuning process may result in a weak baseline, as the hyperparameters chosen may favor the proposed optimizers rather than for AdamW. We validate this concern by showing that tuning just one hyperparameter in the widely adopted GPT-3 recipe (introduced in Brown et al. 2020 and used in Liu et al. 2024a ; Wen et al. 2024 ; Yuan et al. 2025 ; Liu et al. 2025d ; Liu et al. 2025c ; Liang et al. 2025 ; Wang et al. 2025 ) can yield a 2 \times speedup for pretraining ( Figure 1 , top left). To address, this we perform coordinate descent in the hyperparameter space on N hyperparameters for all eleven optimizers, iterating until convergence across for each of six different settings for models with up to 0.5B parameters to ensure that the hyperparameters are near-optimal for each setting. 2. How do speedups differ across different scaling regimes? Large language models are trained in different regimes: depending on the model size, the data-to-model ratio (the number of tokens over the number of parameters) can range from 1 to more than 50 times the Chinchilla optimal (about 20 based on Hoffmann et al. 2022 ). Typical optimizer experiments in 1\times Chinchilla optimal regimes raise concerns about a new optimizer’s effectiveness in high data-to-model ratio. To address this, we benchmark the optimizers in four distinct data-to-model ratios (1 \times , 2 \times , 4 \times and 8 \times the Chinchilla optimal regime) and scale up to 1.2B parameter models (following Everett et al. 2024 ; Zhang et al. 2025a ). Concretely, we employ the Llama 2 architecture Touvron et al. 2023b ; Grattafiori et al. 2024 (ranging from 0.1B to 1.2B parameters) and a data mixture similar to OLMo 2’s OLMo et al. 2025 . We focus on the final validation loss on the C4-EN mixture as a known proxy for downstream performance Bhagia et al. 2024 , while also tracking exact downstream performance on various benchmarks. As previous works Vyas et al. 2025 ; Liu et al. 2025a show that the step-wise computation overhead of matrix-based optimizers can be reduced to under 10\% through proper implementation, we primarily compare algorithms by the number of tokens needed to reach a given loss. Our empirical results show the necessity of careful hyperparameter tuning and end-of-training evaluations across a range of model scales and data-to-model ratios: 1. Hyperparameter transfer between optimizers is non-trivial. Even similar optimizers may need very different hyperparameters (e.g., Lion’s optimal weight decay \approx 0.6 vs. AdamW’s \approx 0.1, Figure 1 , top right), so fixing hyperparameters across optimizers can lead to unfair comparisons. 2. The speedup of new optimizers is

Read the complete text