Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW

Sourabh Medapati, [email protected], Priya Kasimbeg, [email protected], Shankar Krishnan, [email protected], Naman Agarwal, George Dahl, [email protected] · 2025

Read Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW by Sourabh Medapati, [email protected], Priya Kasimbeg, [email protected], Shankar Krishnan, [email protected], Naman Agarwal, George Dahl, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03986.txt.

Introduction

Sourabh Medapati and co-authors offer in 2025 ready-made lists of training settings for a popular neural network optimizer. Training a network means picking many dials, from step size to dropout, and poor picks waste scarce computer time. Their answer is a short ordered list of proven settings to try in turn.

The lists were distilled from wide searches on a hard public benchmark of real training jobs. The best list did well even on jobs it was not built from, and within five tries it beat plain step-size sweeps and a standard automated tuning tool. For teams with tiny budgets, the list is a turn-key start.

Catalog record

CREATORSourabh Medapati, [email protected], Priya Kasimbeg, [email protected], Shankar Krishnan, [email protected], Naman Agarwal, George Dahl, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW Sourabh Medapati, [email protected], Priya Kasimbeg, [email protected], Shankar Krishnan, [email protected], Naman Agarwal, George Dahl, [email protected] Abstract If we want to train a neural network using any of the most popular optimization algorithms, we are immediately faced with a dilemma: how to set the various optimization and regularization hyperparameters? When computational resources are abundant, there are a variety of methods for finding good hyperparameter settings, but when resources are limited the only realistic choices are using standard default values of uncertain quality and provenance, or tuning only a couple of the most important hyperparameters via extremely limited hand-designed sweeps. Extending the idea of default settings to a modest tuning budget, Metz et al. (2020) proposed using ordered lists of well-performing hyperparameter settings, derived from a broad hyperparameter search on a large library of training workloads. However, to date, no practical and performant hyperparameter lists that generalize to representative deep learning workloads have been demonstrated. In this paper, we present hyperparameter lists for NAdamW derived from extensive experiments on the realistic workloads in the AlgoPerf : Training Algorithms benchmark. Our hyperparameter lists also include values for basic regularization techniques (i.e. weight decay, label smoothing, and dropout). In particular, our best NAdamW hyperparameter list performs well on AlgoPerf held-out workloads not used to construct it, and represents a compelling turn-key approach to tuning when restricted to five or fewer trials. It also outperforms basic learning rate/weight decay sweeps and an off-the-shelf Bayesian optimization tool when restricted to the same budget. 1 Introduction In order to train a neural network, practitioners need to choose a training algorithm and the associated hyperparameter values. This procedure generally involves large scale tuning studies requiring lots of computational resources to arrive at the best performing hyperparameters. In resource-constrained settings, such tuning protocols lead to poor hyperparameter choices, as do protocols based on limited, low-dimensional sweeps or library defaults. In some cases, we can only afford a handful of tuning trials. Such a constraint can occur both when computational resources are limited in an absolute sense, and when they are abundant, but the cost of a single training run is also quite large. The latter situation requires deriving scaling ladders based on smaller versions of the same problem and having access to good defaults at lower scales is helpful to practitioners in this situation. For concreteness, let us assume we are restricted to a modest tuning budget of 5 trials, run in parallel, 1 1 1 Although arbitrary, a budget of 5 parallel trials matches the external tuning rules of the AlgoPerf training algorithms benchmark ( Dahl et al., 2023 ) , which we make extensive use of in the rest of this work. and must select a hyperparameter configuration for a new training problem. There are an endless variety of possible tuning procedures we could use. For example, we could fix all other hyperparameters to default values and just sweep the learning rate over 5 different values spaced uniformly on a log scale. We could use a random search on a small, handcrafted search space. We could handpick 5 specific points based on previous experience with other problems and any intuition we might have. We could build a smaller version of the training problem, do more extensive tuning studies, and then apply scaling heuristics (e.g. using \mu -parameterization ( Yang et al., 2022 ) ) to transfer the settings back to the original problem. We could even run a Bayesian optimization tool, such as Vizier ( Song et al., 2023 ) , on a handcrafted search space, although technically if we ran 5 sequential trials that would be a more generous budget than our assumption of 5 trials that ran in parallel. At the end of the day, we need to have some procedure that works well and is reproducible, so it behooves us to try and measure the efficacy of different natural protocols. In this paper, we study the problem of finding a good hyperparameter configuration on a new problem in only a handful of parallel tuning trials. We draw inspiration from Metz et al. (2020) , who suggested a tuning protocol using precomputed, ordered lists of hyperparameter configurations. The essence of their hyperparameter-list approach is to find a list of configurations that do well on a diverse library of training workloads and then try them, in priority order, on a new problem, up to the limit of the tuning budget. By defining an ordered list, Metz et al. (2020) have a concise way of recommending a set of configurations for any budget up to the size of their full list. Unfortunately, the specific precomputed hyperparameter lists offered by Metz et al. (2020) do not perform as well as we would like on typical deep learning benchmark workloads, and do not train successfully on any of the workloads we tried, and thus are unlikely to be very useful to practitioners. To the best of our knowledge, to date there has been no convincing evidence that such an ordered list of hyperparameter configurations can outperform simple alternative tuning procedures on typical deep learning workloads that weren’t used to build the lists, even at modest tuning budgets. In this work, we show that, for a restrictive tuning budget of 5 parallel trials, our hyperparameter lists outperform alternative tuning protocols in leave-one-workload-out cross validation on the workloads of the AlgoPerf: Training Algorithms benchmark ( Dahl et al., 2023 ) , as well as performing well on held-out variant workloads. We chose to construct hyperparameter lists for a variant of Adam ( Kingma & Ba, 2014 ) : Nesterov Adam with decoupled weight decay (NAdamW) ( Dozat, 2016 ; Loshchilov & Hutter, 2019 ) . We elected NAdamW because it outperformed AdamW in Dahl et al. (2023) , as well as in our own anecdotal experience. Specifically, in this work we make the following contributions: 1. Similar to Metz et al. (2020) , we articulate a simple and generic method that produces hyperparameter lists given a (potentially broad) hyperparameter search space and a library of workloads. 2. We show that a 5-point hyperparameter list for NAdamW constructed by our method can successfully train 7 of the 8 AlgoPerf base workloads, measured with leave-one-out cross validation over workloads, and that our best 5-point hyperparameter list generated using all base workloads also successfully trains on official AlgoPerf workload variants designed to measure training algorithm robustness to small architectural changes. 3. We show that our NAdamW hyperparameter lists (once again measured using leave-one-out cross validation over the AlgoPerf base workloads) outperform other natural alternative procedures for resource-constrained hyperparameter tuning that are also restricted to the same budget of a modest number of parallel trials (and also don’t have access to any special information about the workload). Specifically, compared to quasi-random search on the broad hyperparameter search space we used to build our lists, on most workloads our lists are more than three times more efficient in tuning. Collectively, our results show that—perhaps surprisingly—at least for an Adam-style training algorithm at a limited tuning budget, we can find a small set of hyperparameter configurations that generalize well enough to new workloads to

Read the complete text