Tune My Adam, Please!
Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller, Frank Hutter · 2025
Read Tune My Adam, Please! by Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller, Frank Hutter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19733.txt.
Introduction
Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller, and Frank Hutter published a 2025 paper on tuning the Adam optimizer without the usual grind. Adam trains most deep learning models, and its settings strongly affect results. Searching for good settings is slow and expensive. A recent method pauses and resumes training runs to spend the budget wisely, but its predictions ignore how Adam's settings shape learning curves.
Their fix is a prediction model pre-trained specifically on Adam learning curves from a large public collection. It forecasts how a run will improve, guiding which runs deserve more time. A data trick multiplies the training examples by reshaping curves, boosting the forecaster further. On benchmark tuning tasks, the approach extrapolated curves better and found good settings faster. It also held up on unfamiliar tasks. The paper makes a narrow tool do its one job well: cheaper, quicker tuning of deep learning's workhorse optimizer.
Catalog record
| CREATOR | Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller, Frank Hutter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tune My Adam, Please! Theodoros Athanasiadis, Steven Adriaensen, Samuel Müller, Frank Hutter Abstract The Adam optimizer remains one of the most widely used optimizers in deep learning, and effectively tuning its hyperparameters is key to optimizing performance. However, tuning can be tedious and costly. Freeze-thaw Bayesian Optimization (BO) is a recent promising approach for low-budget hyperparameter tuning, but is limited by generic surrogates without prior knowledge of how hyperparameters affect learning. We propose Adam-PFN , a new surrogate model for Freeze-thaw BO of Adam’s hyperparameters, pre-trained on learning curves from TaskSet, together with a new learning curve augmentation method, CDF-augment, which artificially increases the number of available training examples. Our approach improves both learning curve extrapolation and accelerates hyperparameter optimization on TaskSet evaluation tasks, with strong performance on out-of-distribution (OOD) tasks. 1 Introduction Hyperparameter Optimization (HPO) can be costly and time-consuming, especially in the age of LLMs. Although Bayesian Optimization (BO) is a sample-efficient alternative, it still requires training a model in full to get a new evaluation point for the BO loop. Freeze-thaw BO (FT-BO) ( Swersky et al. 2014 ) is a more efficient alternative that allocates resources to the most prominent configurations one step (e.g., one epoch) at a time. Additionally, rather than disregarding a configuration, FT-BO keeps it frozen in memory and can decide whether to thaw that configuration or start a new one. Recently, Rakotoarison et al. 2024 introduced ifBO , a new, state-of-the-art framework for FT-BO. It consists of FT-PFN , a PFN ( Müller et al. 2022 ) based surrogate that performs Bayesian learning curve extrapolation, trained on synthetic hyperparameter (HP) configurations and learning curve combinations. However, ifBO attempts to fit every HPO case with one surrogate model. Our work introduces a drop-in replacement surrogate model for ifBO , Adam-PFN , specialized to tune the HPs of the Adam optimizer ( Kingma & Ba 2015 ) . We also introduce a learning curve augmentation method that allows us to train on real data from TaskSet ( Metz et al. 2020 ) , a collection of learning curves optimized using Adam, instead of sampling them from a synthetic prior. 2 Adam-PFN 2.1 Real Data: TaskSet TaskSet offers a collection of 1162 learning curves on a variety of tasks, including language modeling on words, subwords, and characters with RNNs, text classification with RNNs, CNNs trained on image data, and MLPs, all optimized with Adam. Each task contains the learning curves of the same 1000 randomly sampled HP configurations, including the learning rate, \beta_{1} , \beta_{2} , \epsilon , two parameters that control L_{1} and L_{2} regularization, and two parameters that control the learning rate decay schedule. All hyperparameter values are uniformly sampled on a log scale. Table 2 in Appendix A gives a more detailed overview of the hyperparameters’ lower and upper bounds. 2.2 Augmentation Method: CDF-augment To generate an even more diverse training set, we introduce CDF-augment, a local learning curve augmentation method that uses the cumulative distribution function (CDF) of the Beta distribution. To augment a learning curve, we first sample Beta’s mode \mu uniformly in [0, 1] and concentration \kappa uniformly in [2, 5], and calculate the CDF. We then forward-propagate the learning curve y through the CDF as follows y^{\prime}=F_{Beta}(y;\mu,\kappa) (1) Using this augmentation, we increase the learning curve diversity by non-linearly transforming task-hardness. Due to the nature of the CDF, the rank (ordering) of learning curves is preserved. Figure 1 shows the result of applying CDF-augment on a set of learning curves. (a) (b) Figure 1: (a) A selection of learning curves from TaskSet, the color of each curve is based on the learning rate value. (b) The same set of learning curves augmented with CDF-augment. CDF-augment non-linearly transformed "task hardness" by making the task easier. 2.3 Surrogate Model Training Our surrogate model is a PFN, trained on augmented data from real learning curves of the Adam optimizer, similar to the data it will encounter in practice. PFNs are based on the transformer architecture ( Vaswani et al. 2017 ) , and leverage in-context learning to perform Bayesian learning curve extrapolation in a single forward pass. We follow the same training pipeline for our surrogate model as FT-PFN , the only difference is that we do not sample the learning curves with and their HP configurations from a synthetic prior, rather, we train on real curves from TaskSet, which we augment using CDF-augment. We include more implementation details in Appendix B . Starting with a set of learning curves randomly sampled from 878 tasks and 1000 HP configurations per task, we augment and then split each curve into context and query points. We train our model to extrapolate the performance of a curve at a query point, given the context and HP configuration. More details on the training tasks can be found in Appendix C . 3 Experiments 3.1 Evaluation Procedure Our evaluation set consists of 12 NLP tasks from TaskSet, which were not used during training and are also used in prior work ( Kadra et al. 2023 ) , ( Wistuba et al. 2022 ) , ( Rakotoarison et al. 2024 ) . A list of the evaluation tasks is included in Appendix D . To evaluate the quality and runtime complexity of predictions, we compare our approach against DyHPO ( Wistuba et al. 2022 ) , DPL ( Kadra et al. 2023 ) , FT-PFN ( Rakotoarison et al. 2024 ) , and a Uniform Predictor that outputs a uniform distribution over the [0, 1] range for Log-likelihood and a constant value of 0.5 for MSE. We also evaluated the HPO performance of our surrogate as part of ifBO by replacing FT-PFN while keeping the proposed acquisition function. For HPO performance, we additionally compare against HyperBand ( Li et al. 2018 ) , ASHA ( Li et al. 2020 ) , Freeze-thaw with GPs ( Swersky et al. 2014 ) , and Random Search ( Bergstra & Bengio 2012 ) . We include additional details on the baselines and discuss how they relate to our work in Appendix E . We also included two variants of Adam-PFN, "No Aug" and "Mixup". Adam-PFN (No aug.) was trained on the initial TaskSet tasks, without any learning curve augmentation, while Adam-PFN (Mixup) was trained using the recently introduced Mixup learning curve augmentation method ( Lee et al. 2024 ) . Further details on Mixup are included in Appendix F . 3.2 Learning Curve Extrapolation Results Table 1 presents the learning curve extrapolation results. Adam-PFN (CDF) outperforms all other baselines, both in terms of Log-likelihood (LL) and MSE. As expected, the inference time of all PFN approaches is significantly lower than that of DPL and DyHPO, since they are trained offline. Algorithm Context 400 Context 1000 Context 1600 LL MSE Time LL MSE Time LL MSE Time Adam-PFN (CDF) 5.326 0.00054 0.697 5.422 0.00046 2.151 5.441 0.00043 4.014 Adam-PFN (Mixup) 4.916 0.00080 0.677 4.986 0.00067 2.131 5.022 0.00061 3.972 Adam-PFN (No aug.) 4.947 0.00062 0.671 5.042 0.00054 2.090 5.066 0.00052 3.993 FT-PFN 3.440 0.00184 0.693 3.473 0.00171 2.103 3.443 0.00164 4.017 DPL -29.752 0.00191 17.790 -22.192 0.00175 38.855 -16.154 0.00164 63.748 DyHPO -0.494 0.00336 25.662 -0.397 0.00317 113.217 -0.383 0.00317 264.284