Spend More to Save More (SM 2 ): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

Daniel Geißler, Bo Zhou, Sungho Suh, Paul Lukowicz · 2024

Read Spend More to Save More (SM 2 ): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization by Daniel Geißler, Bo Zhou, Sungho Suh, Paul Lukowicz (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08526.txt.

Introduction

Daniel Geißler, Bo Zhou, Sungho Suh, and Paul Lukowicz present a 2024 paper on tuning machine learning models without wasting energy. Hyperparameter search trains many model variants, and most runs are doomed. Successive halving already prunes losers early; the authors make that pruning energy-aware.

Their method, called Spend More to Save More, spends a little energy up front on exploratory pretraining to spot inefficient configurations cheaply. It tracks real power draw and folds in hardware traits when deciding what survives each round. Across datasets, models, and machines, the approach found strong configurations while cutting the energy burned on hopeless ones. The authors frame frugal tuning as part of sustainable machine learning.

Catalog record

CREATORDaniel Geißler, Bo Zhou, Sungho Suh, Paul Lukowicz
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08526

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spend More to Save More (SM 2 ): An Energy-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization Daniel Geißler, Bo Zhou, Sungho Suh, Paul Lukowicz Abstract A fundamental step in the development of machine learning models commonly involves the tuning of hyperparameters, often leading to multiple model training runs to work out the best-performing configuration. As machine learning tasks and models grow in complexity, there is an escalating need for solutions that not only improve performance but also address sustainability concerns. Existing strategies predominantly focus on maximizing the performance of the model without considering energy efficiency. To bridge this gap, in this paper, we introduce Spend More to Save More (SM 2 ), an energy-aware hyperparameter optimization implementation based on the widely adopted successive halving algorithm. Unlike conventional approaches including energy-intensive testing of individual hyperparameter configurations, SM 2 employs exploratory pretraining to identify inefficient configurations with minimal energy expenditure. Incorporating hardware characteristics and real-time energy consumption tracking, SM 2 identifies an optimal configuration that not only maximizes the performance of the model but also enables energy-efficient training. Experimental validations across various datasets, models, and hardware setups confirm the efficacy of SM 2 to prevent the waste of energy during the training of hyperparameter configurations. 1 Introduction The rising complexity of Artificial Intelligence (AI) applications solved through advanced deep-learning models continuously increases the energy demand of the whole AI sector. It is estimated that large language models like the popular GPT-3 may require up to 1300 MWh of energy for training, whereas inference may account for even more energy consumption over time Patterson et al. 2021 . This trend is driven by the ubiquity of machine learning applications being present in the majority of our daily lives. Considering the life cycle of an AI model from initial development towards its final deployment possibilities, there are multiple stages involved that may be optimized in terms of energy efficiency simply due to the waste of resources. This work accounts for the initial stages of the life cycle, with a focus on optimizing the training process of machine learning models to generate a more energy-aware and efficient solution for AI developers. Besides the quality of the dataset and model architecture fitment to the desired task, hyperparameter optimization (HPO) is an integral part of a well-suited model. Two key hyperparameters, namely batch size and learning rate, significantly affect training convergence quality and speed He et al. 2019 . However, determining the optimal values for these parameters, often hardware and task-specific, remains a challenging task requiring selection by experienced developers at the first attempt. Various strategies have been developed to improve the HPO problem, yet their only aim is to maximize the model’s prediction performance. Additionally, due to trends of supporting AI developers’ work with high-performance data centers, usually, due to economic reasons, it is a simple but energy-intensive process to spawn multiple training runs to manually explore the hyperparameter space. While more efficient hardware has continuously been updated to alleviate the training cost of trial and error, there lacks a holistic method that incorporates hardware energy footprint into the model training HPO process. To resolve this issue, this paper presents Spend More to Save More (SM 2 ), a novel approach combining HPO and energy consumption tracking to generate a profound strategy to sustainably improve the machine learning training process. To the best of our knowledge, this work is the first work to optimize the hyperparameters while considering energy consumption. Our work makes significant contributions to the field of sustainable HPO and can be summarized as follows: • Implementing energy-aware training through hardware-based power monitoring. • Deploying a sequential Successive Halving Algorithm strategy to minimize energy waste. • Extending the traditional training regime with exploratory components to energy-efficiently explore hyperparameter configurations. • Evaluating the SM 2 approach across three different scenarios of models and datasets to prove the possibility of equal model performance while improving energy efficiency. 2 Related Works 2.1 Hyperparameter Optimization The landscape of HPO covers multiple different methodologies from different areas to explore and manifest the best-performing hyperparameter setting. In many cases, there is no clear identification of the best-performing strategy possible because the efficiency of the algorithm depends on the respective machine learning problem to solve and the user’s preferences. Nevertheless, there is a trend of different HPO categories being fused to elevate the performance compared to the traditional algorithms. Bischl et al. 2021 Starting with the classic methods, grid search is the most common and simple strategy next to the manual exploration of hyperparameters. The strategy is based on an initialized grid that covers the range of each hyperparameter LeCun et al. 1998 . Due to its simplicity, it is commonly used but suffers from inefficiencies in high-dimensional scenarios due to many low-performing training runs and their independence from each other. Another simple but more efficient strategy in this scope is random search Bergstra and Bengio 2012 . Instead of exploring a fixed grid, the hyperparameter space is randomly sampled to provide a more stochastic exploration. Due to its ease of implementation, it still serves as a benchmark to compare it to more advanced strategies. Another category involves evolutionary strategies to work out the best hyperparameter setting Beyer and Schwefel 2002 . Such algorithms explore the hyperparameter space based on the biological concept of evolution. Throughout a fitness function, a population of hyperparameter configurations is evaluated whereas the worst-performing configurations are removed. Instead of proceeding with the most promising configurations, similar to natural evolution, the next iteration consists of crossovers and mutations of the previous configurations. Works like Awad et al. 2021 and Bochinski et al. 2017 confirm the usability of evolutionary strategies as state-of-the-art. Bayesian optimization constitutes another strategy for HPO, introduced through works like Snoek et al. 2012 . Such algorithms can efficiently explore the hyperparameter space by constructing probabilistic surrogate models and using acquisition functions to guide the search. Introduced by Bergstra et al. 2011 , Tree-structured Parzen Estimators (TPE) is a Bayesian optimization variant that ranks the performance of HPO configurations. TPE has shown impressive results, especially in high-dimensional optimization tasks, making it one of the current state-of-the-art HPO algorithms Hyperparameter optimization through Reinforcement Learning (RL) was recently introduced through works like Jomaa et al. 2019 and Wu et al. 2023 . Instead of systematically exploring the hyperparameter space, RL consists of a sequential decision-making process to find the best configuration based on the policy that guides the RL agent along the desired path. In terms of sustainably and efficiency, this solution is quite complex and requires many training iterations due to the expensive exploration and exploitation trade-off in RL. Moreover, instead of optimizing only the traditional model, also the RL model requires optimization. Pruning-based methods, such as the common Successive Halving Algorithm (SHA) Li et al. 2018 , aim to efficiently allocate computational resources to promising hyperparameter configurations in a parallelized environment. By iteratively terminating less promising configurations, such methods reduce the overall training time while maintaining a focus on high-performing settings. This bandit-based algorithm demonstrates a practical approach to balancing exploration and exploitation, making it well-suited for high-dimensional HPO scenarios. Throughout extended works like Li et al. 2020 , multi-fidelity optimization improves SHA

Read the complete text