A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
Jacob Adkins, Michael Bowling, Adam White · 2024
Read A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning by Jacob Adkins, Michael Bowling, Adam White (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07165.txt.
Introduction
Tuning choices can make or break a reinforcement learning agent, and Jacob Adkins, Michael Bowling, and Adam White study that fact in a 2024 paper. Small changes in a setting can cause large swings in scores, and the best setting often shifts from one task to the next. The field lacks a shared, scalable way to measure this sensitivity.
The authors propose a practical method to study, compare, and assign a number to that sensitivity for a fixed set of tasks. They then use the method on several variants of the PPO algorithm that differ in how they normalize data. The results suggest that some gains blamed on better methods may in fact come from heavier tuning.
Catalog record
| CREATOR | Jacob Adkins, Michael Bowling, Adam White |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07165
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning Jacob Adkins, Michael Bowling, Adam White Abstract The performance of modern reinforcement learning algorithms critically relies on tuning ever increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm’s performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning. 1 Introduction The performance of reinforcement learning algorithms critically relies on the tuning of numerous hyperparameters. With the introduction of each new algorithm, the number of these critical hyperparameters continues to grow. Consider the progression of value-based reinforcement learning algorithms, starting from DQN ( Mnih et al., 2015 ) , which has 16 hyperparameters that the practitioner must choose, to Rainbow ( Hessel et al., 2018 ) with 25 hyperparameters. This increase can be observed in Figure 1 . Figure 1: A count of hyperparameters for different reinforcement learning algorithms proposed over the last decade. We include value-based, policy-gradient, and model-based methods. The counts do not include hyperparameters controlling the network architectures, such as number of layers, activation functions, etc. See Appendix B for details on how hyperparameters were counted. This proliferation is problematic because performance can vary drastically with respect to hyperparameters across environments. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve the reported good performances ( Franke et al., 2021 ; Eimer et al., 2022 ; Eimer et al., 2023 ; Patterson et al., 2024 ) . Generally speaking, hyperparameter tuning requires a combinatorial search and thus many published results are based on a mix of default hyperparameter settings and informal hand-tuning of key hyperparameters like the learning rate. Our standard evaluation methodologies do not reflect the sensitivity of performance to hyperparameter choices, and this is compounded by a lack of suitable metrics to characterize said sensitivities. There are many different ways one could characterize performance with respect to hyperparameter choices in reinforcement learning, but the community lacks an agreed standard. Hyperparameter sensitivity curves, such as those found in the introductory textbook ( Sutton & Barto, 2018 ) , summarize performance with respect to several values of a key hyperparameter producing U-shaped curves. Sensitivity curves do not work well with many dimensions of hyperparameters, nor is there a well-established way to use them to summarize performance in multiple environments. If computation is of no object, then performance percentiles can be used to compute the likelihood that an algorithm will perform well if its hyperparameters are randomly sampled from some distribution ( Jordan et al., 2020 ) . Although very general, this approach does not reflect how practitioners tune their algorithms. The Cross-environment Hyperparameter Benchmark ( Patterson et al., 2024 ) compares algorithms by a mean normalized performance score across environments but ultimately focuses on the possibility of finding a single good setting of an algorithm’s hyperparameters that performs well rather than characterizing sensitivity. What performance-only metrics lack is a measurement of what proportion of realized performance is due to per-environment hyperparameter tuning. We propose an empirical methodology to better understand the interplay between hyperparameter tuning and an reinforcement learning agent’s performance. Our methodology consists of two metrics and graphical techniques for studying them. The first metric, called an algorithm’s hyperparameter sensitivity , measures the degree to which an algorithm’s peak reported performance relies upon per-environment hyperparameter tuning. This metric captures the degree to which per-environment tuning improves performance relative to the performance of the best-fixed hyperparameter setting across a distribution of environments. The second metric, named effective hyperparameter dimensionality , measures how many hyperparameters must be tuned to achieve near-peak performance. It is often unclear how important specific hyperparameters are and if they should be included in the tuning process. These two metrics can help us better understand existing algorithms and drive research toward algorithmic improvements that reduce hyperparameter sensitivity. We validate the utility of our methodology by studying several variants of PPO ( Schulman et al., 2017 ) that have been purported to reduce hyperparameter sensitivity and increase performance. We performed a large-scale hyperparameter study over variants of the PPO algorithm consisting of over 4.3 million runs (13 trillion environment steps) in the Brax MuJoCo domains ( Freeman et al., 2021 ) . We investigate the relationship between performance and hyperparameter sensitivity with several commonly used normalization variants paired with PPO. We found that normalization variants, which increased PPO’s tuned performance, also increased sensitivity. Other normalization variants had negligible effects on performance and marginal effects on hyperparameter sensitivity. This result contrasts the view that normalization makes reinforcement learning algorithms easier to tune and, as a consequence, results in improved performance. 2 Problem Setting and Notation We formalize the agent-environment interaction as a Markov Decision Process (MDP) with finite state space \mathcal{S} and action space A , bounded reward function \mathcal{R} : \mathcal{S}\times A\times\mathcal{S}\rightarrow\mathcal{R}\subset\mathbb{R} , transition function \mathcal{P}:\mathcal{S}\times A\times\mathcal{S}\rightarrow[0,1] , and discount factor \gamma\in[0,1] . At each timestep t , the agent observes the state S_{t} , selects an action A_{t} , the environment outputs a scalar reward R_{t+1} and transitions to a new state S_{t+1} . The agent’s goal is to find a policy, \pi:\mathcal{A}\times\mathcal{S}\rightarrow[0,1] , that maximizes the expected return, G_{t}\doteq R_{t+1}+\gamma R_{t+2}+... , in all states: \mathbb{E}_{\pi}[G_{t}|S_{t}=s] for all s\in\mathcal{S} . Most reinforcement learning agents learn and use approximate value functions in order to improve the policy through interaction with the world. The state-value function v_{\pi} : \mathcal{S}\rightarrow\mathbb{R} is the state conditioned expected return following policy \pi defined as v_{\pi}(s)\doteq\mathbb{E}_{\pi}[G_{t}|S_{t}=s] . Similarly, the action-value function q_{\pi} : \mathcal{S}\times A\rightarrow\mathbb{R} provides the state and action conditioned expected return following policy \pi defined by q_{\pi}(s,a)\doteq\mathbb{E}_{\pi}[G_{t}|S_{t}=s,A_{t}=a] . The advantage function A_{\pi}(s,a)\doteq q_{\pi}(s,a)-v_{\pi}(s) describes how much better taking an action a in state s is rather than sampling an action according to \pi(\cdot|s) and following policy \pi afterward. Given an estimate of the value function \hat{v} , an agent can update estimates of the value of states based on estimates of the values of successor states. An n -step return is defined as G_{t:t+n}\doteq R_{t+1}+\gamma R_{t+2}+\cdots+\gamma^{n-1}R_{t+n}+\gamma^{n}\hat{v}\left(S_{t+n}\right) . A mixture of n -step returns, called a truncated- \lambda return can be created by weighting n -step returns by a factor \lambda\in[0,1] , G_{t:t+n}^{\lambda}\doteq(1-\lambda)\sum_{j=1}^{n-1}\lambda^{j-1}G_{t:t+j}+\lambda^{n-1}G_{t:t+n} . Truncated- \lambda returns are often used in reinforcement learning algorithms such as PPO ( Schulman et al., 2017 ) to estimate state values and state-action advantages, which are used to approximate the value function and improve the policy. Of particular interest to this work is the setting where an empiricist is evaluating a set of algorithms \Omega across a distribution of environments \mathcal{E} . Each algorithm \omega\in\Omega is presumed to have some number of hyperparameters n(\omega) .