HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks

Jonathan Gornet, Yiannis Kantaros, Bruno Sinopoli · 2025

Read HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks by Jonathan Gornet, Yiannis Kantaros, Bruno Sinopoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19382.txt.

Introduction

Jonathan Gornet, Yiannis Kantaros, and Bruno Sinopoli present a 2025 method for tuning learning agents while they train. Agents that learn by trial and error need many settings chosen well. Poor settings slow training and can make results swing. Tuning by hand or by grid search wastes time and compute.

Their method, called HyperController, treats tuning as a hidden system that shifts in a straight-line way. A classic filter then learns a compact map of the score for each setting and picks the next one. In tests on five gym tasks, the method earned the top middle score in four. Training was faster and steadier than with rival methods.

Catalog record

CREATORJonathan Gornet, Yiannis Kantaros, Bruno Sinopoli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19382

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks Thanks: Jonathan Gornet, Yiannis Kantaros, Bruno Sinopoli Abstract We introduce Hyperparameter Controller (HyperController), a computationally efficient algorithm for hyperparameter optimization during training of reinforcement learning neural networks. HyperController optimizes hyperparameters quickly while also maintaining improvement of the reinforcement learning neural network, resulting in faster training and deployment. It achieves this by modeling the hyperparameter optimization problem as an unknown Linear Gaussian Dynamical System, which is a system with a state that linearly changes. It then learns an efficient representation of the hyperparameter objective function using the Kalman filter, which is the optimal one-step predictor for a Linear Gaussian Dynamical System. To demonstrate the performance of HyperController, it is applied as a hyperparameter optimizer during training of reinforcement learning neural networks on a variety of OpenAI Gymnasium environments. In four out of the five Gymnasium environments, HyperController achieves highest median reward during evaluation compared to other algorithms. The results exhibit the potential of HyperController for efficient and stable training of reinforcement learning neural networks. I Introduction Hyperparameter optimization, a subarea of Automated Machine Learning [ 1 ] , is critical for improving machine learning models. It focuses on finding the optimal configuration of hyperparameters, such as the learning rate or the batch size [ 2 ] . However, hyperparameter optimization cost functions may be unknown, expensive to compute, or nonsmooth (see details in [ 3 ] ), rendering many gradient-based optimization method such as stochastic gradient descent [ 4 ] or ADAM [ 5 ] , infeasible. Bergstra et al. [ 2 ] introduced grid and random search methods for hyperparameter optimization. This has provided a foundation for other hyperparameter optimization methods which have drawn inspirations from results in stochastic multi-armed bandits [ 6 , 7 ] (which study the problem of decision-making under uncertainty), genetic algorithms [ 8 , 9 , 10 , 11 ] , and population-based approaches [ 12 ] . Traditionally, hyperparameter optimization was implemented prior to the training process, using the same configuration throughout the whole training process [ 2 , 7 , 13 ] . Recently, methods have shifted to adjusting hyperparameters during the training process [ 14 , 15 , 16 , 12 , 17 ] . This paper draws inspiration from Bayesian optimization due to its success such as its impact on AlphaGo’s training [ 18 ] and other methods proposed in [ 19 , 20 , 21 , 22 ] . Bayesian optimization methods focus on estimating the cost function while also searching for the optimal value. A key result in the area is the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm proposed by [ 23 ] , which balances exploration and exploitation during optimization. Hybrid forms have appeared where elements from other optimization techniques have been integrated into Bayesian optimization such as random search [ 24 ] , stochastic multi-armed bandit algorithms [ 7 ] (an extension of [ 6 ] ), and distributed optimization [ 25 , 26 ] . This paper addresses the shortcomings of Bayesian optimization within the scope of Reinforcement Learning (RL). RL methods such as Trust Region Policy Optimization (TRPO) introduced by [ 27 ] , later inspiring Proximal Policy Optimization (PPO) by [ 28 ] , have been proposed for monotonic improvement of the RL neural network with little hyperparameter tuning. Previously, a method called Population-Based Bandits (PB2) in [ 17 ] , which draws inspiration from Time-Varying Gaussian Process Upper Confidence Bound (TV-GP-UCB) in [ 29 ] , have proposed to adjust hyperparameters online during neural network training in conjunction with PPO to accomplish better monotonic improvement. However, the authors in [ 17 ] assume that the cost function evolves as a one-step autoregressive process, which may be too rigid for accurate prediction. In addition, both PB2 and other Bayesian optimization methods struggle with the curse of dimensionality [ 3 ] . Finally, Bayesian optimization can become computationally expensive, requiring \mathcal{O}\left(n^{3}\right) work to invert a n\times n matrix where n is the number of samples collected. Results in [ 30 ] have lowered the computational complexity to \mathcal{O}\left(n\log^{2}n\right) work, but further improvements are needed. To address these issues, this paper proposes the Hyperparameter Controller (HyperController), inspired by stochastic control theory. The key contribution of HyperController is its reduction in the required computational work by learning a more efficient representation of the hyperparameter’s objective function to obtain efficient computation and finer control of the hyperparameters, ensuring that the RL neural networks obtains fast training and consistent improvement. This is contrasted with Bayesian optimization, which relies on the prior distribution of the hyperparameter’s objective function. Therefore, HyperController obtains faster hyperparameter optimization. The contributions of this paper are summarized below. • Inspired by prior work in Bayesian optimization, we model the hyperparameter optimization problem as an unknown Linear Gaussian Dynamical System (LGDS), a mathematical model of a system with a state variable evolving linearly over time. The LGDS output is an objective function value given a hyperparameter configuration. • We introduce a method to discretize the LGDS state variable, allowing optimal one-step prediction of the LGDS output using the Kalman filter. • Given the unknown LGDS parameters, we propose learning a representation of the Kalman filter. This representation is computationally efficient to learn, requiring only \mathcal{O}\left(s^{3}\right) work where s is the size of the representation such that s\ll n . In addition, learning the representation allows finer control over the hyperparameters to obtain consistent improvement of the RL neural networks. • HyperController uses two strategies: optimizes each hyperparameter separately instead of searching for the global hyperparameter configuration and only selects the predicted optimal configuration according to the learned representation. This implementation is utilized to avoid searching a large hyperparameter space relative to the training iterations. • To quantify the performance of HyperController, we derive the upper bound on regret, which is the cumulative difference between the optimal objective function’s response and its response based on the method’s selected hyperparameter configuration. • We validate our theoretical results by using HyperController as a hyperparameter optimizer during training of RL neural networks within the OpenAI Gymnasium. The rest of the paper is structured as follows. Section II reviews the mathematical model of the hyperparameter optimization problem and how to approximate it. In Section III , we discuss how to learn the model, leading to Section IV which is an introduction of the HyperController method. Section V implements the HyperController method for training a RL neural network and compares it to a number of well-known results in the area. Section VI concludes the paper and discusses future directions. Notation: For any x\in\mathbb{R}^{n} and y\in\mathbb{R}^{n} , we have the inner product \left\langle x,y\right\rangle=x^{\top}y\in\mathbb{R} . For the natural number i\in\mathbb{N} , we have [i]\triangleq\{1,2,\dots,i\} . II Problem Formulation For this section, we left the details of the problem formulation derivation in Appendices -A , -B , and -C to help the flow of the paper. When considering hyperparameter optimization in machine learning, there exists two variables to optimize, \theta(t)\in\mathcal{X}\subseteq\mathbb{R}^{m} and \zeta(t)\in\mathcal{Y}\subseteq\mathbb{R}^{h} where t\in[n]\triangleq\{1,2,\dots,n\} is the iteration number of the machine learning training process. The variable \theta(t)\in\mathcal{X} is the machine learning model’s parameters (i.e. neural network weights) while \zeta(t)\in\mathcal{Y} would

Read the complete text