Neural Velocity for hyperparameter tuning

Gianluca Dalmasso † , Andrea Bragagnolo ‡ , Enzo Tartaglione § , Attilio Fiandrotti † , Marco Grangetto † · 2025

Read Neural Velocity for hyperparameter tuning by Gianluca Dalmasso † , Andrea Bragagnolo ‡ , Enzo Tartaglione § , Attilio Fiandrotti † , Marco Grangetto † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05309.txt.

Introduction

Gianluca Dalmasso and co-authors steer neural network training by watching how fast its neurons change, in a 2025 paper. Tuning a learning rate schedule and knowing when to stop usually means babysitting validation loss on held-out data. Their approach, NeVe, replaces that signal with neural velocity, the rate at which each neuron's transfer function is moving.

A settling network's neurons slow down, so velocity reads convergence directly. Remarkably, it can be sampled by feeding the network pure noise, no labeled validation set required. The training loop then eases the learning rate and calls the stop based on that internal motion. The authors report the metric's promise for cheaper, more automatic training runs.

Catalog record

CREATORGianluca Dalmasso † , Andrea Bragagnolo ‡ , Enzo Tartaglione § , Attilio Fiandrotti † , Marco Grangetto †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Velocity for hyperparameter tuning Gianluca Dalmasso † , Andrea Bragagnolo ‡ , Enzo Tartaglione § , Attilio Fiandrotti † , Marco Grangetto † Abstract Hyperparameter tuning, such as learning rate decay and defining a stopping criterion, often relies on monitoring the validation loss. This paper presents NeVe, a dynamic training approach that adjusts the learning rate and defines the stop criterion based on the novel notion of “neural velocity”. The neural velocity measures the rate of change of each neuron’s transfer function and is an indicator of model convergence: sampling neural velocity can be performed even by forwarding noise in the network, reducing the need for a held-out dataset. Our findings show the potential of neural velocity as a key metric for optimizing neural network training efficiently. I Introduction Artificial intelligence has made remarkable progress in recent years, driven by the availability of data and powerful deep learning techniques. However, limited training data remains a major challenge in developing robust, generalizable models. This issue is especially critical in small-data scenarios, where acquiring labeled samples is costly or impractical [ 1 , 2 ] . Training with small datasets often leads to models that memorize specific details rather than learning general patterns, resulting in poor generalization and reduced adaptability to real-world variability. These limitations make deploying such models in dynamic environments particularly risky, highlighting the importance of addressing this challenge. Fig. 1 : When training a neural network, the neurons in a model change with a progressively decreasing velocity as they reach a plateau in the performance: we can exploit this property to scale the learning rate and to stop the training. Hyperparameter tuning is critical to training successful neural networks, even in scenarios with limited data. Key hyperparameters include the learning rate, the strategy used to adjust it over time, and termination criteria, which determine when to stop training to prevent overfitting and reduce training time. Hyperparameters can be optimized through various methods, ranging from manual tuning based on expert guidance to more sophisticated and computationally intensive techniques like grid search or gradient-based approaches [ 3 , 4 ] , which automate the process. However, most tuning methods require holding out a validation set from the training data, which presents notable challenges. With limited annotated data, reserving samples for validation reduces the data available for training, potentially impairing the model’s learning ability. Furthermore, validation sets may not accurately reflect real-world scenarios, leading to misleading evaluations [ 5 ] . To address these issues, there is growing interest in validation-free learning approaches, which aim to optimize hyperparameters without relying on separate validation samples, offering a promising alternative in data-constrained environments. Another critical component of training successful neural networks is the stopping criteria. Early stopping is a common technique that halts training when a model’s performance on a validation set stops improving, often using a patience parameter to define the number of epochs without improvement [ 6 ] . Advanced methods, such as that in [ 7 ] , refine this process by adapting stopping criteria based on training dynamics, improving efficiency and generalization. However, these methods rely on a validation set, which can lead to biased decisions if it does not represent the broader data distribution accurately. More recent approaches leverage noisy labels [ 8 ] , avoiding the use of validation data by tracking changes in the model’s predictions during training. These methods determine the optimal stopping point by detecting when the model starts overfitting mislabeled data, thus improving robustness in noisy settings. This paper introduces NeVe, a novel method to optimize learning rate decay and stopping criteria in neural network training without requiring a held-out validation set. NeVe leverages the concept of neural velocity, defined as the rate at which neurons change their input-output behavior during training. The method dynamically adjusts the learning rate when a neuron’s change stabilizes, signaling it has reached a “meta-stable” state (Fig. 1 ). Training is terminated when neural velocity approaches zero, indicating convergence to a stable solution. By utilizing an auxiliary dataset of random samples, NeVe not only minimizes reliance on validation data but also addresses the inefficiencies that arise in data-scarce scenarios, enabling a more efficient and adaptive training procedure while preserving valuable training data for model optimization. The main contributions and findings presented in this paper are summarized in the following key points: • First Validation-less Approach Using Neural Velocity : NeVe is, to the best of our knowledge, the first method to use the concept of neural velocity to both scale the learning rate decay and provide a stopping criterion without a validation set (Sec. III-A ). • Plug-and-Play Strategy : Unlike other methods, NeVe is a versatile, plug-and-play approach that can automatically adapt to different architectures, datasets, and optimizers (Sec. III-A , Sec. IV-D ). This paper is organized as follows. In Section II , we review related works, providing a discussion of their strengths and weaknesses. Section III focuses on our proposed technique, the algorithm, and an overview of its key hyperparameters. Section IV presents experimental results, detailing the characteristics of our method, the experimental setup, and a thorough analysis of the results, including an ablation study. Finally, in Section V , we summarize our findings, revisit the proposed technique, and discuss the broader implications of this work. II Related Works In this section, we provide a primer on hyperparameter tuning strategies, learning rate decay policy, and the stopping criteria that are especially relevant to this work. Hyperparameter tuning is a widely studied topic in machine learning, focusing on finding the optimal configuration to maximize model performance. While grid search [ 9 , 10 ] provides an exhaustive exploration of the hyperparameter space, it becomes computationally expensive in high-dimensional settings due to the curse of dimensionality. Alternative methods, such as Bayesian optimization [ 11 ] , use probabilistic models to map hyperparameters to objectives, achieving better results than grid or random search. Evolutionary algorithms [ 12 , 13 ] offer another approach, iteratively improving a population of hyperparameter configurations by selecting and refining the fittest candidates. Gradient-based techniques [ 14 , 3 , 15 ] , particularly suited for neural networks, enable direct optimization of hyperparameters, such as the learning rate and early stopping thresholds, alongside model parameters. These methods reduce sensitivity to initial choices and automate hyperparameter adjustment during training. The learning rate is often regarded as the most influential hyperparameter [ 16 ] , and unlike others, it is rarely kept constant during training. Often it is dynamically adapted during training to improve convergence and generalization [ 17 ] . Fixed learning rates are rarely effective, as high initial rates help explore the loss landscape, while lower rates stabilize convergence. Strategies such as step-wise decay, exponential decay, or validation-based adjustments [ 18 , 19 , 20 ] are commonly used to control the learning rate over time. Sophisticated learning algorithms with adaptive learning rates have been proposed [ 21 ] , but even with these methodologies, a learning rate decay policy is still applied [ 22 ] . Early stopping is a widespread technique used to limit the overfitting of neural networks and lessen the training cost, thanks to early termination of the training process

Read the complete text