Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification Journal: Machine Learning with Applications
Veerababu Dharanalakota, Ashwin Arvind Raikar, Prasanta Kumar Ghosh · 2025
Read Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification Journal: Machine Learning with Applications by Veerababu Dharanalakota, Ashwin Arvind Raikar, Prasanta Kumar Ghosh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21749.txt.
Introduction
Veerababu Dharanalakota, Ashwin Raikar, and Prasanta Ghosh present a dynamic learning rate scheduler in a paper from two thousand twenty-five. The learning rate strongly shapes neural network training, yet it is usually fixed in advance. Complex problems often need the rate to adapt as training unfolds.
Their scheduler adjusts the rate from the loss values seen during training itself. They test it on physics-informed neural networks and on image classification with plain and convolutional networks. In these tests, the scheduler sped up training and made it more stable. The authors pitch it as a simple fix for a common tuning headache. Adapting the rate to the loss curve replaces guesswork with feedback.
Catalog record
| CREATOR | Veerababu Dharanalakota, Ashwin Arvind Raikar, Prasanta Kumar Ghosh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21749
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification Journal: Machine Learning with Applications Veerababu Dharanalakota, Ashwin Arvind Raikar, Prasanta Kumar Ghosh Abstract Training neural networks can be challenging, especially as the complexity of the problem increases. Despite using wider or deeper networks, training them can be a tedious process, especially if a wrong choice of the hyperparameter is made. The learning rate is one of such crucial hyperparameters, which is usually kept static during the training process. Learning dynamics in complex systems often requires a more adaptive approach to the learning rate. This adaptability becomes crucial to effectively navigate varying gradients and optimize the learning process during the training process. In this paper, a dynamic learning rate scheduler (DLRS) algorithm is presented that adapts the learning rate based on the loss values calculated during the training process. Experiments are conducted on problems related to physics-informed neural networks (PINNs) and image classification using multilayer perceptrons and convolutional neural networks, respectively. The results demonstrate that the proposed DLRS accelerates training and improves stability. † † highlights: An algorithm is proposed to improve the efficiency of the network training process. The method adjusts the learning rate based on the loss values. Performance is test against the standard backpropagation algorithm. Algorithm is applied to solve PINNs, and image classification problems. 1 Introduction The learning rate is an important hyperparameter that determines the convergence of the loss function towards an optimal value while training a neural network. Several other hyperparameters such as the number of epochs, the size of the batch, and the split of the data into training and test sets, have an impact on the training process as well. However, the learning rate controls the size of the update made to the weights during training ( Yu & Zhu, 2020 ) . Hence, it is a crucial factor in determining the convergence rate and precision of the model ( Magoulas et al., 1999 ) . A learning rate that is too high can cause the model to diverge, while values that are too low can cause the model to take a long time to converge, or it may never converge at all. The optimal learning rate depends on the specific problem and the architecture of the neural network ( Smith, 2018 ) . Finding the optimal learning rate manually during the training has several drawbacks, such as interrupting the training process, uncertainty of what value to choose at a given point in time during the training, the need to continuously monitor the model performance metrics, etc. These tasks are time-consuming and tedious. Hence, there is a need for an algorithm that can automatically decide and adapt the learning rate. The dynamic learning rate scheduler (DLRS) algorithm proposed in this paper helps to automatically adjust the learning rate hyperparameter during training based on the observation of previous loss values. This approach considerably improves the overall training accuracy and helps to achieve faster and better convergence. The researchers proposed learning rate algorithms that adapt on the basis of the loss values and their gradients in the past. In Weir, 1991 , an algorithm for the optimum step length in an adaptive learning rate was proposed using the loss values and their gradients. Later, Behera et al., 2006 proposed an alternative method to update the learning rate using the Lyaponav stability theory and compared its performance against standard backpropagation and extended Kalman filtering algorithms on three benchmark problems. Li et al., 2009 provided a comprehensive discussion on theoretical differences between the standard backpropagation algorithm and the improved adaptive learning rate algorithms in his work. The use of search algorithms and advanced concepts of calculus for dynamically updating the learning rate can also be seen in the literature. Takase et al., 2018 used the tree search algorithm to find the learning rate that produces the minimum loss value. On the other hand, Chen et al., 2024 used fractional-order derivatives to evaluate the gradients, thereby finding appropriate step size control to update the learning rate. The DLRS algorithm proposed in this paper also takes advantage of the loss values and their gradient information to dynamically update the learning rate. In line with the literature reported earlier, the performance of the proposed algorithm is compared with the standard backpropagation algorithms. For this purpose, the following problems are considered. 1. Physics-informed neural networks (PINNs) ( Veerababu & Ghosh, 2024 ) 2. Image classification using MNIST dataset ( Deng, 2012 ) 3. Image classification using CIFAR-10 dataset ( Krizhevsky & Hinton, 2009 ) The paper is organized as follows. Section 2 provides an introduction to adaptive learning rate algorithms and briefly discusses recent methods published in the literature. The proposed DLRS algorithm is discussed in detail in Section 3 . The application of the proposed algorithm on different problems and the comparison of its performance against established back propagation algorithms are presented in Section 4 . The paper is concluded in Section 5 with the final remarks. 2 Learning Rate Primer Let us consider the parameter update rule in stochastic gradient descent (SGD), which forms the basis for the parameter update algorithms. According to SGD, the parameters are being updated as per the rule ( Amari, 1993 ) \theta_{t+1}=\theta_{t}-\eta\nabla\mathcal{L}(\theta_{t};x^{(i)},y^{(i)}). (1) Here, \theta_{t} are the parameters of the neural network (weights and biases) at the iteration t , \eta is the learning rate, \mathcal{L} is the loss function, \nabla represents the gradients with respect to the parameters \theta_{t} , x^{(i)} and y^{(i)} represent the features associated with the input data and its corresponding labels, respectively. In general, increasing the value of \eta can expand the exploration range of the optimal solution, but excessively high values of \eta can hinder convergence toward a global optimal solution ( Wilson & Martinez, 2001 ) . A viable strategy to mitigate this challenge involves reducing the learning rate during the training process. However, reducing the learning rate during the training process considerably increases the computational cost ( Golmant et al., 2018 ) . In addition to the incorrect choice of hyperparameters, a phenomenon that hinders the learning process is the problem of exploding gradients that occurs during the training process ( Philipp et al., 2017 ) . 2.1 Problem of Exploding Gradients The problem of exploding gradients occurs when the gradients of the loss function become extremely large, especially at the last hidden layer of the network during the training process ( Philipp et al., 2017 ) . These large gradients propagate from the output layer towards the input layer. At each layer, the incoming gradients multiply with the local gradients, resulting in much larger gradients. These gradients make the parameter update procedure too drastic, leading to overshooting or oscillations of the loss function values. Consequently, the training process becomes unstable and never converges to the optimal value ( Philipp et al., 2017 ) . To mitigate the problem of exploding gradients, several techniques have been developed, such as gradient clipping ( Chen et al., 2020 ) , weight regularization ( Wan et al., 2013 ) , proper-weight initialization ( Narkhede et al., 2022 ) , batch normalization ( Bjorck et al., 2018 ) , etc. In addition to these