A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm
Sakshi Kumari · 2026
Read A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm by Sakshi Kumari (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29273.txt.
Introduction
Sakshi Kumari and colleagues review adaptive optimization methods and propose a new optimizer variant. Adaptive learning-rate optimizers are widely used, yet they do not guarantee convergence. This gap motivated AMSGrad, introduced to probe the non-convergence behavior of Adam. The paper reviews popular methods such as Adam and AMSGrad, stressing their basic design ideas. To address their limits, it proposes C-Adam, a new variant based on the line-of-sight approach. A proof of convergence is provided, and the optimizer is validated through real-life numerical experiments.
Catalog record
| CREATOR | Sakshi Kumari |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40404 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm Sakshi Kumari Abstract ABSTRACT A crucial component of machine learning algorithms is minimizing loss functions with less computational cost and less oscillations. While adaptive learning rate-based optimizers have been widely used for real-world tasks, they do not guarantee convergence, which is why AMSGrad was later introduced to investigate the non-convergence behaviour of Adam. In this paper, popular adaptive optimization methods like Adam and AMSGrad are critically reviewed with an emphasis on their fundamental design concepts. To address limitations of the above mentioned optimizers, a new optimizer variant, C-Adam, is proposed based on the line of sight approach. A theoretical proof for convergence is also provided and the optimizer is validated through a number of real-life based numerical experiments. AMS Subject Classification (2020): 68T07, 68T20 Keywords: Machine Learning, Adaptive Moment Estimation, Adam, AMSGrad, Optimizer 1 Introduction The minimization of objective functions subject to certain constraints is a central problem in optimization theory and lies at the core of many ongoing developments in Artificial Intelligence [ 1 , 17 , 3 ] . For example, solving systems of non-linear partial differential equations using the highly non-convex function leads to several applications in the field of science and engineering [ 18 ] . An optimizer is an algorithm used to update the internal parameters of a model, such as weights and biases, and provide a point, called optimial point, that minimizes the loss function, which in turn works as an objective function in machine learning. There are various optimizers in use, of which the most relevant ones are discussed here. Stochastic Gradient Descent (SGD) has emerged as one of the most widely used optimizers, where gradients are approximated using randomly selected subset of data. Given a convex function f with unbiased stochastic gradients, bounded variance, and step sizes that satisfy the Robbins-Monro conditions, namely \sum_{t}\alpha_{t}=\infty\quad\text{and}\quad\sum_{t}\alpha_{t}^{2}<\infty , the SGD converges almost surely (that is with probability 1) to the optimal point [ 11 , 8 ] . In practice, SGD tends to search for an optimal point near local minima as weight updates are purely based on the current gradient of the loss function. This led to the introduction of the optimizer SGD with momentum to counter the SGD drawback by adding a velocity term that stores information about past gradients. Another limitation of SGD is that the learning rate is fixed, i.e., there is no control over individual parameter learning. AdaGrad [ 4 ] solves this limitation by adjusting the learning rate for each parameter based on the history of past gradients observed. AdaGrad tends to make adaptive learning rate strictly decreasing in nature by scaling the learning rate by a factor of (\sum_{t}g_{t}^{2})^{-1/2} , which raises yet another concern of learning rate decay in the case of dense gradients. To overcome this challenge of AdaGrad, RMSProp [ 16 ] and AdaDelta [ 20 ] were introduced. In order to address these issues simultaneously, Adam [ 7 ] was later introduced as a mix of RMSProp and SGD with momentum. This made Adam effective in adjusting the adaptive learning rate for individual parameters and in navigating complex, noisy, or non-stationary landscapes without being stuck at the local minima. Despite being one of the most popular optimizers [ 13 , 6 ] , Adam, with its own demerits, has shown a non-convergence nature due to the non-monotonic characteristics of the second order moment [ 12 ] . To ensure this, AMSGrad was introduced [ 12 ] retaining the maximum of second order moment observed and by scaling the learning rate using the past seen second moments during parameter updates. Inspite of resolving certain theoretical issues of Adam, AMSGrad was shown to lead to poor generalization in case of extreme learning rate [ 9 ] . AdaBound was then introduced, in which dynamic bounds on adaptive learning rates were applied to avoid extreme updates [ 9 ] . In AdaBound, a combination of Adam and SGD was used leading to early convergence and better generalization. The authors in [ 19 ] developed a method to address convergence and generalization issues in previous optimizers like RMSProp, Adam, AdaDelta. They then went on to develop a new adaptive optimization algorithm, YOGI, that controls the increase in effective learning rate for better performance, maintaining the theoretical guarantee of convergence. AMSGrad and its variants are widely used when stable convergence and reliable optimization are required in deep neural networks, and have shown effective results in several problem domains [ 14 , 10 ] . However, by forcing the second moment to be non-decreasing, AMSGrad might tend to keep the learning rate too small for too long, hindering the ability to escape saddle points or navigate complex, non-convex surfaces. This leads to a fundamental problem in the fix itself, which renders it too cautious. In search of an optimizer that guarantees the non-decreasing character of the second moment while limiting the decay in the adaptive learning rate caused by noisy gradients, a customized version of Adam, namely C-Adam, is proposed, that relaxes the aggressive maximization of the previously observed second moments in AMSGrad while ensuring the non-decreasing nature of the same. The now proposed optimizer tries to incorporate the theoretical correctness of AMSGrad with enhanced practical applications. To deliver this, we shift from point-wise update to “line of sight” update, ensuring that the optimizer accounts for the transitionary gradients between discrete steps. The structure of the paper is as follows. In Section 2 , all the preliminaries and notations are given. In Section 3 , the algorithm of C-Adam is discussed and the regret bound for the proposed optimizer is derived, mathematically ensuring the performance is either better or comparable with the existing optimizers. Examples where the suggested optimizer arrives at a final optimal point with nearly zero regret while both Adam and AMSGrad are unable to learn the solution surface, are also provided. The performance of C-Adam, in comparison with Adam and AMSGrad, on various real-life based examples are demonstrated in Section 4 . Concluding remarks and the possible drawbacks of C-Adam are discussed in Section 5 . 2 Preliminaries Throughout the paper, matrix A denotes a positive definite d\times d matrix, unless otherwise mentioned. Similarly for x\in\mathbb{R}^{d} denotes a d -dimensional vector. \sqrt{x} and x^{2} are used to denote the vectors (\sqrt{x_{i}})_{i} and (x_{i}^{2})_{i} , the entrywise square root and square respectively of the vector x . Given any two vectors x,y\in\mathbb{R}^{d} , max \{x,y\} represents the vector (max \{x_{i},y_{i}\})_{i} . In a similar fashion, x\leq y or x=y represent the entrywise inequality and equality of the vectors. A^{\frac{1}{2}} is used to represent \sqrt{A} , which is the matrix all of whose entries are the square roots of the corresponding entries of A . \|A_{i}\|_{2} is used to denote the l_{2} -norm of the i^{th} row of A . The projection operation \Pi_{\Omega,A}(y) , where \Omega is a set with bounded diameter \Omega_{\infty} , is defined as \Pi_{\Omega,A}(y)=\text{arg min}_{x\in\Omega}\|A^{\frac{1}{2}}(x-y)\| , for y\in\mathbb{R}^{d} . In the online optimization setup, at each time step t , the algorithm picks a vector (of parameters of the model to be learned) x_{t}\in\Omega , where \Omega\subset\mathbb{R}^{d} is the feasible set of points.