An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
Saurabh Saini 1 , Kapil Ahuja 1 2 § , Thomas Wick 2 , Saurav Kumar 3 · 2026
Read An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning by Saurabh Saini 1 , Kapil Ahuja 1 2 § , Thomas Wick 2 , Saurav Kumar 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21968.txt.
Introduction
Saurabh Saini and co-authors improve an adaptive PID optimizer for deep learning, combining control theory with the Adam optimizer. Momentum-based stochastic gradient descent suffers noisy, varying gradients and overshoot, and Adam fixed the noise but not the overshoot. Their IAdaPID-ADG optimizer merges a PID-based extension of Adam with an adaptive gradient idea, targeting both flaws at once. Tested on MNIST, CIFAR10, and real-world IARC and AnnoCerv datasets, it substantially beats competing optimizers, and an ablation study on MNIST shows each added part contributes.
Catalog record
| CREATOR | Saurabh Saini 1 , Kapil Ahuja 1 2 § , Thomas Wick 2 , Saurav Kumar 3 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39385 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21968
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning § Corresponding author: Kapil Ahuja ([email protected]) S. Saini: [email protected]; T. Wick: [email protected]; S. Kumar: [email protected] Saurabh Saini 1 , Kapil Ahuja 1 2 § , Thomas Wick 2 , Saurav Kumar 3 Abstract Optimization is essential in deep learning. The foundational method upon which most optimizers are built is momentum-based stochastic gradient descent. However, it suffers from two key drawbacks. First , it has noisy and varying gradients, and second , it has an overshoot phenomenon. To address noisy gradients, Adam was proposed, which remains the most widely used adaptive optimizer. To address the overshoot phenomenon, a control-theory-based PID optimizer was proposed. To tackle both the limitations within a single framework, several variants of Adaptive PID (AdaPID) have recently been proposed. Although AdaPID performs well, it still inherits two critical drawbacks from Adam, namely convergence and stability issues. In this work, we address both these limitations. To fix the convergence issue, we uniquely integrate the idea of using a non-increasing effective learning rate into AdaPID (originally proposed in AMSGrad, an extension of Adam). To fix the stability issue, we innovatively integrate a gradient difference based modulation factor into AdaPID (originally proposed in DiffGrad, another extension of Adam). Combining both these ideas in AdaPID, results in our novel IAdaPID-ADG optimizer. We evaluate our proposed optimizer on multiple datasets, including benchmark datasets (MNIST and CIFAR10) and real-world datasets (IARC and AnnoCerv). The IAdaPID-ADG substantially outperforms all competing optimizers. Additionally, we perform an ablation study on the MNIST dataset to demonstrate the contribution of each added component. I Introduction Deep learning has emerged as a transformative paradigm across a wide range of domains, including computer vision [ 15 , 16 ] , image processing [ 6 , 2 ] , signal processing [ 28 , 29 ] , robotics [ 19 ] , and natural language processing [ 9 , 4 ] . This success is largely driven by the availability of large-scale datasets and powerful GPU-based computational resources. At the heart of every deep learning system lies the training process. From an optimization perspective, training amounts to minimizing a loss function over a high-dimensional parameter space. The choice of optimizer directly governs its speed, stability, and quality of convergence, and therefore design of efficient and reliable optimization algorithms is a fundamental challenge in deep learning research. Stochastic Gradient Descent (SGD) is the classical foundation of modern deep learning optimizers [ 24 ] . Despite its simplicity and strong generalization properties, SGD suffers from slow convergence in large-scale settings. This is because it relies solely on the current gradient, without using any historical information. To address this, momentum was introduced, which accumulates past gradients to accelerate and stabilize training [ 22 , 21 ] . However, momentum-based SGD still has two key drawbacks. First is noisy and varying gradients across parameters. A single global learning rate cannot handle this well, as different parameters may require very different step sizes during training. Adam [ 13 ] addressed this by assigning an adaptive learning rate to each parameter. The second is the overshoot phenomenon, where accumulated momentum drives parameters past the optimal point, causing a back-and-forth movement around the optimum. The PID optimizer [ 26 ] addressed this by regulate parameter updates via proportional, integral, and derivative terms, inspired by classical control theory. To tackle both the limitations at once, multiple variants of AdaPID [ 27 , 5 , 11 ] have been proposed, combining the adaptive learning rate idea with the PID control structure. Despite its strengths, AdaPID inherits two critical drawbacks from Adam, namely convergence and stability issues. We address both with the following contributions: (a). To address the convergence issue in AdaPID, we draw inspiration from AMSGrad [ 23 ] , which replaces the second-moment estimate with its running maximum, ensuring a non-increasing effective learning rate. We uniquely integrate this idea into AdaPID. (b). To address the stability issue in AdaPID, we derive the idea from DiffGrad [ 7 ] , which introduces a modulation (scaling) factor based on the difference between the current and previous gradient. We innovatively integrate this idea into AdaPID. (c). We combine both the above ideas into AdaPID, resulting in our novel Improved AdaPID AMSDiffGrad (IAdaPID-ADG) optimizer. (d). We test IAdaPID-ADG on two types of datasets. The first includes benchmark datasets, namely MNIST and CIFAR10, while the second consists of real-world cervical cancer datasets, including IARC and AnnoCerv. The performance of our optimizer is evaluated using three metrics, namely training loss, training accuracy, and testing accuracy. Across all datasets, IAdaPID-ADG achieves training losses that are multiple orders of magnitude lower than those of the competing optimizers. In addition, our optimizer achieves significantly higher training and testing accuracy with more stable curves than the existing optimizers. (e). We also conduct an ablation study on the MNIST dataset to analyze the contribution of each component of IAdaPID-ADG. The study demonstrates that every added component consistently improves the performance of our proposed optimizer. The rest of this manuscript consists of five more sections. In Section II , we review the existing literature and provide the necessary background. The proposed IAdaPID-ADG optimizer is described in Section III . In Section IV , we present the datasets and discuss the experimental results. The ablation study, examining the contribution of each component of the proposed optimizer, is provided in Section V . Section VI presents the conclusion and future directions of this work. II BACKGROUND OF THE INVENTION Deep learning optimizers are fundamentally based on gradient descent, with Stochastic Gradient Descent (SGD) serving as the core method for training neural networks [ 24 ] . These networks are trained using iterative algorithms that minimize a loss function \mathcal{L}(w) with respect to the parameter vector w\in\mathbb{R}^{d} . At iteration t , the SGD update is defined as follows: w_{t}=w_{t-1}-\eta g_{t}, (1) where \eta represents the learning rate controlling the step size and g_{t}=\nabla_{w}\mathcal{L}_{t}(w_{t-1}) denotes the gradient of the loss at iteration t . Polyak et al. [ 22 ] have shown that SGD can exhibit slow convergence and unstable updates. This is because it uses only the current gradient and does not use past information. To address this limitation, momentum was introduced to accelerate convergence [ 22 ] . It accumulates gradients from previous iterations, enabling faster and more stable movement along relevant directions. The momentum term m_{t} is computed as follows: m_{t}=\alpha m_{t-1}+g_{t}, (2) where \alpha denotes the momentum coefficient that controls the contribution of past gradients. By substituting the gradient term in ( 1 ) with the momentum term m_{t} , the update rule becomes w_{t}=w_{t-1}-\eta m_{t}. (3) The momentum-based SGD algorithm is outlined in Algorithm 1 . Given: Scalar \in\mathbb{N}_{0} ; t : iteration index; t=0 Given: Scalars \in\mathbb{R} ; • \alpha : first moment decay rate; \alpha=0.9 • \eta : learning rate; \eta=0.001 Given: Vectors \in\mathbb{R}^{d} ; • w_{t} : parameters (weights) at iteration t ; w_{0} = randomly initialized weights • g_{t} : gradient • m_{t} : first moment estimate; m_{0}=\mathbf{0} while not converged do 1: t=t+1 ; 2: g_{t}=\nabla_{w}\mathcal{L}_{t}(w_{t-1}) ; 3: m_{t}=\alpha m_{t-1}+g_{t} ; 4: w_{t}=w_{t-1}-\eta m_{t} ; end while Algorithm 1 Momentum-Based SGD Despite