Dynamic Momentum Recalibration in Online Gradient Learning
Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li · 2026
Read Dynamic Momentum Recalibration in Online Gradient Learning by Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06120.txt.
Introduction
Yao and colleagues view gradient updates through signal processing and argue that a fixed momentum rate skews the bias-variance balance in stochastic gradient descent. Their optimizer, SGDF, adjusts its gain at each step to refine gradient estimates by limiting mean-squared error. They argue the filtering idea can carry to other optimizers, and benchmark tests show SGDF beating standard momentum methods and matching or exceeding leading ones.
Catalog record
| CREATOR | Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 141020 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06120
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Momentum Recalibration in Online Gradient Learning Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li Abstract Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers. † † footnotetext: ⋆ Work initiated at 1 and further developed at 2 , \dagger Equal advising. 1 Introduction In deep learning optimization, the optimizer plays a pivotal role in refining model parameters to capture underlying data patterns, while strategically navigating complex loss landscapes 18 to identify regions that promote strong generalization 38 . Its choice profoundly affects training efficiency, convergence speed, generalization, and robustness to data shifts 2 , with suboptimal selections potentially causing slow or failed convergence, whereas effective ones accelerate learning and bolster model resilience 66 . Thus, the design and refinement of optimizers remain essential challenges in enhancing the capabilities of models. Stochastic Gradient Descent (SGD) 57 and its variants, including momentum-based methods 63 ; 71 and adaptive techniques like Adam 40 and RMSprop 32 , which have advanced training efficiency 7 . However, these approaches face challenges in high-dimensional, non-convex settings 27 , where adaptive methods often yield rapid convergence but suffer from poor generalization 39 . Efforts to mitigate this have led to Adam variants 8 ; 52 ; 49 ; 89 that refine adaptive learning rates, yet they fall short of fully closing the generalization gap, highlighting the ongoing need for innovative optimization strategies that better balance estimation accuracy and practical performance. Actually, the issues that arise from the optimizer during training, particularly in terms of optimization and generalization, are inherently tied to the trade-off between bias and variance 24 ; 58 . High bias leads to underfitting, while high variance results in overfitting. Similarly, the gradients used by the optimizer to update weights also face this challenge. Intuitively, high bias in the gradients may lead to convergence at a suboptimal plateau 84 ; 77 , while high variance can lead to instability in the optimization path, causing oscillations that hinder convergence 6 ; 20 . Therefore, a good optimizer should strike a balance between the bias and variance in its gradient estimates. From a statistical signal processing perspective, we analyze the mechanism behind optimizer updates. Specifically, we decompose the optimizer’s gradients used for updating model weight into bias and variance components. Then, We identify a key limitation in momentum-based optimization techniques supplemented with examining the statistical distribution of gradients within the model: they struggle to balance bias and variance components in gradients, often introducing a gradient shift phenomenon, which we term bias gradient estimate . This bias estimate, arising from fixed momentum coefficients, accumulates over time, leading to bias. As a result, the model may struggle to adapt to variations in curvature across different layers, resulting in suboptimal or directionally skewed updates 85 ; 17 . To address this issue, we introduce SGDF, a novel method that uses principles from Optimal Linear Filter to adjust gradient estimation dynamically. SGDF derives an optimal, time-varying gain to minimize mean-squared error in gradient estimation, balancing noise reduction with signal preservation. This filter mechanism provides a more accurate first-order gradient estimate and avoids the limitations of fixed momentum parameters, allowing SGDF to adjust dynamically throughout training. Additionally, SGDF’s flexibility extends to other optimization frameworks, which enhance performance across a range of tasks. Through extensive empirical validation across diverse model architectures and visual tasks, we demonstrate that SGDF consistently outperforms traditional momentum-based and variance reduction methods, achieving competitive or superior results relative to state-of-the-art optimizers. The main contributions can be summarized as follows: • We quantify the bias-variance trade-off in momentum-based gradient estimation (EMA and CM) using a unified SDE framework, revealing their static limitations. • We introduce SGDF, an optimizer that combines historical and current gradient data to estimate the gradient, addressing the trade-off between bias and variance in the momentum method. • We theoretically analyze the convergence property of SGDF in both convex optimization and non-convex stochastic optimization (Sec. 3.3 ), and empirically verify the effectiveness of SGDF (Sec. 4 ). • We preliminarily explore the extension of SGDF’s first-moment filter estimation to adaptive optimization algorithms ( e.g . , Adam), which shows a promising enhancement in their generalization capability (Sec. 4.2 ), surpassing traditional momentum-based methods. 2 The Gradient Estimation Dilemma 2.1 Bias and Variance Stochastic gradient-based optimization lies at the core of modern machine learning. We revisit this and found that it grapples with a fundamental challenge: the trade-off between gradient bias and variance. To dissect this dilemma, we begin by unifying two prominent momentum strategies under a single framework. The proof of this section can be found in Appendix A . Definition 2.1 . The unified momentum update rule is defined as: m_{t}=\beta m_{t-1}+ug_{t},\quad\theta_{t}=\theta_{t-1}-\alpha m_{t}, (1) where \alpha is the learning rate, \beta\in[0,1) is the momentum coefficient, and u\geq 1-\beta scales the current gradient. For all \theta\in\mathbb{R}^{d} , f_{t}(\theta)=f(\theta;\xi_{t}) denotes the stochastic objective at iteration t with data sample \xi_{t}\sim\mathcal{D} . The expected objective is f(\theta)=\mathbb{E}_{\xi}[f(\theta;\xi)] , and g_{t}=\nabla f_{t}(\theta_{t})+\epsilon_{t} (where \epsilon_{t}\sim\mathcal{N}(0,\sigma^{2}I) ) is the stochastic gradient. Specific cases include: • u=1-\beta : Exponential Moving Average (EMA), • u=1 : Classical Momentum (CM) 63 ; 71 . This formulation encapsulates EMA and CM, two cornerstones of gradient estimation, differing in how they weight the current gradient against historical trends. EMA through a balanced mean update, while CM aggressively incorporates the gradient direction. We dissect the nature of the two methods, quantified by the mean square error. Lemma 2.2 . For any gradient estimator \hat{g}_{t}=\mathcal{A}(g_{1},...,g_{t}) , the estimation of the mean square error decomposes as: \mathbb{E}[(\hat{g}_{t}-\nabla f(\theta_{t}))^{2}]=\underbrace{(\mathbb{E}[\hat{g}_{t}]-\nabla f(\theta_{t}))^{2}}_{\mathrm{Bias}^{2}}+\underbrace{(\hat{g}_{t}-\mathbb{E}[\hat{g}_{t}])^{2}}_{\mathrm{Variance}} (2) Equation 2 establishes that the error in gradient estimation arises from two sources: bias, reflecting systematic deviation from the true gradient, and variance, capturing sensitivity to stochastic fluctuations. To explore how EMA and CM navigate this trade-off, we extend prior work on stochastic differential equations (SDEs) for vanilla SGD 70 , reformulating momentum in continuous time. Theorem 2.3 . Consider the unified momentum estimator m(t) defined by the stochastic differential equation (SDE) from Lemma A.4 , with solution given in Lemma A.6 . Let the bias be defined relative to the expected true gradient: \mathrm{Bias}(m(t))=\mathbb{E}[m(t)]-\mathbb{E}[\nabla f(\theta(t))] . Assuming that the gradient \nabla f(\theta(t)) is bounded and Lipschitz continuous, the asymptotic bounds (as t\to\infty ) for the bias and variance of m(t) as an estimator are given by: \left\|\mathrm{Bias}(m(t))\right\|^{2}\leq\Bigg(\frac{u^{2}\alpha LG}{(1-\beta)^{3}}+\frac{u^{2}\alpha\sigma L}{\sqrt{2}(1-\beta)^{2.5}}\\ +\left(\frac{u}{1-\beta}-1\right)G\Bigg)^{2}, (3) where L is the Lipschitz constant, G bounds