Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Sharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells · 2026

Read Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization by Sharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04269.txt.

Introduction

Sharan Sahu and three co-authors analyse Adam, a widely used training optimizer, when the objective keeps changing over time, as in shifting data or moving targets. In that nonstationary setting, the optimizer must track a moving solution rather than settle on a fixed one.

Their bounds split the tracking error into clear parts: starting point, drift of the objective, and errors tied to Adam's two averaging parameters. They also give a high-probability guarantee for a projected stationarity measure under distribution shift. The results expose a trade-off: averaging and adaptive scaling help when noise dominates, but stale averages hurt when drift dominates.

Catalog record

CREATORSharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE121 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04269

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization Sharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells Abstract We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general L L -smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by β 1 \beta_{1} , and a preconditioner perturbation governed by β 2 \beta_{2} . We characterize the burn-in time to reach Adam’s irreducible tracking floor under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise–drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can improve the high-probability error, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can compound the cost of nonstationarity, allowing vanilla SGD to achieve a smaller tracking floor. Our explicit ( β 1 , β 2 , ϵ ) (\beta_{1},\beta_{2},\epsilon) -dependent bounds delineate when adaptive step-sizing is beneficial versus harmful, and provide a theoretical mechanism for Adam’s empirical instability and stabilization under distribution shift. 1 Introduction Consider the optimization problem on a compact convex set \Theta\subset\mathbb{R}^{d} posed by a smooth objective function G_{t}:\mathbb{R}^{d}\rightarrow\mathbb{R} defined as: \bm{\theta}_{t}^{\star}\in\argmin_{\bm{\theta}\in\Theta}G_{t}(\bm{\theta}),\;\;G_{t}(\bm{\theta})=\mathbb{E}_{X_{t}\sim\Pi_{t}}[g(\bm{\theta},X_{t})]. (Opt) Here g(\bm{\theta},X_{t}) is a noise perturbed measurement of G_{t}(\bm{\theta}) and X_{t} is a random variable sampled from a time-varying distribution \Pi_{t} . This setting connects naturally to classical stochastic tracking in signal processing [ 22 , 35 ] , concept drift in online learning [ 12 , 3 ] , and minimum divergence-type statistical objectives [ 33 ] , where the goal is not convergence to a fixed solution but faithful tracking of an unknown, time-varying minimizer. Adaptive gradient methods such as Adam [ 18 ] have become the default optimizer for training large-scale deep learning models [ 1 , 8 ] , combining coordinate-wise adaptive step sizes derived from second-moment estimates with first-moment gradient averaging. In stationary settings, Adam is known to enjoy favorable convergence properties [ 13 , 14 ] , and its adaptivity is widely credited with reducing sensitivity to hyperparameter tuning and accelerating progress on ill-conditioned objectives. Recent work analyzing stochastic gradient descent (SGD) and its momentum-based variants (SGDM), including Polyak’s Heavy-Ball [ 31 ] and Nesterov acceleration [ 26 ] , in non-stationary settings [ 34 ] has shown that gradient averaging induces tracking lag under distribution shift. Since Adam incorporates momentum in both its first- and second-moment estimates, it is natural to ask if similar degradation occurs, and whether the adaptive preconditioning compounds or alleviates it. These failure modes have indeed been observed empirically in continual learning and non-stationary optimization [ 6 , 25 , 9 , 7 ] , yet a precise theoretical account of when and why Adam degrades under distribution shift has remained unclear. These observations motivate the following central question in the non-stationary tracking setting: When does Adam’s adaptive preconditioning help under distribution shift, and when can its moment estimates provably hurt relative to vanilla SGD, for L -smooth time-varying functions G_{t}(\bm{\theta}) ? Our contributions. We answer this question under predictable distribution shift and make the following advances. 1. Finite-time tracking bounds under adaptive strong monotonicity. We prove finite-time tracking bounds for Adam that separate four distinct error sources: initialization decay, drift-induced tracking lag, a first-moment tracking error governed by \beta_{1} , and a preconditioner perturbation governed by \beta_{2} . To obtain a tracking recursion, we impose strong monotonicity on the mean-gradient map after Adam-style preconditioning. Since the realized Adam preconditioner is computed from the fresh sample X_{t+1} at the same step, we formulate this condition using a predictable proxy rather than the sample-dependent preconditioner itself. This yields a population-level contraction condition, while the difference between the realized preconditioner and its predictable proxy is controlled explicitly as an Adam-specific perturbation term. 2. A noise–drift tradeoff separating Adam from SGD. Our explicit (\beta_{1},\beta_{2},\epsilon) -dependent bounds reveal a sharper optimizer-specific noise–drift tradeoff than SGD and take a different form than in SGDM. In noise-dominated regimes, first-moment averaging and adaptive preconditioning can reduce the high-probability tracking floor, and our bounds identify parameter regimes in which Adam can improve over SGD, consistent with our numerical simulations. In drift-dominated regimes, however, the \beta_{1} -induced memory bias the \beta_{2} -induced preconditioner perturbation can compound the cost of nonstationarity, allowing vanilla SGD to achieve a smaller tracking floor. The bounds also give a theoretical mechanism for the empirical observation of [ 25 ] that increasing \epsilon stabilizes Adam under task changes: larger \epsilon dampens adaptive second-moment variability at the cost of slower adaptation to drift. 3. Projected stationarity bounds under general Adam preconditioners. Without the adaptive monotonicity condition, we prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. This is the natural constrained analogue of a preconditioned gradient-norm guarantee and reduces to one when the projection is inactive. The same Adam-specific error structure persists: objective drift, first-moment bias-variance tradeoffs, and second-moment preconditioner perturbation all contribute explicitly, showing that the qualitative picture identified in the tracking regime extends to general Adam preconditioners. 4. Experiments corroborating the noise–drift tradeoff. We validate the theory on strongly convex least squares and three non-convex online problems: teacher–student MLP regression, phase retrieval, and matrix factorization under controlled drift and time-varying noise. Across these settings, the results support the predicted tradeoff: SGD performs better when drift dominates because it adapts more quickly, whereas Adam is more effective when stochastic noise dominates due to momentum and adaptive scaling. 1.1 Related Work Adam [ 18 ] has been the subject of extensive theoretical study in stationary settings. Indeed, in a variety of nonconvex settings, Adam provably converges under realistic conditions and adapts to the geometry of the problem in ways that SGD cannot [ 23 , 13 , 38 , 15 ] . The advantage of Adam over SGD has been attributed to several sources, including heavy-tailed gradient distributions in language modeling [ 21 ] , Hessian heterogeneity across layers in Transformer architectures [ 42 ] , and the implicit sign-descent behavior of its updates [ 20 ] . However, Adam does not universally outperform simpler methods: it has been shown to generalize worse than SGD in certain settings [ 40 ] , fail to converge without modification in convex problems [ 32 ] , and exhibit instability and loss of plasticity under distribution shift [ 25 , 6 , 9 ] . Most existing convergence theory for Adam assumes a fixed stationary objective. Under non-stationarity, past gradients become stale and temporal averaging can be actively harmful. Building on the foundational dynamic regret framework of [ 44 ] , non-asymptotic tracking and regret guarantees have been established for SGD under time drift and dependent data [ 2 , 4 , 43 , 37 ] . Most directly related to our work, [ 34 ] provide finite-time tracking bounds for SGD and its momentum variants under strongly convex smooth objectives, showing that momentum amplifies drift-induced tracking lag. Nonconvex tracking guarantees under distribution shift

Read the complete text