Why Adam Works Better with β 1 = β 2 \beta_{1}=\beta_{2} : The Missing Gradient Scale Invariance Principle
Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí · 2026
Read Why Adam Works Better with β 1 = β 2 \beta_{1}=\beta_{2} : The Missing Gradient Scale Invariance Principle by Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21739.txt.
Introduction
Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, and Enrique S. Quintana-Ortí explain an odd fact about the Adam optimizer. Adam has powered large-scale training for nearly a decade. Users kept noticing that setting its two momentum parameters equal works better, in scores and in the feel of training runs.
No principle accounted for that fact. The authors name a missing gradient scale invariance principle behind it. When the two parameters match, the updates respect that invariance. A clear principle turns a folk trick into a rule builders can apply on purpose, and it may guide the design of the next optimizer.
Catalog record
| CREATOR | Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Adam Works Better with β 1 = β 2 \beta_{1}=\beta_{2} : The Missing Gradient Scale Invariance Principle Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ortí Abstract Adam has been at the core of large-scale training for almost a decade, yet a simple empirical fact remains unaccounted for: both validation scores and the qualitative behaviour of the training runs improve when the momentum parameters satisfy β 1 = β 2 \beta_{1}=\beta_{2} . Some recent studies have reported this pattern, but there is still no explanation for why this choice helps. We show that this choice is closely tied to a structural property that we refer to as gradient scale invariance . We formalize this notion and prove that Adam becomes gradient scale invariant of first order if and only if β 1 = β 2 \beta_{1}=\beta_{2} . This perspective places the balanced regime of Adam in direct alignment with the design principles underlying several recent optimizers that explicitly enforce scale-robust updates. The theory is supported by experiments across vision and language tasks, and across different architectural families, in which rescaling the gradient has a markedly smoother effect on the update when β 1 = β 2 \beta_{1}=\beta_{2} . Overall, our results offer a coherent explanation for an open question in the behavior of Adam and provide a simple principle that helps guide the design of future optimizers. 1 Introduction Adam ( 10 ) has played a central role in the training of neural networks ever since it was introduced nearly a decade ago. Its combination of momentum ( 20 ) and per-parameter variance estimation has made it a default choice across architectures and domains, from convolutional networks to LLMs. The original paper proposed the now standard defaults \beta_{1}=0.9 and \beta_{2}=0.999 , and although there have been variations and some task-specific recommendations, such as using \beta_{2}=0.95 in certain LLMs ( 21 ) , the rationale for keeping two separate momentum parameters and for how they should be related has remained largely unclear. Recently, some empirical studies driven by 16 have reported that training can become more stable and accurate when these two parameters are set equal, that is, when \beta_{1}=\beta_{2} . This observation appears consistently across a range of settings, yet there is no rigorous explanation for why the choice \beta_{1}=\beta_{2} leads to improved behavior or what aspect of Adam’s dynamics makes this configuration distinctive. In parallel, many recent optimizers continue to build on ideas introduced by Adam, while incorporating mechanisms that reduce their sensitivity to the magnitude of the gradient. Examples include methods presented by 4 ; 12 ; 17 , which are based on normalized updates, sign information, or various forms of clipping. These approaches differ in motivation and formulation, but many of them share the common aim of limiting the influence of gradient scale during training. This trend is becoming increasingly visible in practice and points to the idea that reducing sensitivity to gradient scale is emerging as a desirable property in modern optimizers, even if its connection to the structure of Adam has not yet been clarified. This work offers a connection between the internal dynamics of Adam and the direction taken by many recent optimizers. The guiding principle is a property that the text refers to as gradient scale invariance . Informally, an optimizer is scale invariant if rescaling the gradient by a positive factor leaves the update essentially unchanged. While this property is exact in some simple methods, it is not built into Adam by design, and it is not obvious whether Adam can approximate it. We show that the answer lies in the relation between \beta_{1} and \beta_{2} . When these two coefficients differ, the update produced by Adam reacts to a change in gradient scale in a way that is linear in the size of the perturbation. When they coincide, the leading dependence disappears and the sensitivity is pushed to second order. This makes the updates far more stable under fluctuations in gradient magnitude, improving both the smoothness of the training dynamics and the overall behaviour of the method. Our contributions are as follows: • We formalize gradient scale invariance and examine how Adam relates to it. Using the continuous-time limit of the algorithm, we analyze how both momentum terms depend on the gradient, and identify when their combined effect removes the first-order dependence on gradient scale in the parameter update. This leads to a clear statement: Adam has gradient scale invariance of first order if and only if \beta_{1}=\beta_{2} . • We validate this analysis experimentally. Synthetic examples illustrate how the updates change under multiplicative rescalings of the gradient, and real training runs on a range of vision and language models (NanoGPT on SlimPajama and WikiText, EfficientNet-B0 on TinyImageNet, ResNet18 and ViT-B16 on CIFAR-100, and T5 on SQuAD) show that the norm of the update becomes noticeably smoother when the two parameters coincide. The rest of the paper is organized as follows. Section 2 reviews related work, touching on theoretical analyses of Adam, recent families of optimizers where invariance plays a central role, and the empirical evidence that points to the regime \beta_{1}=\beta_{2} . Section 3 introduces gradient scale invariance and sets up the discrete form of Adam. Section 4 presents the Adam flow, develops the analytical expansion that leads to our main result, and illustrates it with synthetic experiments. Section 5 examines the choice of \beta when \beta_{1}=\beta_{2} and explores its relation to the number of batches required to train a model. Sections 6 and 7 contain conclusions and limitations. 2 Related Work A substantial body of work has studied Adam ( 10 ) and related adaptive methods from a theoretical perspective. One line of research develops continuous-time limits that capture a wide class of first-order algorithms as non-autonomous differential equations. In particular, 5 derive an Ordinary Differential Equation (ODE) framework that includes AdaGrad ( 6 ) , RMSProp ( 9 ) , Adam and Nesterov ( 15 ) as special cases, and establish well-posedness and convergence properties of the associated flows. A complementary viewpoint, developed in the optimization literature, emphasizes that Adam often behaves like a normalized or sign-driven method: 1 interpret Adam as combining a sign-based direction with a magnitude modulated by relative variance, while sign methods such as signSGD ( 3 ) make scale invariance explicit by construction. These perspectives motivate asking whether, and in which regimes, Adam itself can inherit a principled form of scale robustness. Another important line is the work of 14 who study in detail the dynamic behavior of RMSProp and Adam. They observe a characteristic three-phase structure in the loss evolution, consisting of fast initial decrease, an oscillatory regime and late spikes, and interpret these features through a continuous-time limit of Adam that resembles a sign-based gradient flow. Their empirical analysis shows how the choice of hyperparameters, and in particular making \beta_{1} and \beta_{2} closer, can smooth the dynamics and improve stability in high-accuracy regimes. Related empirical evidence in transformers indicates that Adam’s behavior in low-noise regimes can be closely matched by sign descent with momentum ( 11 ) , further supporting the relevance of sign-like mechanisms. However, these works do not isolate the role of gradient scale