When Can You Get Away with Low Memory Adam?
Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli, Tom Goldstein · 2025
Read When Can You Get Away with Low Memory Adam? by Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli, Tom Goldstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01843.txt.
Introduction
Kalra and co-authors ask when memory-hungry parts of the Adam optimizer can be shrunk without hurting training. Adam, the default trainer for modern models, stores running averages of gradients and squares; that second store is the heavy one. Their tool is a layer-wise signal-to-noise analysis: where the second-moment values in a layer barely vary, the block can be replaced by its mean. The analysis shows how architecture, settings, and data decide where that is safe.
The result is SlimAdam, which compresses second moments only along the dimensions the analysis clears, and leaves the rest untouched. Across varied architectures and training setups, it matched Adam's accuracy and stability while cutting up to 98 percent of the second-moment storage. Code is public. For teams training large models on tight hardware, the paper replaces guesswork about optimizer memory with a measured rule.
Catalog record
| CREATOR | Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli, Tom Goldstein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Can You Get Away with Low Memory Adam? Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli, Tom Goldstein Abstract Adam is the go-to optimizer for training modern machine learning models, but it requires additional memory to maintain the moving averages of the gradients and their squares. While various low-memory optimizers have been proposed that sometimes match the performance of Adam, their lack of reliability has left Adam as the default choice. In this work, we apply a simple layer-wise Signal-to-Noise Ratio (SNR) analysis to quantify when second-moment tensors can be effectively replaced by their means across different dimensions. Our SNR analysis reveals how architecture, training hyperparameters, and dataset properties impact compressibility along Adam’s trajectory, naturally leading to SlimAdam , a memory-efficient Adam variant. SlimAdam compresses the second moments along dimensions with high SNR when feasible, and leaves when compression would be detrimental. Through experiments across a diverse set of architectures and training scenarios, we show that SlimAdam matches Adam’s performance and stability while saving up to 98 % 98\% of total second moments. Code for SlimAdam is available at https://github.com/dayal-kalra/low-memory-adam . 1 Introduction Adam with weight decay ( Loshchilov & Hutter, 2019 ) has become the standard optimizer choice in modern machine learning, consistently outperforming non-adaptive optimizers such as Stochastic Gradient Descent with momentum (SGD-M). Its success is typically attributed to adapting to the geometry of the landscape by estimating the “effective learning rate” for each parameter using a moving average of the squared gradients. An additional benefit of this adaptive mechanism is that the optimal learning rate is less sensitive to changes in the training recipe. Figure 1 : Comparison of common low-memory optimizers on GPT pre-training task using Fineweb-Edu dataset. SlimAdam matches Adam’s performance with a nearly identical U-shaped loss curve. While these factors conspire to make Adam the go-to optimizer for training language models, it requires additional memory beyond SGD-M. It requires storing moving averages of both first and second moments, doubling the optimizer’s memory footprint. This memory cost becomes particularly crucial in resource-limited settings, where the memory allocated to the optimizer states could otherwise be used for the model parameters or activations. To avoid the extra memory footprint of Adam, various low-memory optimizers have been proposed ( Shazeer & Stern, 2018 ; Ginsburg et al., 2020 ; Anil et al., 2019 ; Modoranu et al., 2024 ) . These optimizers are a free lunch in some settings – slashing memory usage with no detectable loss in performance ( Zhao et al., 2024 ; Zhang et al., 2024b ) – but they compromise training performance in others ( Luo et al., 2023 ) . While the potential benefits of low-memory optimizers are clear, a lack of understanding as to when they will perform well is a major barrier to widespread adoption, as the expense of training modern generative models makes engineers unwilling to take risks such as modifying core components in the training recipe. We argue that a practical low-memory alternative to Adam should exhibit the following properties. First and foremost, it must maintain optimization efficacy, showing no degradation in performance. Additionally, it should preserve Adam’s robustness to minor changes in the training hyperparameters. Finally, the low-memory optimizer should immediately work with the same hyperparameter choices as Adam so that users can swap in a low-memory optimizer without major re-tuning. Figure 1 reveals a natural dichotomy in the space of low-memory optimizers: (1) those that yield learning rate sensitivity curves similar to Adam’s, and (2) those that deviate substantially, exhibiting major shifts in optimal learning rates and expected training dynamics. The first group comprises Adam-mini and our proposed SlimAdam which are both constructed by replacing individual second moments with their means along specific dimensions, whereas the latter group composed of Lion, SM3, and Adafactor are all significantly different algorithms. In this work, we focus on the first category of low-memory optimizers, as they can serve as a drop-in replacement for Adam. Our goal here is to develop a principled framework to help users understand and quantify when these low-memory variants of Adam are appropriate for their problem, thereby improving the reliability of low-memory optimizers and providing deeper insights into Adam’s dynamics. Contributions: We propose and study a simple measure of the compressibility of Adam’s second-moment memory. By examining the Signal-to-Noise Ratio (SNR) of the second moment tensor in each layer, we quantify when individual second moments can be effectively replaced by their means across different matrix/tensor dimensions (such as \text{fan}_{\text{in}} , \text{fan}_{\text{out}} , or both dimensions). Our SNR-based metrics reveal that layers exhibit varying degrees of compressibility along different dimensions, and this compressibility can depend strongly on the architecture, training hyperparameters, and dataset properties. For example, when training a transformer language model, an optimizer should compress key and query second moments in only the \text{fan}_{\text{in}} dimension, as behaviors in the \text{fan}_{\text{out}} dimensions are inconsistent across the multiple heads stacked in that dimension. While some layer types show consistent compressibility patterns across training configurations, we also observe that some layer types show varying compression trends. These inconsistent patterns suggest that a one-size-fits-all approach to low-memory optimization is suboptimal. To demonstrate the utility of our findings, we implement SlimAdam , a memory-efficient variant of Adam that adaptively compresses the second moments along the most efficient dimensions, or selectively leaves layers uncompressed when needed to maintain stability. By taking an adaptive approach to compression, SlimAdam preserves desirable properties of Adam while significantly reducing memory usage. For instance, it saves 98\% of second moments in \sim 124M parameter GPT-style Transformer trained on language tasks. Further, we show that SlimAdam matches Adam’s performance as well as robustness to the choice of learning rate. Our analysis also reveals a surprising property of Adam: it uses significantly more second moments at large learning rates than required for optimal performance. For instance, in GPT-style Transformers trained on language modeling, while the SNR analysis suggests that \sim 35\% of second moments could be compressed at Adam’s optimal learning rate, SlimAdam actually achieves Adam’s performance while compressing 98\% of them. This intriguing finding suggests that the majority of Adam’s per-parameter adaptivity isn’t necessarily required for optimal training. 1.1 Related Work The superiority of Adam is observed primarily in language modeling, with SGD performing comparably to Adam in image classification settings ( Zhang et al., 2020 ) . This disparity has motivated several investigations into the unique challenges of language modeling landscapes, with studies identifying several explanations. ( Zhang et al., 2020 ; Ahn et al., 2024 ) demonstrated that the heavy-tailed distribution of the stochastic gradient noise in language modeling cases causes SGD to perform worse than Adam. ( Pan & Li, 2022 ) attributed Adam’s faster convergence to “directional sharpness,” which is the curvature along the update direction. Adding to these findings, ( Zhang et al., 2024a ) illustrated that the Hessian spectrum across parameter blocks varies heavily and suggested that SGD performs worse because it applies a single learning rate to all blocks. Further insights come from ( Kunstner et al., 2024 ) , who showed that, in settings with heavy-tailed class imbalance, SGD struggles to decrease loss in infrequent classes, while adaptive optimizers are less sensitive to this imbalance. ( Zhao et al., 2024