The Rich and the Simple: On the Implicit Bias of Adam and SGD

Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, Mahdi Soltanolkotabi · 2025

Read The Rich and the Simple: On the Implicit Bias of Adam and SGD by Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, Mahdi Soltanolkotabi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24022.txt.

Introduction

Bhavya Vasudeva and three co-authors show that two workhorse training algorithms quietly prefer different kinds of solutions, in a 2025 paper. Neural networks trained by plain gradient descent tend to settle on simple decision rules, a simplicity bias. Adam, the optimizer behind most large models, behaves otherwise, and the paper pins down the contrast.

Training small two-layer networks on a simple classification task with bell-curve data, the team finds gradient descent drawing a straight boundary with a poor safety margin. Adam instead grows richer, more varied features and a curved boundary. The difference matters because the bias an optimizer brings decides what a model learns when data fit many rules. Builders choosing an optimizer are also choosing a taste in solutions.

Catalog record

CREATORBhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, Mahdi Soltanolkotabi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE111 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Rich and the Simple: On the Implicit Bias of Adam and SGD Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, Mahdi Soltanolkotabi Abstract Adam is the de facto optimization algorithm for several deep learning applications, but an understanding of its implicit bias and how it differs from other algorithms, particularly standard first-order methods such as (stochastic) gradient descent (GD), remains limited. In practice, neural networks (NNs) trained with SGD are known to exhibit simplicity bias — a tendency to find simple solutions. In contrast, we show that Adam is more resistant to such simplicity bias. First, we investigate the differences in the implicit biases of Adam and GD when training two-layer ReLU NNs on a binary classification task with Gaussian data. We find that GD exhibits a simplicity bias, resulting in a linear decision boundary with a suboptimal margin, whereas Adam leads to much richer and more diverse features, producing a nonlinear boundary that is closer to the Bayes’ optimal predictor. This richer decision boundary also allows Adam to achieve higher test accuracy both in-distribution and under certain distribution shifts. We theoretically prove these results by analyzing the population gradients. Next, to corroborate our theoretical findings, we present extensive empirical results showing that this property of Adam leads to superior generalization across various datasets with spurious correlations where NNs trained with SGD are known to show simplicity bias and do not generalize well under certain distributional shifts. 1 Introduction Adaptive optimization algorithms, particularly Adam [ 1 ] , have become ubiquitous in training deep neural networks (NNs) due to their faster convergence rates and better performance, particularly on large language models (LLMs), as compared to (stochastic) gradient descent (SGD) [ 2 ] . Despite its widespread use, the theoretical understanding of how Adam works and when/why it outperforms (S)GD remains limited. Modern NNs are heavily overparameterized and thus the training landscape has numerous global optima. As a result, different training algorithms may exhibit preferences or biases towards different global optima a.k.a. implicit bias . There is extensive prior work on the implicit bias of GD [ 3 , 4 , 5 , 6 ] , for both linear and nonlinear models (see Section 5 for a detailed discussion of related work). However, there is limited work investigating the implicit bias of Adam. Recently, Zhang et al. [7] showed that for linear logistic regression with separable data, Adam iterates directionally converge to the minimum \ell_{\infty} -norm solution, in contrast to GD which converges to minimum \ell_{2} -norm [ 3 ] . This difference between the implicit bias of Adam vs GD in simple linear settings motivates the central question of this paper: What is the implicit bias of Adam for nonlinear models such as NNs, and how does it differ from the implicit bias of (S)GD? Figure 1 : Illustration of the synthetic dataset considered in this work, and comparison of the Bayes’ optimal predictor with the decision boundaries of two-layer NNs trained with Adam and GD. Figure 2 : Training with Adam leads to better performance across different test set metrics on six benchmark datasets with spurious correlations, as compared to SGD. See Section 4 for details. Given the popularity of Adam, surprisingly little is known about the implicit bias of Adam for training NNs. A notable exception is the recent work Tsilivis et al. [8] , which characterized the late-stage implicit bias of a family of steepest descent algorithms, on homogeneous NNs, in terms of maximizing an algorithm-dependent geometric margin ( \ell_{\infty} for signGD or Adam without momentum) and convergence to KKT points. However, the stationary points of the respective margin-maximization may not be unique. Additionally, this characterization does not relate the implicit bias of the algorithm with properties of the learned solution, such as the type of features learned, or the complexity of the decision boundary, which in turn impact its generalization. In particular, in many applications, NNs trained with SGD are known to exhibit a simplicity bias , that is, they learn simple solutions [ 9 ] , which can lead to suboptimal generalization. For example, SGD-trained two-layer NNs rely on low-dimensional projections of the data to make predictions [ 10 ] . This simplicity bias can be particularly detrimental in the presence of spurious features where it is simpler for NNs trained with SGD to utilize them to achieve zero training error 1 1 1 We remark that simplicity bias may not always be detrimental; for instance, it can be beneficial for in-distribution generalization (see Appendix C for further discussion). Our focus is characterizing and contrasting the implicit bias of Adam vs (S)GD in terms of rich vs simple feature learning, not advocating for one to always be better. . This leads to the question: Does training with Adam lead to solutions that are resistant to this simplicity bias? In this paper, we answer these questions in two ways. Theoretically , we show that two-layer ReLU NNs trained on a Gaussian mixture data setting (see Fig. 2 ) with SGD exhibit simplicity bias while training with Adam leads to richer feature learning. Empirically , we demonstrate that training with Adam can lead to better performance as compared to SGD on various benchmark datasets with spurious features (see Fig. 2 ). Our main contributions are as follows. • We identify a simple yet informative setting with mixture of Gaussians where GD and Adam exhibit different implicit biases (see Fig. 2 ). The Bayes’ optimal predictor in this setting is nonlinear (piecewise linear), and we show — both theoretically and empirically — that while GD exhibits a simplicity bias resulting in a linear predictor, Adam encourages reliance on richer features leading to a nonlinear decision boundary , which is closer to the Bayes’ optimal predictor. We theoretically prove this difference in the implicit bias by analyzing the population gradients and updates of GD and Adam without momentum (signGD). We also show that this leads to better test accuracy in distribution as well as across some distribution shifts. Additionally, to theoretically understand the behaviour of Adam with momentum, we analyze a simpler setting where the variance of the Gaussians approaches 0 , in the infinite width limit. We show that the decision boundaries learned with signGD and Adam are more nonlinear (and closer to the Bayes’ optimal predictor) than the one learned with GD. • We conduct extensive experiments on various datasets and show that Adam leads to richer features that are more robust compared to simpler features learned via SGD, allowing Adam to achieve higher test accuracy both in distribution and under certain distribution shifts . First, we consider an MNIST-based task with a colored patch as a spurious feature and show that compared to SGD, training with Adam leads to a more nonlinear decision boundary, larger margins overall, and has better generalization on a test set with flipped correlation. Next, we show that Adam achieves better worst-group accuracy on four benchmark datasets (Waterbirds, CelebA, MultiNLI, and CivilComments) for subgroup robustness, and better decoded worst-group accuracy on the Dominoes or MNIST-CIFAR dataset [ 9 ] , with images from CIFAR and MNIST classes as the complex/core and simple/spurious features, respectively. Finally, we study the Boolean features dataset proposed

Read the complete text