How do simple rotations affect the implicit bias of Adam?

Adela DePavia, Vasileios Charisopoulos, Rebecca Willett · 2025

Read How do simple rotations affect the implicit bias of Adam? by Adela DePavia, Vasileios Charisopoulos, Rebecca Willett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23804.txt.

Introduction

Adela DePavia and two co-authors probe a quirk of Adam, the workhorse optimizer of machine learning: its hidden bias can flip when the data are rotated. Earlier work suggested Adam has a richness bias, learning curved decision boundaries closer to the best possible than gradient descent finds. But Adam rescales each coordinate separately, which makes it sensitive to feature orientation.

The authors show that even small rotations of the data distribution can make Adam drop its rich boundary and settle for a linear one, worse than what gradient descent reaches. The competitive advantage reverses from a change of coordinates. As a remedy, they use a recent reparameterization that rotates the optimization objective, giving any first-order method built-in indifference to data rotations. Experiments confirm the fix restores Adam's pull toward rich boundaries. The lesson: an optimizer's bias is not a fixed trait but depends on representation.

Catalog record

CREATORAdela DePavia, Vasileios Charisopoulos, Rebecca Willett
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23804

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How do simple rotations affect the implicit bias of Adam? Adela DePavia, Vasileios Charisopoulos, Rebecca Willett Abstract Adaptive gradient methods such as Adam and Adagrad are widely used in machine learning, yet their effect on the generalization of learned models—relative to methods like gradient descent—remains poorly understood. Prior work on binary classification suggests that Adam exhibits a “richness bias,” which can help it learn nonlinear decision boundaries closer to the Bayes-optimal decision boundary relative to gradient descent. However, the coordinate-wise preconditioning scheme employed by Adam renders the overall method sensitive to orthogonal transformations of feature space. We show that this sensitivity can manifest as a reversal of Adam’s competitive advantage: even small rotations of the underlying data distribution can make Adam forfeit its richness bias and converge to a linear decision boundary that is farther from the Bayes-optimal decision boundary than the one learned by gradient descent. To alleviate this issue, we show that a recently proposed reparameterization method—which applies an orthogonal transformation to the optimization objective—endows any first-order method with equivariance to data rotations, and we empirically demonstrate its ability to restore Adam’s bias towards rich decision boundaries. 1 Introduction Adaptive gradient methods—including Adam , Adagrad , and variants like AdamW —are extremely popular optimizers in neural network training and other areas of modern machine learning [ 2 ] . Recent work has explored the impact of the choice of optimizer on generalization. For instance, Vasudeva et al. 2025 ask, “What is the implicit bias of Adam for nonlinear models such as [neural networks], and how does it differ from the implicit bias of (stochastic) gradient descent?” Answering this question can provide insights into the ramifications of optimizer choice in machine learning. The central result of Vasudeva et al. 2025 is essentially that adaptive methods can overcome the “simplicity bias” of gradient descent ( GD ) [ 10 , 12 , 11 ] . To illustrate this, they consider a binary classification setting, construct a family of data distributions, and analyze training a two-layer ReLU network classifier. For their family of distributions, the Bayes decision boundary is nonlinear, but training using GD provably yields a linear classifier. In contrast, training using an adaptive optimization method yields a nonlinear decision boundary similar to the Bayes decision boundary, as illustrated in Fig. 1(a) . At first glance, these findings may suggest that adaptive methods naturally prefer “richer” predictors than gradient descent. However, most algorithms in this family (including Adam ) are essentially preconditioned variants of gradient descent; crucially, their preconditioners are diagonal —i.e., applied coordinate-wise—meaning that the overall method is not equivariant to simple transformations (e.g., rotations) of the model parameters. As we highlight in this work, rotations in data space induce rotations in parameter space for feedforward neural networks. Thus, the sensitivity of adaptive algorithms to parameter rotations implies sensitivity to data rotations. Consequently, it is natural to ask: How do simple transformations of the data distribution affect the implicit bias of adaptive gradient methods? In this paper, we study this question in the stylized setting of Vasudeva et al. 2025 , considering rotations of the feature space. We show that even for small rotations of the same distribution considered in [ 14 ] , adaptive optimization yields a linear classifier which, for almost all rotation angles, is further from the Bayes decision boundary than the result of GD training, as illustrated in Fig. 1(b) . Our results complement the work of Vasudeva et al. 2025 : taken together, these results establish that arbitrarily small perturbations of the training data can significantly worsen the generalization of adaptive gradient methods. This dramatic shift in qualitative behavior prompts the following question: can we algorithmically identify a coordinate system in which adaptive methods produce rich decision boundaries? To answer this, we turn our attention to a simple reparameterization method proposed by the current authors [ 3 ] , which corresponds to an orthonormal transformation of model parameters derived from the expected gradient outer product (EGOP) matrix of the training loss. In particular, we show that this reparameterization method coupled with adaptive optimization results in a rotation invariance that restores the implicit bias identified by Vasudeva et al. 2025 , as illustrated in Fig. 1(c) . Our analysis establishing rotation invariance under EGOP-reparameterization extends to related algorithms, including \mathtt{SOAP} [ 15 ] and \mathtt{Shampoo} [ 4 ] . Organization. In the rest of this section, we introduce the notation used throughout our paper. In Section 2 we provide brief formal statements of the main results of the paper and review related literature in Section 3 . In Section 4 , we analyze the decision boundaries learned by adaptive algorithms on binary classification problems, emphasizing their sensitivity to data rotations. In Section 5 , we show that EGOP-reparameterization endows adaptive algorithms with equivariance to data rotations, producing generalization behavior that is invariant to these orthogonal transformations. Finally, in Section 6 , we present a suite of numerical experiments illustrating our theoretical predictions, which remain valid even in settings not explicitly covered by our theory. 1.1 Notation and background We write \langle\cdot,\cdot\rangle for the Euclidean inner product on \mathbb{R}^{d} and \lVert\cdot\rVert=\sqrt{\langle\cdot,\cdot\rangle} for the induced Euclidean norm. For a matrix A\in\mathbb{R}^{m\times n} , we denote its (column-major) vectorization by \operatorname{vec}(A)\in\mathbb{R}^{mn} and its adjoint operation by \mathrm{mat}:\mathbb{R}^{mn}\rightarrow\mathbb{R}^{m\times n} , such that \mathrm{mat}(\operatorname{vec}(A))=A . We denote the i^{\text{th}} entry of a vector \theta by \theta_{i} and the k^{\text{th}} row of a matrix A\in\mathbb{R}^{m\times n} by A_{k,:}\in\mathbb{R}^{n} . We use \odot to denote the Hadamard (element-wise) product operation between vectors and matrices of compatible dimensions and \otimes for the Kronecker product. We write \mathbf{diag}(v) for the square matrix which equals v across its diagonal and 0 elsewhere. We use \mathbb{I}_{d}\in\mathbb{R}^{d\times d} to denote the identity matrix. We let O(d) denote the set of orthogonal d\times d matrices. Finally, we write \mathds{1}(\cdot) for the indicator function. In Appendix A , we formally define the methods that we study—namely, Adam ( Algorithm 1 ), \mathtt{SignGD} ( Algorithm 2 ), gradient descent ( Algorithm 3 ), and EGOP-reparameterization ( Algorithm 4 ); they all assume first-order access to the loss function \mathcal{L} . 2 Overview of main results In this section, we introduce the problem setup and summarize our main results. Our goal is to study the implicit bias of Adam ( Algorithm 1 ) and gradient descent ( GD , Algorithm 3 ) for the synthetic binary classification task introduced by Vasudeva et al. 2025 under rotations to the input data distribution. As in Vasudeva et al. 2025 , we focus on 2-layer neural networks with ReLU activations and fixed outer layer weights: f(W;x)\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}a^{\mathsf{T}}\operatorname{ReLU}\big(Wx\big), (1) where a\in\mathbb{R}^{m} is a fixed vector of outer layer weights, m is the hidden layer width, \operatorname{ReLU}(x)=\max(0,x) is applied elementwise, and W\in\mathbb{R}^{m\times d} are the hidden layer weights. While we are ultimately interested in the implicit bias of Adam , we analyze a simplified, analytically tractable version known as \mathtt{SignGD} ( Algorithm 2 ) — a memoryless instantiation of Adam ( \beta_{1}=\beta_{2}=0 ) which reduces to the following update when \epsilon=0 : \theta_{t+1}=\theta_{t}-\eta\operatorname{sign}(\nabla\mathcal{L}(\theta_{t})),\qquad\text{where }\operatorname{sign}(z):=\begin{pmatrix}\operatorname{sign}(z_{i})\end{pmatrix}_{i=1}^{p}. (2) From an analytical standpoint, \mathtt{SignGD} is a tractable proxy for Adam because the setting \beta_{1}=\beta_{2}=0 removes the dependence on trajectory history.

Read the complete text