Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression

Xuheng Li Quanquan Gu · 2025

Read Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression by Xuheng Li Quanquan Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14123.txt.

Introduction

Xuheng Li and Quanquan Gu prove, in a 2025 paper, why a training trick steadies learning. The trick, exponential moving average, keeps a running blend of a model's recent states. It is now standard in training image generators, yet its theory lagged its practice. The authors study it in the cleanest setting that still resembles a neural network: high-dimensional linear regression trained by online stochastic gradient descent. There the model's guesses update example by example.

Two results stand out. The averaged method's variance error, the noise in its final answer, is always smaller than plain training's. Its bias error, the systematic pull away from truth, fades exponentially fast along every direction of the data's geometry when the average starts later. That is a sharper decay than starting the average from the first step. The proof tools, the authors note, extend to other averaging schemes.

Catalog record

CREATORXuheng Li Quanquan Gu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14123

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression Xuheng Li Quanquan Gu Abstract Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there have been few theoretical results explaining the effectiveness of EMA. In this paper, to better understand EMA, we establish the risk bound of online SGD with EMA for high-dimensional linear regression, one of the simplest overparameterized learning tasks that shares similarities with neural networks. Our results indicate that (i) the variance error of SGD with EMA is always smaller than that of SGD without averaging, and (ii) unlike SGD with iterate averaging from the beginning, the bias error of SGD with EMA decays exponentially in every eigen-subspace of the data covariance matrix. Additionally, we develop proof techniques applicable to the analysis of a broad class of averaging schemes. 1 Introduction The exponential moving average (EMA, Polyak and Juditsky 1992 ; Ruppert 1988 ) in conjunction with stochastic optimization algorithms is being extensively used in training deep learning models. EMA is most popular in training generative models based on GAN ( Yaz et al., 2018 ; Karras, 2019 ; Kang et al., 2023 ) , and more recently in diffusion models ( Song et al., 2020b ; Dhariwal and Nichol, 2021 ; Nichol and Dhariwal, 2021 ; Song et al., 2020a ; Balaji et al., 2022 ; Karras et al., 2022 ; Rombach et al., 2022 ; Karras et al., 2024 ) , among other applications ( Block et al., 2023 ; Busbridge et al., 2024 ) . By maintaining an averaged set of model parameters, EMA displays the capability to stabilize training by suppressing the noise of stochastic gradients, and it has been shown empirically that the effect of EMA is similar to that of learning rate scheduling ( Sandler et al., 2023 ) . However, this phenomenon is less studied from a theoretical perspective. Notable exceptions include a recent work by Ahn and Cutkosky (2024) , which studied Adam with EMA in nonconvex optimization. However, this work is restricted to the finite-dimensional setting, which departs from the practical training of overparameterized neural networks. Block et al. (2023) revealed the variance-reducing benefit of EMA, but the bias contraction of stochastic optimization algorithms with EMA remains unknown. Meanwhile, a recent line of works ( Défossez and Bach, 2015 ; Dieuleveut et al., 2017 ; Jain et al., 2018b ; Berthier et al., 2020 ; Zou et al., 2021 ; Wu et al., 2022 ) characterized the generalization properties of SGD in overparameterized linear regression with other averaging schemes (e.g., iterate averaging from the beginning and tail averaging). In particular, Zou et al. (2021) presented an instance-dependent and dimension-free excess risk bound for SGD with iterate averaging and tail averaging. Given these results, a characterization of the generalization properties of SGD with EMA and a comparison against SGD with other averaging schemes becomes an urgent subject of study, especially in the setting of setting of high-dimensional linear regression. In this paper, we tackle this open problem by studying SGD with EMA in the overparameterized linear regression setting, and comparing the results with SGD without averaging, along with iterate averaging and tail averaging in Zou et al. (2021) . Our contributions are summarized as follows: Table 1: Comparison of SGD with EMA against SGD without averaging, SGD with iterate averaging from the beginning and with tail averaging. We fix the eigenvalue spectrum of the covariance matrix \{\lambda_{i}\} , the learning rate \delta , and the number of iterations N . We assume that tail averaging is performed over the last N-s iterates, and \alpha is the weight of the moving average in EMA. We study not only the effective bias and the variance error, but also the effective dimension that plays a role similar to the model dimension in our excess risk bound. Compared with SGD without averaging which has the same exponentially decaying effective bias as EMA, SGD with EMA has a smaller variance error. SGD with either iterate averaging or tail averaging enjoys a smaller variance error than SGD without averaging, and the variance error and the effective dimension of SGD with tail averaging are identical to that of EMA when (1-\alpha)(N-s)=1 . However, SGD with neither iterate averaging nor tail averaging achieves the effective bias decay rate that is exponential in N . Averaging scheme Effective bias decay rate Variance error in subspace of \lambda_{i} Eigenvalue at effective dim. w/o averaging Exponential in N \mathcal{O}(\min\{\delta\lambda_{i},N\delta^{2}\lambda_{i}^{2}\}) 1/(N\delta) Iterate averaging Polynomial in N \mathcal{O}(\min\{1/N,N\delta^{2}\lambda_{i}^{2}\} ) 1/(N\delta) Tail averaging Exponential in s \mathcal{O}(\min\{1/(N-s),\delta\lambda_{i},N\delta^{2}\lambda_{i}^{2}\}) 1/((N-s)\delta) and 1/(N\delta) EMA Exponential in N \mathcal{O}(\min\{1-\alpha,\delta\lambda_{i},N\delta^{2}\lambda_{i}^{2}\}) (1-\alpha)/\delta and 1/(N\delta) • We derive the first instance-dependent excess risk bound of the linear regression model trained with SGD with EMA. We also show that the analysis is tight by presenting a lower bound that is almost matching with the upper bound. The excess risk bound consists of the effective bias and the effective bias, both of them further decomposed into each eigen-subspace of the data covariance matrix. Therefore, the excess risk bound is only related to the eigenvalue spectrum, and is irrelevant to the ambient model dimension, making the result applicable to the overparameterized setting. • We compare the excess risk bound of SGD with EMA against SGD without averaging as well as other averaging schemes, e.g., iterate averaging from the beginning and tail averaging, which was studied in Zou et al. (2021) . The comparison is summarized in Table 1 . We show that (i) the effective bias of SGD with EMA decays exponentially in the number of iterations, and (ii) the effective variance of SGD with EMA is smaller than SGD without averaging, and is comparable to that of SGD with iterate averaging or tail averaging. Specifically, we observe a strong connection between EMA and tail averaging in terms of the effective variance: Suppose the tail averaging is performed over the last N-s iterates in a total of N iterations; if \alpha , the averaging parameter of EMA, satisfies (1-\alpha)(N-s)=1 , then the effective variance of SGD with EMA is identical to that of SGD with tail averaging. However, the exponential decay rate of the effective bias can be achieved by SGD with tail averaging only when setting s=\Theta(N) with a known training horizon N . This indicates that SGD with EMA has an advantage over tail averaging in the setting of continual learning. • From a technical viewpoint, we identify a broad class of averaging schemes that covers all averaging methods discussed in this work. Using a standard bias-variance decomposition, we derive a crucial reformulation of the both the bias error and the variance error. Built on this reformulation, an analysis framework for all averaging schemes belonging to this class is developed in this work. Notations. For a vector \mathbf{x} , we use (\mathbf{x})_{i} to denote its i -th entry. We use \otimes to denote the tensor product, and \circ to denote the operation of linear operators on matrices. We use \langle\mathbf{A},\mathbf{B}\rangle\coloneqq\mathop{\mathrm{tr}}(\mathbf{A}\mathbf{B}^{\top}) to denote the inner product of matrices \mathbf{A} and \mathbf{B} . For a PSD matrix \mathbf{A} and a vector \mathbf{x}\in\mathcal{H} , define \|\mathbf{x}\|_{\mathbf{A}}\coloneqq\sqrt{\mathbf{x}^{\top}\mathbf{A}\mathbf{x}} . For any positive

Read the complete text