SVRG and Beyond via Posterior Correction

Nico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan · 2025

Read SVRG and Beyond via Posterior Correction by Nico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01930.txt.

Introduction

Stochastic Variance Reduced Gradient, or SVRG, is a training method that corrects noisy gradient steps with occasional exact ones to speed up learning. Nico Daheim and three co-authors show in 2025 that this older method has surprising, deep connections to a recent Bayesian technique called posterior correction.

SVRG and its variants promised faster training yet never caught on widely in deep learning. Viewing the method through the Bayesian lens explains its behavior in a new way and suggests fresh variants. The connection also gives posterior correction, developed for a different purpose, an unexpected second life as a tool for understanding optimization.

Catalog record

CREATORNico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SVRG and Beyond via Posterior Correction Nico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan Abstract Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections, but have seen limited success in deep learning. Here, we show surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction. Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families. We derive two new SVRG variants by using Gaussian families: First, a Newton-like variant that employs novel Hessian corrections, and second, an Adam-like extension that improves pretraining and finetuning of Transformer language models. This is the first work to connect SVRG to Bayes and use it to boost variational training for deep networks. 1 Introduction Variance Reduction is a powerful technique to speed-up stochastic optimization. For example, stochastic variance reduced gradient (SVRG) uses full-batch gradients to stabilize future mini-batch updates ( Johnson & Zhang 2013 ) . The method originates in the works of Roux et al. 2012 ; Schmidt et al. 2017 ; Shalev-Shwartz & Zhang 2013 which require storing individual gradients over the full dataset. Since then, a large number of variants have been proposed exploring various aspects of this method ( Nguyen et al. 2017 ; Fang et al. 2018 ; Cutkosky & Orabona 2019 ) . Variance reduction has become a useful tool to accelerate both convex and nonconvex optimization. Our goal here is to explore new connections of SVRG to Bayes. Currently, no work exists in this space and no foundational connections are known. We are particularly interested in investigating whether variance reduction can be effective in non-traditional settings, for example, in variational continual pre-training and fine-tuning of language models ( Shen et al. 2024 ) . Variance reduction has not yet seen a lot of success in deep learning ( Defazio & Bottou 2019 ) . Our exploration here is meant to assess its potential in non-traditional variational training. In this paper, we present surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction ( Khan 2025 ) . Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families ( Fig. 1 (left)). This result is surprising because posterior correction is not a variance reduction method, rather a knowledge transfer method. The result offers a new perspective of variance reduction as knowledge transfer, for instance, through frequent mega-batch gradient computations. Using this result, we derive new SVRG variants that go beyond existing proposals. For example, by using full-covariance Gaussians, we obtain a new variance-reduction method that employs Hessian corrections within a variational Online Newton algorithm. This differs from most works on Newton steps that only use corrections for the gradient and never for the Hessian ( Derezinski 2023 ; Sadiev et al. 2024 ; Garg et al. 2024 ; Sun et al. 2025 ) . Another Adam-like extension is obtained by using diagonal covariances, implementing posterior correction over the IVON optimizer ( Shen et al. 2024 ) . Empirically, the new variant boosts IVON’s speed in non-traditional settings and shows promising results at scales up to pretraining an LLM from scratch on 50B tokens; see the middle and right panels Fig. 1 . In this setting, our method outperforms Adam too, which is unlike many other traditional applications where no effective gains in performance are observed. This result also far exceeds the scale of any prior work on SVRG-based methods that use mega-batches. We validate these findings with various small to large experiments on several architectures, including ResNet, GPT and ViT. Overall, our work encourages further investigation of the new connection to obtain real improvements with SVRG in deep learning. (a) SVRG vs. PoCo (b) GPT-2 (125M) pretraining (c) Different correction starts Figure 1: (a) We present a generalization of SVRG by using Posterior Correction (PoCo) where gradients used in SGD are replaced by natural gradients of VB objectives via the Bayesian Learning Rule (BLR). (b) Our new IVON-PoCoMo (red) improves performance over IVON and AdamW when pretraining GPT2-125M from scratch on ca. 50B tokens from OpenWebText. Until 50K steps IVON-PoCo takes the same steps as IVON, and a huge boost is obtained when correction is started. Validation Perplexities at the end are 17.4, 18.0, 18.4. (c) We show three different IVON-PoCo runs where correction is started at a different iteration (pink to red). We see consistent improvements irrespective of the starting iteration. Algorithm 1 SVRG 1: Initialize \boldsymbol{\theta}_{\text{in}} 2: while not converged do 3: \mathbf{g}_{\text{out}}\leftarrow\sum_{i=1}^{N}\nabla\ell_{i}(\boldsymbol{\theta}_{\text{in}}) 4: \boldsymbol{\theta}_{\text{out}}\leftarrow\boldsymbol{\theta}_{\text{in}} 5: for t=1,2,\ldots,m do 6: Randomly pick i\in\{1,2,\ldots,N\} 7: \mathbf{g}_{\text{in}}\leftarrow\nabla\ell_{i}(\boldsymbol{\theta}_{\text{in}})-\nabla\ell_{i}(\boldsymbol{\theta}_{\text{out}})+\frac{1}{N}\mathbf{g}_{\text{out}} 8: \boldsymbol{\theta}_{\text{in}}\leftarrow\boldsymbol{\theta}_{\text{in}}-\eta\mathbf{g}_{\text{in}} 9: end for 10: end while Algorithm 2 Posterior Correction (PoCo) 1: Initialize \boldsymbol{\lambda}_{\text{in}} 2: while not converged do 3: \widetilde{\mathbf{g}}_{\text{out}}\leftarrow\sum_{i=1}^{N}\widetilde{\nabla}\mathcal{L}_{i}(\boldsymbol{\lambda}_{\text{in}}) 4: \boldsymbol{\lambda}_{\text{out}}\leftarrow\boldsymbol{\lambda}_{\text{in}} 5: for t=1,2,\ldots,m do 6: Randomly pick i\in\{1,2,\ldots,N\} 7: {\widetilde{\mathbf{g}}_{\text{in}}\leftarrow\widetilde{\nabla}\mathcal{L}_{i}(\boldsymbol{\lambda}_{\text{in}})-\widetilde{\nabla}\mathcal{L}_{i}(\boldsymbol{\lambda}_{\text{out}})+\frac{1}{N}\widetilde{\mathbf{g}}_{\text{out}}} 8: \boldsymbol{\lambda}_{\text{in}}\leftarrow(1{\color[rgb]{1,0,0}-\eta})\boldsymbol{\lambda}_{\text{in}}-\eta{\color[rgb]{1,0,0}N}\widetilde{\mathbf{g}}_{\text{in}} 9: end for 10: end while Figure 2: Pseudo-code for SVRG (left) and our Bayesian generalization using PoCo (right). The latter replaces all instances of \boldsymbol{\theta} and \nabla\ell_{i} in SVRG by the natural parameter \boldsymbol{\lambda} (of the posterior q ) and natural gradient \widetilde{\nabla}\mathcal{L}_{i} . It also uses the BLR for the inner loop in line 8. We show that these differences disappear with an isotropic Gaussian q and SVRG is recovered as a special case of PoCo. 2 Background on SVRG and Variational Bayes In SVRG, the goal is to minimize an empirical risk \sum_{i=1}^{N}[\ell_{i}(\boldsymbol{\theta})/N] averaged over losses \ell_{i} for examples i=1,2,\ldots,N . A regularizer, denoted by \ell_{0} , is often added and handled by redefining the losses as \ell_{i}+\ell_{0}/N . Standard stochastic gradient descent (SGD) steps use stochastic gradients, for instance, the following update where the i ’th example is randomly sampled at each iteration, \boldsymbol{\theta}\leftarrow\boldsymbol{\theta}-\eta\nabla\ell_{i}(\boldsymbol{\theta}). (1) SVRG reduces the variance of such steps by using an outer loop where full-batch gradients are computed at parameters \boldsymbol{\theta}_{\text{out}} , and then parameters \boldsymbol{\theta}_{\text{in}} are updated using stochastic increments at a randomly sampled i at every iteration, \boldsymbol{\theta}_{\text{in}}\leftarrow\boldsymbol{\theta}_{\text{in}}-\eta\left[\nabla\ell_{i}(\boldsymbol{\theta}_{\text{in}})-\nabla\ell_{i}(\boldsymbol{\theta}_{\text{out}})+\frac{1}{N}\sum_{j=1}^{N}\nabla\ell_{j}(\boldsymbol{\theta}_{\text{out}})\right]. (2) These steps can be seen as a full-batch gradient descent where old gradients \nabla\ell_{i}(\boldsymbol{\theta}_{\text{out}}) are ‘corrected’ by adding the fresh new gradients \nabla\ell_{i}(\boldsymbol{\theta}_{\text{in}}) . The use of the full-batch gradient can reduce variance and speed up future mini-batch steps. Pseudo-code is shown in Algorithm 1 where m steps are taken in the inner loop and a constant learning rate \eta is used. SVRG is built upon earlier ideas in Stochastic Average Gradient (SAG) ( Roux et al. 2012 ; Schmidt et al. 2017 ) and Stochastic Dual Coordinate Descent (SDCA) ( Shalev-Shwartz & Zhang 2013 ) methods where the full batch gradient is stored for all examples separately and entries are refreshed whenever corresponding examples are chosen during updating. Subsequently, many new practical variants have been proposed, for instance, SARAH ( Nguyen et al. 2017 ) and SPIDER ( Fang et al. 2018 ) , among many other proposals ( Dubois-Taine et al. 2022 ; Lei et al. 2017 ; Babanezhad Harikandeh et al. 2015 ) . Instead of full-batch, it is also possible to use

Read the complete text