Variational Learning Finds Flatter Solutions at the Edge of Stability
Avrajit Ghosh, Bai Cong, Rio Yokota, Saiprasad Ravishankar, Rongrong Wang, Molei Tao, Mohammad Emtiyaz Khan, Thomas Möllenhoff · 2025
Read Variational Learning Finds Flatter Solutions at the Edge of Stability by Avrajit Ghosh, Bai Cong, Rio Yokota, Saiprasad Ravishankar, Rongrong Wang, Molei Tao, Mohammad Emtiyaz Khan, Thomas Möllenhoff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12903.txt.
Introduction
Avrajit Ghosh and seven co-authors study why a Bayesian-flavored training method finds flat solutions in this 2025 paper. Variational learning trains neural networks while tracking uncertainty over the weights. Flat solutions, where small weight changes barely hurt performance, tend to generalize better.
The team analyzes the method through the edge of stability, a lens used to explain how gradient descent settles into flat spots. They prove the variational method can land in even flatter spots than plain descent. Two knobs control the effect: the shape of the weight distribution and the number of samples drawn during training. Tests on large image networks matched the theory closely. The authors say theirs is the first such analysis of this training method.
Catalog record
| CREATOR | Avrajit Ghosh, Bai Cong, Rio Yokota, Saiprasad Ravishankar, Rongrong Wang, Molei Tao, Mohammad Emtiyaz Khan, Thomas Möllenhoff |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-12903
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variational Learning Finds Flatter Solutions at the Edge of Stability Avrajit Ghosh, Bai Cong, Rio Yokota, Saiprasad Ravishankar, Rongrong Wang, Molei Tao, Mohammad Emtiyaz Khan, Thomas Möllenhoff Abstract Variational Learning (VL) has recently gained popularity for training deep neural networks. Part of its empirical success can be explained by theories such as PAC-Bayes bounds, minimum description length and marginal likelihood, but there are few tools to unravel the implicit regularization in play. Here, we analyze the implicit regularization of VL through the Edge of Stability (EoS) framework. EoS has previously been used to show that gradient descent can find flat solutions and we extend this result to VL to show that it can find even flatter solutions. This is obtained by controlling the posterior covariance and the number of Monte Carlo samples from the posterior. These results are derived in a similar fashion as the standard EoS literature for deep learning, by first deriving a result for a quadratic problem and then extending it to deep neural networks. We empirically validate these findings on a wide variety of large networks, such as ResNet and ViT, to find that the theoretical results closely match the empirical ones. Ours is the first work to analyze the EoS dynamics in VL. 1 Introduction Variational Learning (VL) has been used to perform deep learning from early on ( Graves 2011 ; Blundell et al. 2015 ) and recently also started to show good results at large scale. It can now outperform state-of-the-art optimizers without any increase in the cost. For example, on ImageNet, VL substantially improves overfitting commonly seen in AdamW and for pretraining GPT-2 from scratch, VL achieves a lower validation perplexity than AdamW ( Shen et al. 2024 ) . For low-rank fine-tuning of Llama-2 (7B), VL improves both accuracy (by 2.8\% ) and calibration (by 4.6\% ) ( Cong et al. 2024 ; Li et al. 2025 ) . Moreover, VL methods explicitly derived from minimizing PAC-Bayes bounds ( Wang et al. 2023b ; Zhang et al. 2024b ) have shown consistent improvements over AdamW. All such results confirm the importance of VL for deep learning. Despite these successes, the theoretical mechanisms behind the good performance of VL remain poorly understood. It is often assumed that simplistic Gaussian posteriors, such as those used currently for deep learning, may not be enough because they are poor approximations of the true posterior; lack of a good prior is another issue. Despite these concerns, VL shows good performance in practice. Theories such as minimum description length ( Hinton and Van Camp 1993 ; Hochreiter and Schmidhuber 1997 ; Blier and Ollivier 2018 ) , PAC-Bayes ( Dziugaite and Roy 2017 ; Zhou et al. 2019 ; Lotfi et al. 2024 ) and marginal likelihood ( Smith and Le 2017 ; Immer et al. 2021 ) can partially explain the success. In particular, PAC-Bayes theory provides a natural explanation for why flatter solutions may generalize better, see for instance the discussion in ( Alquier et al. 2024 , Section 3.3) . However these theories say little about the regularizing properties of the learning algorithm. Similarly to deep learning, the presence of implicit regularization is likely also at play in VL, but few tools exist (if any) to unravel these effects. In this work, we analyze the implicit regularization of VL algorithms at the EoS. The EoS analysis has previously been used to show that Gradient Descent (GD) with constant learning rate \rho implicitly biases the trajectories towards flatter solutions, where the sharpness (defined as the operator norm of the loss Hessian) hovers around 2/\rho . Here, we extend this analysis to VL and show that the sharpness can be made lower by controlling the posterior covariance and the number of Monte-Carlo samples used to compute posterior expectations. Similarly to the standard EoS technique, we first derive an exact expression for the stability threshold for VL on a quadratic problem (Figure 1 a) and then propose extensions to general loss functions (Figure 1 b). We then empirically validate these finding on a wide variety of networks, including Multi-Layer Perceptron, ResNet, and Vision Transformers. We observe similar results when posterior shape is automatically learned, for instance, by using diagonal covariance Gaussians and heavy-tailed posteriors. (a) (b) Figure 1 : (a) The left figure shows trajectory traces of VL on a quadratic problem with an isotropic posterior whose mean is learned but variance is fixed. The trajectory becomes more unstable as the variance is increased (columns) and number of Monte-Carlo samples are decreased (rows). We provide an exact expression to compute the stability threshold at which the iterations become unstable (Theorem 3.1 ). We show the validity of the threshold on neural network training. (b) The right figure (top) shows this on CIFAR-10 for an MLP where VL achieves lower sharpness than GD by increasing the posterior variance. The bottom figure shows that the sharpness (solid line) matches the stability threshold obtained by our theorem (dashed line). 2 Theoretical Tools to Analyze Variational Learning Variational Learning optimizes the variational reformulation of Bayesian learning ( Zellner 1988 ) , where the goal is to find good approximations of the Gibbs distribution \exp(-\ell(\boldsymbol{\theta}))/\mathcal{Z} with partition function \mathcal{Z} over parameters \boldsymbol{\theta}\in\mathbb{R}^{d} . Specifically, we seek the closest distribution q(\boldsymbol{\theta}) in a set of distributions \mathcal{Q} that minimizes the expected loss regularized by the entropy: \mathop{\mathrm{arg\,min}}_{q\in\mathcal{Q}}\;\mathbb{E}_{\boldsymbol{\theta}\sim q}[\ell(\boldsymbol{\theta})]-\mathcal{H}(q). (1) This objective is equivalent to minimizing the KL divergence to the Gibbs posterior and naturally encourages higher-entropy (flatter) solutions due to the entropy \mathcal{H}(q) ; for example, see the illustrative example in Khan and Rue 2023 . Similar arguments can also be made with the minimum-description-length principles ( Hinton and Van Camp 1993 ; Graves 2011 ; Blier and Ollivier 2018 ) . In practice, Variational Learning has started to show good results achieving generalization performance better than the state of the art optimizers across several tasks ( Shen et al. 2024 ; Cong et al. 2024 ) . This matches with the intuition: because the variational objective prefers wider distributions, we expect the posterior to be located in the region where the loss \ell(\boldsymbol{\theta}) is flatter. Despite this intuition, there is little work to analyze the implicit regularization that could help us understand how VL favors flatter regions and how we can control it. The answer could lie in the shape of the posterior but currently there are no results explicitly characterization the properties of the posterior responsible for improved generalization. Variational learning is also closely connected to weight-noise or weight-perturbation methods where the goal is to optimize \mathbb{E}_{q(\boldsymbol{\epsilon)}}[\ell(\boldsymbol{\theta}+\boldsymbol{\epsilon)}] where q(\boldsymbol{\epsilon)} is a fixed distribution to inject weight noise. This can be seen as a special case of variational learning where the shape of the posterior is fixed and only the location is learned. Multiple works ( Zhu et al. 2019 ; Nguyen et al. 2019 ; Zhang et al. 2019 ; Jin et al. 2017 ; Simsekli et al. 2019 ) have analyzed generalization behavior of such weight-noise/variational methods but, even for this simple case, there are no studies connecting them to the EoS results for GD. In this paper, we will address these gaps and provide both theoretical and empirical results