Gaussian Mean Field Variational Inference can Overestimate Predictive Variance

James Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin · 2026

Read Gaussian Mean Field Variational Inference can Overestimate Predictive Variance by James Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25745.txt.

Introduction

Everyone knows mean field variational inference, the workhorse approximation for Bayesian posteriors, cheats by understating uncertainty. James Odgers, Ben Riegler, Siddharth Swaroop, and Vincent Fortuin show in 2026 that the story flips where it matters. Working with conjugate Bayesian linear regression, where the exact posterior is known, they prove the approximation must rob variance from some directions and overpay others. The overpaid directions are exactly where the training data concentrate. For a fresh test point drawn from the training distribution itself, the method's expected predictive variance actually exceeds the exact posterior's.

A pathological case drives the point home: on in distribution data the approximation can fail to reduce predictive variance below the prior at all. The results connect to the cold posterior effect in Bayesian deep learning. Cooling the posterior's temperature can correct the overestimation and pull predictions back toward the exact answer.

Catalog record

CREATORJames Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25745

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaussian Mean Field Variational Inference can Overestimate Predictive Variance James Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin Abstract Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI’s expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks. 1 Introduction (a) Input space (b) Parameter space Figure 1 : Although the MFVI posterior underestimates variance on average, it overestimates variance along the direction in which the data lie. Here we show this on a simple 2D linear regression, where the input data is restricted to lie on the subspace x_{1}=x_{2} ( Figure 1(a) ). We see that, for most of the input space, MFVI predictive uncertainty is less than the exact posterior predictive (blue region). At the same time, along the data direction, the opposite is true (red region). This is also the case when looking at the weight-space posterior ( Figure 1(b) ), with the exact posterior clearly taking up more volume, but also with the MFVI posterior having a larger variance along the crucial \theta_{1}=\theta_{2} subspace. Bayesian approaches to machine learning offer compelling advantages, including a principled approach to uncertainty estimation, and learning hyperparameters without overfitting ( 18 ) . Unfortunately, the full Bayesian posterior, p(\theta|\mathcal{D}) , is usually intractable and needs to be approximated. One of the most popular approaches to approximate this posterior is Variational Inference (VI) ( 14 ; 5 ) . Here, a family of distributions is proposed, \mathcal{Q} , from which the distribution q^{*}(\theta) is optimized, which minimises the reverse KL divergence to the true posterior, q^{*}(\theta)=\arg\min_{q\in Q}D_{KL}\left(q(\theta)||p(\theta|\mathcal{D})\right) . Practitioners often use Mean Field Variational Inference (MFVI), in which the variational posterior factorises across parameters, q(\theta)=\prod_{p=1}^{P}q_{p}(\theta_{p}) . In practice, the marginal distributions are often restricted to be Gaussian, so q_{p}(\theta_{p})=N(\mu_{p},\sigma^{2}_{p}) . We study this popular setting in this paper. MFVI is typically considered to underestimate the uncertainty of the posterior ( 25 ; 35 ; 5 ) . This is because it is a mode-fitting approximation in parameter space: the approximate posterior avoids assigning probability mass to regions with low true probability, thereby underestimating the variance in parameter space ( 21 ) . However, in this paper, we argue that this picture is incomplete once the goal is prediction rather than parameter inference. In particular, the predictions from an MFVI posterior can overestimate uncertainty on In-Distribution (ID) data. Illustrative example ( Figure 1 ). Figure 1(a) shows some simple training data restricted to the subspace x_{1}=x_{2} , and Figure 1(b) shows the exact posterior and MFVI posterior (in parameter space) for this linear regression example. The highly uneven, non-axis-aligned covariance of the training inputs produces a posterior with a strong covariance between \theta_{1} and \theta_{2} in the parameter space. We can see how MFVI appears to underestimate the variance in general: its mass is over a smaller region of the parameter space than the mass of the exact posterior. However, in this example, we are not interested in the parameters learned by the model; instead, we care about the predictions made on test data. A realistic assumption is often that future data points are similar to training data points, such that x_{1}=x_{2} . If this is the case, the only direction in parameter space relevant for predictions is \theta_{1}=\theta_{2} , as illustrated by the dashed line in Figure 1(b) . In this direction, the MFVI posterior overestimates the variance compared to the true posterior. Our main result, given in Theorem 3.7 , is to show that the overestimation of predictive variance in MFVI in Bayesian Linear Regression (BLR) is very general. A Cold Posterior Effect without model mismatch. The Cold Posterior Effect (CPE) from Bayesian Deep Learning (BDL) ( 37 ) , describes an effect where artificially reducing the uncertainty of an approximate posterior improves predictive performance. In this work, we add another justification for why we may expect the CPE: lowering the temperature corrects for overestimated predictive variance from MFVI on in-distribution data, producing predictions which match the exact posterior predictions more closely. We demonstrate that the reverse effect is true for Out-Of-Distribution (OOD) data, where a warm posterior matches the exact posterior predictive distribution more closely and performs better on predictive tasks. Contributions. In this paper, our main contributions are: • We provide a theoretical analysis of conjugate Bayesian Linear regression, showing that the MFVI posterior overestimates the exact posterior variance along the first principal component of the training data, and that the average predicted variance at the training data points is higher for MFVI than the exact posterior. • We theoretically and empirically examine the predictive performance of MFVI on a pathological setup and demonstrate that in the high-dimensional limit, the MFVI posterior can fail to reduce the predictive variance at all compared to the prior. • We show theoretically and empirically that both a Cold Posterior Effect and Warm Posterior Effect can occur in linear regression models approximated with MFVI, and argue that both of these effects can be understood as scaling the predictive variance of the MFVI approximation to better match the exact posterior predictions. 2 Related Work Variational Inference (VI). VI research has mostly focused on the approximation quality in parameter space, not the approximation quality on predictions (see 5 for an example of this). One line of related research is Functional VI ( 33 ; 6 ; 7 ) , to allow more interpretable priors, such as GPs, to be applied to BNNs. In our linear regression setting, there is a one-to-one mapping between functions and parameters, so there is no difference in the functional and parametric variational objectives ( 6 , see Proposition 3,) . A functional approach to VI is also common in Gaussian Processes ( 34 ) to improve computational efficiency. We do not consider these approaches in this paper, as these approaches are equivalent to different variational families in parameter space which we do not study here. The paper closest to the theoretical analysis we provide is 21 , who studied the theoretical properties of the Gaussian MFVI posterior when the exact posterior is a Multivariate Gaussian. Crucially, this current work focuses on the properties of the predictions of VI, whereas 21 focus on the properties in parameter space. Cold Posterior Effect (CPE). There are several works proposing explanations for the CPE, showing how this effect can naturally arise from data augmentation ( 13 ; 26 ; 4 )

Read the complete text