Evaluating Uncertainty in Deep Gaussian Processes
Matthijs van der Lende · 2025
Read Evaluating Uncertainty in Deep Gaussian Processes by Matthijs van der Lende (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17719.txt.
Introduction
Matthijs van der Lende and co-authors put two Bayesian deep learning methods through a hard test of how honest their uncertainty is, in this 2025 paper. Deep Gaussian processes and their sigma point cousins promise principled confidence estimates. What was missing was a check of how those estimates hold up when the data shift away from the training mix.
The team ran the models on one data set for regression and one for classification, scoring accuracy, calibration, and survival under several artificial shifts. The sigma point model calibrated beautifully on home data. Deep ensembles, a simpler rival, proved tougher under shift on both performance and calibration. The Gaussian methods showed real fragility in places. The authors close by sharing their code and warning that good home calibration is not enough.
Catalog record
| CREATOR | Matthijs van der Lende |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17719
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Uncertainty in Deep Gaussian Processes Matthijs van der Lende Abstract Reliable uncertainty estimates are crucial in modern machine learning. Deep Gaussian Processes (DGPs) and Deep Sigma Point Processes (DSPPs) extend GPs hierarchically, offering promising methods for uncertainty quantification grounded in Bayesian principles. However, their empirical calibration and robustness under distribution shift relative to baselines like Deep Ensembles remain understudied. This work evaluates these models on regression (CASP dataset) and classification (ESR dataset) tasks, assessing predictive performance (MAE, Accuracy), calibration using Negative Log-Likelihood (NLL) and Expected Calibration Error (ECE), alongside robustness under various synthetic feature-level distribution shifts. Results indicate DSPPs provide strong in-distribution calibration leveraging their sigma point approximations. However, compared to Deep Ensembles, which demonstrated superior robustness in both performance and calibration under the tested shifts, the GP-based methods showed vulnerabilities, exhibiting particular sensitivity in the observed metrics. Our findings underscore ensembles as a robust baseline, suggesting that while deep GP methods offer good in-distribution calibration, their practical robustness under distribution shift requires careful evaluation. To facilitate reproducibility, we make our code available at https://github.com/matthjs/xai-gp . 1 Introduction Modern machine learning models, particularly in high-stakes domains like healthcare and autonomous systems, require not only accurate predictions but also reliable estimates of uncertainty. While deep learning models excel at prediction tasks, their deterministic formulation leads to overconfident/uncalibrated uncertainty estimates, especially on out-of-distribution data. Gaussian processes (GPs) [ 23 , 19 ] are nonparametric Bayesian models that provide well-calibrated predictive distributions, effectively capturing epistemic uncertainty—the uncertainty that arises from not knowing the true underlying model. They are used, for instance, in Bayesian optimization, where they act as a probabilistic surrogate model that is used to find promising hyperparameters in a hyperparameter space. There is also an interesting theoretical relationship between neural networks and GPs, namely, as the width of a neural network approaches infinity, the model converges to a GP with a specific kernel determined by their architecture [ 19 ] . Vanilla GPs rely on a kernel or covariance function to define a notion of similarity between datapoints. As a result, the function class that can be modeled is limited by the choice of kernel. Deep GPs (DGPs), introduced as multi-layer compositions of GPs [ 6 ] , inherit the nonparametric flexibility of GPs while enabling richer hierarchical feature learning. More recently, Deep Sigma Point Processes (DSPPs) [ 14 ] have been introduced as an alternative formulation of DGPs that admit a simpler training procedure. Both models theoretically provide well-calibrated uncertainty quantification by propagating uncertainty through layers, yet their empirical calibration performance relative to uncertainty quantification baselines, such as deep ensembles, remains understudied. In particular, using metrics other than the negative log likelihood (NLL), which, although a proper scoring rule, can over-emphasize tail probabilities [ 21 ] . Additionally, there is a lack of empirical results on the use of non-Gaussian likelihoods with DGPs/DSPPs that arise in classification [ 14 ] . In particular, we are interested in determining how well calibrated the uncertainty quantification is of DGPs and DSPPs compared to a baseline (e.g., ensembles) for both regression and classification. For this purpose, we include several standardized uncertainty quantification evaluation methods, like expected calibration error and calibration plots. We will also look at out-of-distribution detection, in particular looking at how calibration behaves under distribution shift, which has not been studied in the model’s respective papers, following the methodology from Ovadia et al. [20] . 2 Background 2.1 Gaussian Processes A GP is a collection of random variables (RVs) \{f_{\text{GP}}(\mathbf{x})\mid\mathbf{x}\in\mathcal{X}\} , any finite number of which has a joint Gaussian distribution [ 23 ] . The index set \mathcal{X} is related to the input of some function f:\mathcal{X}\rightarrow\mathcal{Y} that we want to approximate. It can be interpreted as: At each point \mathbf{x}\in\mathcal{X} , the output of a GP model is a RV denoted f_{\text{GP}}(\mathbf{x}) . A GP, denoted f_{\text{GP}}(\mathbf{x})\sim\mathcal{GP}(m(\mathbf{x}),k(\mathbf{x},\mathbf{x}^{\prime})) , is fully specified by a mean function m:\mathcal{X}\rightarrow\mathbb{R} and covariance or kernel function k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R} which are defined as: \displaystyle m(\mathbf{x}) \displaystyle=\mathbb{E}[f_{\text{GP}}(\mathbf{x})], (1) \displaystyle k(\mathbf{x},\mathbf{x}^{\prime}) \displaystyle=\mathbb{E}[(f_{\text{GP}}(\mathbf{x})-m(\mathbf{x}))(f_{\text{GP}}(\mathbf{x}^{\prime})-m(\mathbf{x}^{\prime}))]. A GP offers a Bayesian approach to nonparametric regression or classification. Without any data, the kernel function represents our prior belief about the function we are trying to model, as it encodes similarity between data points, with closer points having higher covariance. A GP can then be conditioned on a dataset, \mathcal{D} , to get a posterior GP f_{\text{GP}}(\mathbf{x})\sim\mathcal{GP}(m_{\text{post}}(\mathbf{x}),k_{\text{post}}(\mathbf{x},\mathbf{x}^{\prime})) , which is our posterior belief about the function. A common choice of kernel function is the Radial Basis Function (RBF) kernel: k_{\text{RBF}}(\mathbf{x},\mathbf{x}^{\prime};l,\sigma_{f})=\sigma_{f}\exp\bigg(-\frac{\parallel\mathbf{x}-\mathbf{x}^{\prime}\parallel^{2}}{2l^{2}}\bigg), (2) with lengthscale and outputscale parameters l,\sigma_{f}\in\mathbb{R} , which when used result in smooth infinitely differentiable functions being sampled from the GP. Aside from computing the posterior, GP hyperparameters \theta , such as parameters of the kernel, can be fit to data by maximizing the marginal log likelihood (MLL): p(\mathbf{y}|\mathbf{X},\theta)=\int_{\mathbb{R}^{N}}p(\mathbf{y}|\mathbf{f},\mathbf{X})p(\mathbf{f}|\mathbf{X},\theta)\,d\mathbf{f}, (3) where \mathbf{X}=(\mathbf{x}_{1},\ldots\mathbf{x}_{N})\in\mathbb{R}^{N\times D} are data points with targets \mathbf{y}\in\mathbb{R}^{N} where y_{i}=f(\mathbf{x}_{i})+\epsilon_{y},\epsilon_{y}\sim\mathcal{N}(0,\sigma_{y}) and \mathbf{f}\in\mathbb{R}^{N} are the instantiations of the function latent variables f_{\text{GP}}(\mathbf{x}_{i}) for each \mathbf{x}_{i} . Both the MLL p(\mathbf{y}|\mathbf{X},\theta) and the posterior distribution p(\mathbf{f}_{*}|\mathbf{X},\mathbf{y},\mathbf{X}_{*}) for test points \mathbf{X}_{*} can be computed in closed form, but this required an inversion of the kernel matrix over all data points, which is O(N^{3}) in time and O(N^{2}) in space. 2.2 Sparse Variational Gaussian Processes Sparse Variational Gaussian Processes (SVGPs) [ 26 , 12 ] approximate the GP predictive posterior distribution through variational inference 1 1 1 See Appendix B for a quick overview of variational inference. . This is done by introducing inducing variables (f_{\text{GP}}(\mathbf{z}_{1}),\ldots,f_{\text{GP}}(\mathbf{z}_{M}))^{\top}=\mathbf{u} that depend on variational parameters \mathbf{Z}=(\mathbf{z}_{1},\ldots,\mathbf{z}_{M}) called inducing points where M\ll N . The inducing points together with the associated inducing variables serve as an approximation of the full dataset \mathcal{D}=(\mathbf{x}_{i},y_{i})_{i=1,...,N} that the GP conditions on. SVGPs consider the following variational distribution: q(\mathbf{f},\mathbf{u})=p(\mathbf{f}|\mathbf{u},\mathbf{X},\mathbf{Z})q(\mathbf{u})\quad q(\mathbf{u})=\mathcal{N}(\mathbf{m},\mathbf{S}). (4) For an SVGP, the variational parameters are \psi=\{\mathbf{Z},\mathbf{m},\mathbf{S}\} but may also include more parameters such as kernel hyperparameters. The approximate posterior over the function values is calculated by marginalizing over the inducing variables: p(\mathbf{f}_{*}|\mathcal{D},\mathbf{X}_{*})\approx\int_{\mathbb{R}^{M}}p(\mathbf{f}_{*}|\mathbf{u})q(\mathbf{u})\;d\mathbf{u}. (5) The variational objective is derived by maximizing the Evidence Lower Bound (ELBO), which in this context becomes: L_{\text{SVGP}}(\psi|\mathcal{D})=\sum^{N}_{i=1}\mathbb{E}_{q(f_{\text{GP}}(\mathbf{x}_{i}))}[\log p(y_{i}|f_{i})]-D_{\mathbb{KL}}(q(\mathbf{u})\parallel p(\mathbf{u}|\mathbf{Z})), (6) where p(y_{i}|f_{i}) is the likelihood for the observations given latent function values f_{\text{GP}}(\mathbf{x}_{i})=f_{i} . It can be shown that the complexity now becomes O(NM^{2}) in time and O(NM+M^{2}) in space [ 19 ] . It is worth noting that as M increases, the approximation quality of exact inference is recovered. Too few inducing points may make the GP behave as if it was underfitting [ 2 ] . 2.3 Deep Gaussian Processes and Deep Sigma Point Processes DGPs extend the GP framework to multiple layers, allowing for the construction of hierarchies of latent functions [ 6 , 19 ] : \begin{array}[]{l l}\mathcal{DGP}(\mathbf{x})&=\bm{f}_{L}\circ\cdots\circ\bm{f}_{1}(\mathbf{x}),\\ &\bm{f}_{i}(\cdot)=[f_{\text{GP},i}^{(1)}(\cdot),\ldots,f_{\text{GP},i}^{(H_{i})}(\cdot)]^{\top},\\ &f_{\text{GP},i}^{(j)}\sim\mathcal{GP}(m_{i}(\cdot),k_{i}(\cdot,\cdot)).\end{array} (7) DGPs have a neural network-like structure with L layers, each containing H GPs. Posterior inference in GPs is no longer tractable as it requires marginalizing over a large number of RVs, corresponding to the latent function values at each layer. The stochastic variational inference method from Section 2.2 can be generalized to DGPs [ 24 ] , known as doubly stochastic variational inference. Each layer is agumented with inducing variables \mathbf{u}^{(l)} and corresponding variational distribution q(\mathbf{u}^{(l)}) . The variational ELBO for DGPs is