Can Bayesian Neural Networks Make Confident Predictions?

Katharine Fisher, Youssef Marzouk · 2025

Read Can Bayesian Neural Networks Make Confident Predictions? by Katharine Fisher, Youssef Marzouk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11773.txt.

Introduction

Katharine Fisher and Youssef Marzouk ask whether Bayesian neural networks can make confident predictions, in two thousand twenty-five. Such networks promise a principled way to state how unsure a prediction is. In practice, the full spread of plausible network settings is hard to map, and its meaning is murky.

Under a stepped prior on the inner weights, the pair can write the prediction spread exactly as a mix of bell curves. That form lets them group settings that give the same training error. They find settings where unlike settings fit training data equally well yet predict unlike outcomes. They also chart when the prediction spread tightens as more data arrives.

Catalog record

CREATORKatharine Fisher, Youssef Marzouk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Bayesian Neural Networks Make Confident Predictions? Katharine Fisher, Youssef Marzouk Abstract Bayesian inference promises a framework for principled uncertainty quantification of neural network predictions. Barriers to adoption include the difficulty of fully characterizing posterior distributions on network parameters and the interpretability of posterior predictive distributions. We demonstrate that under a discretized prior for the inner layer weights, we can exactly characterize the posterior predictive distribution as a Gaussian mixture. This setting allows us to define equivalence classes of network parameter values which produce the same likelihood (training error) and to relate the elements of these classes to the network’s scaling regime—defined via ratios of the training sample size, the size of each layer, and the number of final layer parameters. Of particular interest are distinct parameter realizations that map to low training error and yet correspond to distinct modes in the posterior predictive distribution. We identify settings that exhibit such predictive multimodality, and thus provide insight into the accuracy of unimodal posterior approximations. We also characterize the capacity of a model to “learn from data” by evaluating contraction of the posterior predictive in different scaling regimes. 1 Introduction Uncertainty is key to learning. Questions of how to quantify neural network prediction uncertainty are inextricable from questions of how expressive models learn to generalize [ 63 , 64 , 51 ] . Progress on these questions has been made through analysis of relatively simple networks, including random feature models [ 52 ] and neural tangent kernels [ 36 ] , which demonstrate the double descent phenomenon [ 7 , 6 , 46 , 11 , 1 ] . An array of uncertainty metrics have been proposed for neural networks, as detailed by [ 21 ] , but most approaches rely on heuristics which make interpretation challenging even in simple networks. Bayesian neural networks (BNNs) promise a principled framework for obtaining predictive distributions conditioned on training data [ 47 , 45 , 4 ] . Realizing this promise has been complicated by the need to design appropriate prior and likelihood models and to characterize multimodal posterior distributions. Locating all modes via sampling is generally intractable, though mode connectivity and algorithm choice may aid in discovering parameter values that successfully generalize [ 20 , 35 , 55 , 49 ] . Many strategies for approximate inference in BNNs have also been developed. The Laplace approximation [ 16 , 44 ] represents the predictive distribution with a single mode. Variational inference methods [ 29 , 5 , 14 ] are more flexible, but typically capture at most a few posterior modes. Such approaches seem to risk underestimating uncertainty, though the debate about “cold posteriors” has raised the possibility that narrower distributions may produce better generalization [ 61 ] . Partially Bayesian networks [ 38 , 58 ] could offer uncertainty estimates without introducing the challenge of learning distributions over all parameters. Broadly, however, there is a need for tools that provide insight into what these approximations of the Bayesian posterior miss. In this work, we demonstrate that adopting a discrete prior on the inner layer weights of a BNN is a useful tool for accessing the predictive distribution without exhaustively sampling parameter space. Such priors allow us to identify cases where different posterior modes map to distinct modes in the predictive distribution. Then we can determine when predictions based on a single posterior mode will fail. To the authors’ knowledge, this approach to analyzing multimodality is unique, though the different treatment of inner and final layer parameters during inference bears some resemblance to work on subnetwork inference [ 15 , 58 ] , partial Bayesian networks [ 38 , 58 ] , and random feature models [ 52 ] . Furthermore, characterizing the posterior predictive distribution allows us to identify settings where the predictive uncertainty does not contract as the network and training set size grow proportionally. This behavior raises the question of whether overparameterized BNNs can produce “confident predictions,” i.e., predictions whose posterior distribution contracts around the truth as the network and data set size grow. Section 2 outlines our model and approach to inference. Sections 3 and 4 examine the impact of network and training set size on predictive uncertainty for a discretized Gaussian prior and then for a prior which puts mass on optimal parameter values. We put our findings in the context of related work in Section 5 , then conclude with a discussion of the implications of multimodality for approximate inference tools and the role of Bayesian uncertainty in successful generalization. 2 Predictive distribution of a Bayesian neural network We consider an L -layer neural network in a regression setting, 1 1 1 Extensions to classification settings are provided in Appendix A.1 . \displaystyle\hat{y} \displaystyle= \displaystyle w^{\top}x_{L}, (1) \displaystyle x_{\ell} \displaystyle= \displaystyle\sigma(\Theta_{\ell-1}^{\top}x_{\ell-1}\ -\ b_{\ell-1}),\ \qquad\ 1<\ell\leq L, (2) where x_{1}\in\mathbb{R}^{d} is the network input, \hat{y}\in\mathbb{R} is the output, and \sigma is a nonlinear activation function that operates component-wise. The trainable parameters include the final layer weights w\in\mathbb{R}^{p} and interior parameters \Theta\coloneqq\{\Theta_{\ell}\in\mathbb{R}^{d_{\ell}\times d_{\ell+1}},\ b_{\ell}\in\mathbb{R}^{d_{\ell}}\}_{\ell=1}^{L-1} . Note that d_{1}=d and d_{L}=p . We make the prior assumption that \displaystyle w\sim\mathcal{N}(0,p^{-1}\mathbf{I}_{p}),\qquad\mathbb{P}(\Theta=\Theta^{(j)})=\rho_{j},\qquad\sum_{j=1}^{J}\rho_{j}\ =\ 1, (3) where each \Theta^{(j)} is a fixed realization of the interior parameters. Crucially, the discrete prior on \Theta allows us to derive an analytical representation of the Bayesian posterior predictive distribution. Our training set has the form \{(x_{1}^{(i)},y^{(i)})\}_{i=1}^{n} where we assume that \displaystyle y^{(i)}=g(x_{1}^{(i)})\ +\ \varepsilon^{(i)},\qquad\varepsilon^{(i)}\overset{\text{iid}}{\sim} \displaystyle N(0,\,\gamma^{2}), (4) for some (unknown) data-generating function g:\mathbb{R}^{d}\to\mathbb{R} . For convenience, we define X_{\ell}\coloneqq[x_{\ell}^{(1)},\dots,x_{\ell}^{(n)}]\in\mathbb{R}^{d_{\ell}\times n} , for any \ell\in[L] , and Y\coloneqq(y^{(1)},\dots,y^{(n)})\in\mathbb{R}^{n} .The training data can thus be written more concisely as (X_{1},Y) . Note that we impose no assumption on the distribution of X_{1} . Let \widetilde{x}_{1}\in\mathbb{R}^{d} be an input value at which we will test our network predictions and let \widetilde{y}\in\mathbb{R} denote the corresponding output. Under our model assumptions, the posterior predictive density for \widetilde{y} is a J -component Gaussian mixture: \displaystyle\pi(\widetilde{y}\,|\,X_{1},Y,\widetilde{x}_{1}) \displaystyle= \displaystyle\ \sum_{j=1}^{J}\mathbb{P}(\Theta^{(j)}\,|\,X_{1},Y)\ \pi(\widetilde{y}\,|\,X_{1},Y,\widetilde{x}_{1},\Theta^{(j)}). (5) For each j , Bayesian linear regression yields \displaystyle\pi(\widetilde{y}\,|\,X_{1},Y,\widetilde{x}_{1},\Theta^{(j)}) \displaystyle= \displaystyle\mathcal{N}\bigl(\widetilde{y};\,p^{-1}\widetilde{x}_{L}^{\top}X_{L}(p^{-1}X_{L}^{\top}X_{L}\ +\ \gamma^{2}\mathbf{I})^{-1}Y, (6) \displaystyle\qquad\gamma^{2}\ +\ \gamma^{2}p^{-1}\widetilde{x}_{L}^{\top}(p^{-1}X_{L}X_{L}^{\top}\ +\ \gamma^{2}\mathbf{I})^{-1}\widetilde{x}_{L}\bigr), where dependence on \widetilde{x}_{1} in the mean and variance terms above enters via \widetilde{x}_{L} , as described in ( 2 ). Note that both X_{L} and \widetilde{x}_{L} depend on \Theta^{(j)} . By Bayes’ rule, the mixture weights are \displaystyle\mathbb{P}(\Theta^{(j)}\big|X_{1},Y)\ =\ \frac{\rho_{j}\,\pi(Y|X_{1},\Theta^{(j)})}{\pi(Y|X_{1})}\ =\ \left(1+\sum_{k\neq j}\frac{\rho_{k}\pi(Y|X_{1},\Theta^{(k)})}{\rho_{j}\pi(Y|X_{1},\Theta^{(j)})}\right)^{-1}, (7) where \displaystyle\pi(Y|X_{1},\Theta^{(j)})=\mathcal{N}\bigl(Y;\mathbf{0},\ p^{-1}X_{L}^{T}X_{L}+\gamma^{2}\mathbf{I}\bigr)\eqqcolon\mathcal{L}(X_{L}(\Theta^{(j)})) (8) is the marginal likelihood function for \Theta^{(j)} . For simplicity, we may abbreviate \mathcal{L}(X_{L}(\Theta^{(j)})) as \mathcal{L}(\Theta^{(j)}) . Assuming that \rho_{j}=1/J for all j\in[J] , the j^{th} mode of the posterior predictive will have a large weight only if \mathcal{L}(\Theta^{(j)}) is large compared with the marginal likelihood of all other candidate \Theta values. 3 Multimodality under a discretized Gaussian prior At this stage, it is not obvious whether multimodal distributions on (w,\Theta) map to multimodal distributions in the space of predictions, e.g., the distribution of \widetilde{y} at at given input \widetilde{x}_{1} . 2 2 2 In this paper, we focus only on marginal predictive distributions. It is straightforward, however, to characterize the joint distribution of predictions at any collection of different input values. As ( 5 ) shows, the posterior predictive distribution is the average over predictive distributions obtained by fixing each \Theta^{(j)} and inferring

Read the complete text