Partial Trace-Class Bayesian Neural Networks
Arran Carter, Torben Sell · 2025
Read Partial Trace-Class Bayesian Neural Networks by Arran Carter, Torben Sell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01628.txt.
Introduction
Arran Carter and Torben Sell make Bayesian neural networks cheaper. These networks treat their weights as probability distributions, giving honest uncertainty estimates, but the full Bayesian treatment is often prohibitively slow and memory-hungry. Their partial trace-class designs keep only some parameters Bayesian while retaining most of the benefit.
Three architectures are proposed, all built on trace-class priors, which order a network's parameters by importance and so suggest naturally which ones deserve the full treatment. The resulting networks quantify uncertainty comparably to standard Bayesian networks while using significantly fewer Bayesian parameters, with gains in speed and memory. A simulation study verifies the claimed advantages, and a real-world dataset illustrates how the method performs in practice.
Catalog record
| CREATOR | Arran Carter, Torben Sell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Partial Trace-Class Bayesian Neural Networks Arran Carter, Torben Sell Abstract Bayesian neural networks (BNNs) allow rigorous uncertainty quantification in deep learning, but often come at a prohibitive computational cost. We propose three different innovative architectures of partial trace-class Bayesian neural networks (PaTraC BNNs) that enable uncertainty quantification comparable to standard BNNs but use significantly fewer Bayesian parameters. These PaTraC BNNs have computational and statistical advantages over standard Bayesian neural networks in terms of speed and memory requirements. Our proposed methodology therefore facilitates reliable, robust, and scalable uncertainty quantification in neural networks. The three architectures build on trace-class neural network priors which induce an ordering of the neural network parameters, and are thus a natural choice in our framework. In a numerical simulation study, we verify the claimed benefits, and further illustrate the performance of our proposed methodology on a real-world dataset. 1 Introduction Neural networks are important and powerful tools for the prediction and modelling of complex data. Often, one large concern is in quantifying the uncertainty in the predictions of such a network. Bayesian neural networks (BNNs) can be used to quantify this uncertainty ( Neal 1996 ; Izmailov et al. 2021 ; Fortuin et al. 2021 ; Fortuin 2022 ; Papamarkou et al. 2024 ) , however the uncertainty quantification often comes with a high computational cost. In an attempt to reduce the sometimes prohibitively expensive running of BNNs, we wish to exploit the computational speed and efficiency of training and running a standard neural network, and combine this with a Bayesian model in a way that facilitates meaningful uncertainty quantification. With a similar goal in mind of capturing uncertainty at a lower cost, previously, the idea of only applying Bayesian inference on the last layer of a network has been studied in both the context of Bayesian Optimization Snoek et al. 2015 as well as in our context of Bayesian Neural Networks, Zeng et al. 2018 and Brosse et al. 2020 , whose results suggested there is not much benefit to having any more than a single uncertainty layer within the model. In contrast to these hybrid approaches, which allow layers to be only either fully Bayesian or completely non-Bayesian ( Valentin Jospin et al. 2020 ; Chang 2021 ; Prabhudesai et al. 2023 ) , a partial Bayesian Neural Network (pBNN) approach ( Calvo-Ordoñez et al. 2024 ) , also known as subnetwork inference ( Izmailov et al. 2020 ; Daxberger et al. 2021b ) , allows layers to be split into Bayesian and non-Bayesian parameters. Andrade and Sato 2024 investigate different strategies when selecting Bayesian parameters in the neural network, and show that pBNNs can even improve the predictive power of BNNs. Unlike standard BNNs, trace-class Bayesian neural networks ( Sell and Singh 2023 ) introduce a natural ordering of the prior weights and thus naturally lend themselves to truncation. We will recall the definition and some properties of trace-class BNNs in Section 2.1 , before investigating partial trace-class Bayesian neural networks (PaTraC BNNs), which we introduce in Section 3 . In particular, we look at three different potential structures for such PaTraC BNNs, and evaluate their effectiveness in a numerical study in Section 4 , as well as on the abalone data set ( Nash et al. 1994 ) in Section 5 . A particular focus of this study lies on the PaTraC BNN’s ability to accurately quantify the uncertainty in the target function of interest. Our approach differs from these existing pBNNs in two important ways. First, we employ a trace-class BNN prior ( Sell and Singh 2023 ) which by construction enforces a natural ordering of the network parameters, and which further suggests a node-based procedure to select Bayesian parameters. Second, we select nodes based on their relative importance in the trained network, and turn the associated weights into Bayesian parameters; rather than selecting parameters on their respective values after training or picking full layers to be Bayesian. An exception to this is our out-PaTraC BNN, which is more similar to the construction used by Franssen and Szabó 2022 , where we focus only on a subset of the final layer weights to be Bayesian. Other relevant work related to ours include variational inference for BNNs ( Graves 2011 ; Wu et al. 2018 ) , which can be considered the state-of-the-art in BNN inference due to its low computational cost, although it comes at a relatively high price in terms of posterior approximation ( Foong et al. 2020 ) . Laplace approximations to BNNs ( Daxberger et al. 2021a ) also reduce the computational cost of full BNN inference. Different priors for BNNs are discussed e. g. by Wenzel et al. 2020 ; Noci et al. 2021 ; Fortuin 2022 . 2 Statistical setting Given an input dimension d\in\mbox{$\mathbb{N}$} and output dimension m\in\mbox{$\mathbb{N}$} , we are interested in estimating a function f:\mbox{$\mathbb{R}$}^{d}\rightarrow\mbox{$\mathbb{R}$}^{m} . Suppose there exists a true function f^{\star} and a data generating mechanism which gives rise to the likelihood function \mathcal{L}(f|\mathfrak{D}) , where we denoted \mathfrak{D} the observed data set. For estimation purposes, we can now define a function class \mathcal{F} , from which the classical maximum likelihood estimation approach is to find a function \hat{f}_{\mathrm{MLE}}\in\mathcal{F} which maximises the likelihood, i. e. \hat{f}_{\mathrm{MLE}}:=\argmax_{f\in\mathcal{F}}\mathcal{L}(f|\mathfrak{D}) . Due to the monotonicity of the logarithm, this is, of course, equivalent to minimising the negative log-likelihood, which we call the loss function L(f):=-\log(\mathcal{L}(f|\mathfrak{D})) . The choice of the function class \mathcal{F} is of key importance to the practitioner, this involves considerations such as model interpretability, perceived complexity of the target function to be estimated, computational cost of available inference methods, and flexibility of the model. For the purposes of this work, we are working with a nonparametric function class \mathcal{F} imposed by a choice of a neural network structure. Such function classes have been shown to be capable of estimating a wide range of functions to an arbitrary level of precision and are now ubiquitously used across disciplines ( Härdle 1990 ) . One further major advantage of using neural networks is that inference can be easily done using many existing optimisation libraries. One of the limitations of optimising the neural network weights to estimate the function of interest is that it does not provide understanding of the uncertainty around the estimate, i. e. it could be that another neural network would be similarly capable at estimating the true function of interest. Furthermore, trained neural networks are known to be overconfident even in regions with lower amounts of training data ( Kristiadi et al. 2020 ) , and a desirable property of an estimation procedure would be honest about the uncertainty around the estimated function, which should generally be larger in regions with little training data. One principled approach to tackle these issues is given by the Bayesian paradigm, which yields an entire distribution over possible neural networks, giving more posterior probability to networks which fit the data well and are a priori probable. More precisely, a prior distribution P(f) over all neural networks is defined by placing prior distributions on the weights and biases of the network, which are then weighted by the likelihood to give rise to the posterior distribution , which