On weight and variance uncertainty in neural networks for regression tasks
Moein Monemi, Morteza Amini, S. Mahmoud Taheri, Mohammad Arashi · 2025
Read On weight and variance uncertainty in neural networks for regression tasks by Moein Monemi, Morteza Amini, S. Mahmoud Taheri, Mohammad Arashi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04272.txt.
Introduction
Moein Monemi and co-authors study uncertainty in neural networks for regression tasks in a 2025 paper. Earlier work by Blundell and others modeled doubt about network weights. These authors extend that idea by also modeling uncertainty in the variance, treating it as a full distribution rather than a fixed value.
The team argues that modeling variance uncertainty improves how well Bayesian neural networks predict new data. They test the approach on a function approximation example and on the riboflavin genetic dataset. Tests cover dense networks and dropout networks, with two kinds of priors on the weights. Readers get a broad view of how variance uncertainty shapes model performance across different network designs.
Catalog record
| CREATOR | Moein Monemi, Morteza Amini, S. Mahmoud Taheri, Mohammad Arashi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-04272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On weight and variance uncertainty in neural networks for regression tasks Moein Monemi, Morteza Amini, S. Mahmoud Taheri, Mohammad Arashi Abstract We consider the problem of weight uncertainty proposed by [Blundell et al. (2015). Weight uncertainty in neural network. In International conference on machine learning, 1613-1622, PMLR.] in neural networks (NNs) specialized for regression tasks. We further investigate the effect of variance uncertainty in their model. We show that including the variance uncertainty can improve the prediction performance of the Bayesian NN. Variance uncertainty enhances the generalization of the model by considering the posterior distribution over the variance parameter. We examine the generalization ability of the proposed model using a function approximation example and further illustrate it with the riboflavin genetic data set. We explore fully connected dense networks and dropout NNs with Gaussian and spike-and-slab priors, respectively, for the network weights. Keywords: Bayesian neural networks, Variational Bayes, Kullback-Leibler divergence, posterior distribution, Regression task, variance uncertainty. 1 Introduction Bayesian Neural Networks (BNNs) have been introduced and comprehensively discussed by many authors (among others see [ 14 , 2 ] ). BNNs are suitable for modeling uncertainty by considering values of the parameters that might not be learned by the available data. This is achieved by randomizing unknown parameters, such as weights and biases while incorporating prior knowledge. Such an approach naturally regularizes the model and helps prevent overfitting, a common challenge in neural networks (NNs). The BNNs can also be considered as a suitable and more reliable alternative to ensemble learning methods [ 15 ] , including bagging and boosting of the neural networks. The primary objective in the BNNs is to estimate the posterior distribution of the parameters. However, obtaining the exact posterior is often infeasible due to the integrals’ intractability. Thus, it is necessary to use approximation techniques for the posterior density. One common approach is the Markov Chain Monte Carlo (MCMC), including the Metropolis-Hastings algorithm [ 10 ] , which generates samples from the posterior distribution by constructing a Markov chain, typically of first orders. Despite its effectiveness, flexibility, and strong theoretical foundations, MCMC has significant limitations in high-dimensional spaces, including slow convergence and high computational costs. Variational Bayes (VB) has emerged as a faster and more scalable approximation technique. First introduced by [ 11 ] and [ 19 ] , VB seeks to approximate the intractable posterior distribution by identifying a simpler, parametric probability distribution that closely resembles the true posterior. The central idea in VB is to recast the problem of posterior inference as an optimization task. Specifically, VB minimizes the Kullback-Leibler (KL) divergence [ 13 ] between the approximate distribution and the true posterior [ 3 ] . This involves iteratively updating the hyper-parameters of the approximate distribution to improve its fit to the posterior while maintaining computational efficiency [ 1 ] . The VB method has limitations related to the conjugacy of the full conditionals of parameters. In such cases, models that go beyond the conjugacy of the exponential family have been proposed [ 20 ] . Black Box Variational Inference (BBVI) [ 16 ] eliminates the need for conditional conjugacy by introducing a flexible framework for approximating posterior distributions. It leverages stochastic optimization techniques to estimate gradients and employs variance reduction strategies to enhance computational efficiency and stability. Bayes by Backprop (weight uncertainty), introduced by Blundell et al. [ 4 ] , is another approach for performing Bayesian inference in NNs. This method leverages stochastic optimization and employs the re-parametrization trick to enable efficient sampling and gradient-based optimization of posterior distributions, for a NN weights. The model is used for classification and regression tasks, by considering suitable multi-nomial and Gaussian likelihood for the target variable, respectively. Another approach for modeling uncertainty in weights of a NN is considered in [ 7 ] . They modeled the variational approximation of the weights as the multiplication of a fixed hyperparameter (which might be optimized) by a bernoulli random varaible for modeling the droupout mechanism. There are two main differences between the method considered in [ 7 ] and that in [ 4 ] . The first difference is that no statistical distribution is considered for modelling uncertainty in weights around the hyperparameter in the method proposed in [ 7 ] . The second difference is the approach for modeling the dropout mechanishm, since the dropout mechinism was modeled by considering the slab-and-spike prior for the weights of the NN in [ 4 ] . For the regression task, the variance of the Gaussian likelihood is assumed fixed in [ 4 ] and is determined by cross-validation. In this paper, we consider the problem of variance uncertainty in the model proposed by [ 4 ] to investigate its effectiveness on the prediction performance of the BNNs for regression tasks. Throughout a simulation study from a nonlinear function approximation problem, it is shown that the Bayes by Backprop with variance uncertainty performs better than the model with fixed variance. We have also considered the riboflavin data set, which is a genetic high dimensional data set to evaluate the performance of the proposed model, compared to the other methods, by applying PCA-BNN and the dropout-BNN methods. The dropout-BNN is applied by considering the spike-and-slab prior for the weights of the BNN. As the results presented in Section 6 shows, our proposed method outperforms other approaches in terms of MSPE and coverage probabilities for the regression curve, as well as MSPE for the riboflavin dataset. The remainder of the paper is organized as follows. Section 2 reviews the foundations of the BNNs. The VB method for posterior approximation is introduced in Section 3. Section 4 describes the Bayes by Backprop method proposed by [ 4 ] , for the regression task with fixed variance. In Section 5, we propose the variance uncertainty for the BNNs. The experimental results and the evaluation of the proposed model is considered in Section 6. Some concluding remarks are given in Section 7. The codes for this study is available online on github (soon). 2 Bayesian Neural Networks In this section, our goal is to describe BNNs and their challenges in practical applications. Before examining BNNs, we briefly describe NNs from a statistical perspective. A NN aims to approximate the unknown function y=\phi(x) where x and y are the input and the output vectors, respectively. From a statistical point of view, given the sample points x,y , the objective function of NN is the log-likelihood function of weights and biases, W , and extra parameters \rho , of the model, \log L(W,\rho|x,y) which is aimed to be maximized to learn the parameters of the network \widehat{(W,\rho)}_{\text{\rm MLE}}=\arg\max_{(W,\rho)}\log L(W,\rho\mid x,y). The likelihood function L(W,\rho\mid x,y) depends on the specific problem being addressed: • In regression tasks, the likelihood function is typically modeled by a Gaussian density of y with a mean provided by the network given the input data set x and an unknown variance \sigma^{2} , and the negative log-likelihood corresponds to the sum of squared errors (SSE). • In classification problems, the likelihood function is often modeled as a categorical distribution, and the negative log-likelihood corresponds to the cross-entropy loss. In a BNN, a prior distribution p(W,\rho) is considered for the parameters