Cooperative Bayesian and variance networks disentangle aleatoric and epistemic uncertainties
Jiaxiang Yi, Miguel A. Bessa · 2025
Read Cooperative Bayesian and variance networks disentangle aleatoric and epistemic uncertainties by Jiaxiang Yi, Miguel A. Bessa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02743.txt.
Introduction
Real data carry two kinds of doubt: noise in the data itself, and doubt about the model. Jiaxiang Yi and Miguel Bessa propose training two networks together so each handles one kind. A variance network learns the data noise, while a Bayesian network tracks model doubt. Earlier setups forced one network to do both jobs, with awkward fixes.
Trained as partners, the pair pulls the two doubts apart cleanly. Their method also sharpens the main prediction itself. The authors test it on many data sets, including a time-based one they built where the true noise is known. Their 2025 paper reports the approach is simple to add to many model types and scales well.
Catalog record
| CREATOR | Jiaxiang Yi, Miguel A. Bessa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02743
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cooperative Bayesian and variance networks disentangle aleatoric and epistemic uncertainties Jiaxiang Yi, Miguel A. Bessa Abstract Real-world data contains aleatoric uncertainty – irreducible noise arising from imperfect measurements or from incomplete knowledge about the data generation process. Mean variance estimation (MVE) networks can learn this type of uncertainty but require ad-hoc regularization strategies to avoid overfitting and are unable to predict epistemic uncertainty (model uncertainty). Conversely, Bayesian neural networks predict epistemic uncertainty but are notoriously difficult to train due to the approximate nature of Bayesian inference. We propose to cooperatively train a variance network with a Bayesian neural network and demonstrate that the resulting model disentangles aleatoric and epistemic uncertainties while improving the mean estimation. We demonstrate the effectiveness and scalability of this method across a diverse range of datasets, including a time-dependent heteroscedastic regression dataset we created where the aleatoric uncertainty is known. The proposed method is straightforward to implement, robust, and adaptable to various model architectures. Figure 1: Illustration of the proposed cooperative training of a mean network, a variance network, and a Bayesian neural network for disentangling aleatoric and epistemic uncertainties. The top left figure shows the unseen ground truth mean (thick dashed gray line) and aleatoric uncertainty (credible interval within the thin dashed gray lines), as well as the respective data for training (magenta crosses). The method starts by training the mean network to only estimate the mean (green solid line in Step 1). Then, without updating the mean estimate, a variance network is trained to only predict aleatoric uncertainty (orange credible interval in Step 2). Subsequently, considering this aleatoric uncertainty estimation, a Bayesian neural network is trained to obtain an updated mean and corresponding epistemic uncertainty (solid blue line for the new mean, and shaded blue credible interval for the epistemic uncertainty in Step 3). If needed, the method can iterate between steps 3 and 2 to improve the disentanglement of uncertainties. Note the disentanglement of uncertainties together with the improvement of the mean estimation away from the data support ( x<0 and x>10 in Step 3). 1 Introduction Non-probabilistic neural network models that only estimate the mean (expected value) tend to be overconfident and vulnerable to adversarial attacks ( Guo et al. 2017 ; Guo et al. 2019 ) . Quantifying aleatoric (or data) uncertainty alleviates these issues by characterizing and handling noise ( Skafte et al. 2019 ; Seitzer et al. 2022 ) . Estimating epistemic (or model) uncertainty enables active learning and risk-sensitive decision-making ( Kendall & Gal 2017 ; Depeweg et al. 2018 ) . Consequently, except in cases with negligible or constant (homoscedastic) data noise, simultaneously predicting aleatoric and epistemic uncertainties is essential for a wide range of safety-critical applications ( Hüllermeier & Waegeman 2021 ) . In such cases, an outcome with good mean performance but large aleatoric uncertainty may be unacceptable. Therefore, the principle of reducing epistemic uncertainty behind active learning or decision-making needs to be balanced by the respective prediction of aleatoric uncertainty. This is particularly important when the aleatoric uncertainty is heteroscedastic (input-dependent) due to imperfect measurements, environmental variability, and other factors ( Kiureghian & Ditlevsen 2009 ) . Summary of contributions. We propose a cooperative learning strategy for uncertainty disentanglement based on sequential training of (1) a mean network, (2) a variance network, and (3) a probabilistic neural network. Figure 1 illustrates the method for one-dimensional heteroscedastic regression, briefly describing it at the figure caption. 2 Related work 2.1 Aleatoric uncertainty Aleatoric uncertainty or data noise can be estimated by parametric or nonparametric models. The latter do not explicitly define the likelihood function and instead focus on learning to sample from the data distribution ( Mohamed & Lakshminarayanan 2016 ) . Their ability to estimate nontrivial aleatoric uncertainty distributions comes at the cost of training difficulties and sampling inefficiencies ( Sensoy et al. 2020 ; Harakeh et al. 2023 ) . Therefore, parametric models are more common. They assume a parameterized observation distribution, usually a Gaussian as in Mean Variance Estimation (MVE) networks ( Nix & Weigend 1994 ) , and learn the corresponding parameters by minimizing the Negative Log-Likelihood (NLL) loss with associated regularization. Similar parametric models have been developed replacing the Gaussian distribution with other distributions ( Meyer & Thakurdesai 2020 ) . However, training these models can be challenging. MVE networks have been observed to lead to good mean but overconfident variance estimations in regions within the data support, and exhibit generalization issues outside these regions ( Skafte et al. 2019 ) . As analyzed by different authors ( Skafte et al. 2019 ; Seitzer et al. 2022 ; Immer et al. 2023 ; Sluijterman et al. 2024 ) , a strong dependence of the gradients on the predictive variance causes most of these issues by creating imbalances in the loss optimization. Different solutions have been proposed to improve the parametric estimation of aleatoric uncertainty, including a Bayesian treatment of variance ( Stirn & Knowles 2020 ) , calibration by distribution matching via maximum mean discrepancy loss ( Cui et al. 2020 ) , or modifying the loss function to include an additional hyperparameter to balance it ( Seitzer et al. 2022 ) . However, a recent study ( Sluijterman et al. 2024 ) demonstrated that training an MVE network that simultaneously predicts mean and variance leads to significantly worse predictions for both estimations, even when considering the above-mentioned modified losses. Interestingly, the same study shows that separately training the mean and variance networks led to equivalent or better predictions without modifying the loss. The same conclusion has been reached by other authors ( Skafte et al. 2019 ) , and verified by us in the work herein (see also Section A.1 ). Nevertheless, we note that parametric deterministic models capable of estimating mean and aleatoric uncertainty without estimating epistemic uncertainty are not sufficient. This is also visible in Figure 1 (Step 2), as the model has an overconfident mean outside the data support (see x>10 ). Furthermore, the inability to estimate epistemic uncertainty is problematic in itself, as discussed in the Section 1 . This motivates the need to consider methods that are capable of predicting both aleatoric and epistemic uncertainties while leading to mean predictions that are not overconfident outside the data support (as obtained by our method in Step 3 of Figure 1 ). 2.2 Epistemic uncertainty Epistemic or model uncertainty is usually estimated by probabilistic models that impose a prior distribution on the model parameters ( Abdar et al. 2021 ) . Instead of finding point estimates as in deterministic models, they predict a posterior predictive distribution (PPD). Unfortunately, accurate and computationally tractable determination of the PPD is challenging in most cases, except for a few models like Gaussian processes where inference can be done exactly under strict assumptions and with limited data scalability ( Rasmussen & Williams 2005 ) . Bayesian neural networks (BNNs) are more scalable, but the accuracy and scalability are strongly dependent on the type of Bayesian inference. Bayesian inference is commonly done by Markov Chain Monte Carlo (MCMC) sampling methods ( Neal 1995 ; Welling & Teh 2011 ; Li et al. 2016 ) or Variational Inference (VI) methods that are faster to