Accurate and Reliable Uncertainty Estimates for Deterministic Predictions: Extensions to Under and Overpredictions

Rileigh Bandy, Enrico Camporeale, Andong Hu, Thomas Berger, Rebecca Morrison · 2026

Read Accurate and Reliable Uncertainty Estimates for Deterministic Predictions: Extensions to Under and Overpredictions by Rileigh Bandy, Enrico Camporeale, Andong Hu, Thomas Berger, Rebecca Morrison (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08755.txt.

Introduction

Bandy and four colleagues extend the ACCRUE framework to learn input based, non Gaussian uncertainty for fixed predictions, including under and overpredictions. Sampling based methods prove costly for real time use, and many uncertainty forms ignore input effects or assume Gaussian shapes that miss asymmetry and heavy tails. Their neural network learns two piece Gaussian and asymmetric Laplace forms with a loss balancing accuracy and reliability. It improves probabilistic forecasts on synthetic and real tests while modeling skewed errors.

Catalog record

CREATORRileigh Bandy, Enrico Camporeale, Andong Hu, Thomas Berger, Rebecca Morrison
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39892
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accurate and Reliable Uncertainty Estimates for Deterministic Predictions: Extensions to Under and Overpredictions Thanks: Submitted to the editors April 8, 2026. Rileigh Bandy, Enrico Camporeale, Andong Hu, Thomas Berger, Rebecca Morrison Abstract Computational models support high-stakes decisions across engineering and science, and practitioners increasingly seek probabilistic predictions to quantify uncertainty in such models. Existing approaches generate predictions either by sampling input parameter distributions or by augmenting deterministic outputs with uncertainty representations, including distribution-free and distributional methods. However, sampling-based methods are often computationally prohibitive for real-time applications, and many existing uncertainty representations either ignore input dependence or rely on restrictive Gaussian assumptions that fail to capture asymmetry and heavy-tailed behavior. Therefore, we extend the ACCurate and Reliable Uncertainty Estimate (ACCRUE) framework to learn input-dependent, non-Gaussian uncertainty distributions, specifically two-piece Gaussian and asymmetric Laplace forms, using a neural network trained with a loss function that balances predictive accuracy and reliability. Through synthetic and real-world experiments, we show that the proposed approach captures an input-dependent uncertainty structure and improves probabilistic forecasts relative to existing methods, while maintaining flexibility to model skewed and non-Gaussian errors. keywords input-dependent uncertainty, probabilistic prediction, accuracy–reliability tradeoff, skewed distributions Funding. This work was funded by NASA under Grant No. 80NSSC20K1580 (Ensemble Learning for Accurate and Reliable Uncertainty Quantification). † † runningheads: ACCRUE for Under and Overpredictions / R. Bandy, E. Camporeale, A. Hu, T. Berger, and R. Morrison, MSC 65C20, 62F25, 68T07, 62P12 1 Introduction Computational models are often treated as black-boxes, where we know the inputs and outputs, but we do not have access to the precise mapping between the two. For these black-box models, outputs are usually deterministic, providing a single-point prediction. However, decision-making requires uncertainty quantification (UQ), motivating the development of a-posteriori methods that augment deterministic model outputs with probabilistic predictions. Classical approaches propagate uncertainty through input parameter distributions (e.g., ensemble or Bayesian sampling [ 10 , 15 , 19 ] ). More generally, Bayesian frameworks have been developed to model both input and model-form uncertainty [ 17 , 18 ] , including flexible nonparametric approaches that capture complex features such as skewness and multimodality [ 26 ] . Recent developments extend these ideas to neural network models, where uncertainty is represented through posterior distributions using structured priors and advanced sampling strategies [ 4 ] . However, these approaches typically require posterior inference and repeated simulation, making them computationally prohibitive for expensive or high-dimensional models. To address these challenges, deterministic outputs can instead be augmented with probabilistic uncertainty representations. These approaches broadly fall into distribution-free and distributional categories. Distribution-free approaches, such as conformal prediction (CP), construct prediction regions around model outputs with minimal assumptions [ 22 ] . Distribution-based approaches learn predictive distributions, either nonparametrically—via conditional cumulative distribution functions (CDFs), quantile estimation [ 14 ] , or regression-based methods such as EasyUQ [ 25 ] —or parametrically. While distribution-free and nonparametric methods offer flexibility, they can be difficult to interpret or deploy in structured settings. Parametric distribution-based approaches address these limitations by learning input-dependent uncertainty distributions; in particular, the ACCurate and Reliable Uncertainty Estimate (ACCRUE) framework [ 2 , 3 ] learns a mapping from inputs to the parameters of a predictive distribution using a loss that balances accuracy and reliability. Existing ACCRUE formulations assume Gaussian uncertainty, which limits their ability to capture skewness and heavy tails commonly observed in practice. In this work, we extend the ACCRUE framework to non-Gaussian uncertainty distributions, which allows for asymmetric and heavy-tailed predictive models. Specifically, we introduce two-piece Gaussian and asymmetric Laplace distributions within the ACCRUE loss framework in order to represent systematic biases and non-symmetric error structures. Our approach learns a catch-all uncertainty representation for observation–prediction pairs as a function of model inputs using a neural network parameterization. The resulting probabilistic predictions retain the interpretability and input dependence of ACCRUE while significantly increasing distributional flexibility. The rest of the paper is organized as follows. In Section 2 , we review ACCRUE with Gaussian distributed uncertainties. In Section 3 , we detail the methodology extending ACCRUE to skewed uncertainty distributions, specifically two-piece Gaussian and asymmetric Laplace distributions. We present synthetic results in Section 4 , apply ACCRUE to weather forecasting in Section 5 , and give concluding remarks in Section 6 . 2 Background: ACCRUE with Gaussian Distributed Uncertainties In general, uncertainty in model predictions and observations are not uniformly distributed but instead functions of the inputs. Camporeale et al. [ 3 , 2 ] developed a framework for adding a Gaussian uncertainty distribution with unique variance to each prediction. The uncertainty distribution is centered at the prediction m , its variance is dependent on inputs \bm{X} , and the error between an observation y and the model prediction is \varepsilon^{o}=y-m . Therefore, the uncertainty distribution is defined as \mathcal{N}\left(m,\sigma(\bm{X})^{2}\right), (1) and variance is learned. Specifically, variance is the output of a neural network (NN), where the inputs of the NN are the model inputs that generate the predictions. The functional relationship between the inputs and outputs is learned by minimizing the ACCRUE loss function. ACCRUE balances accuracy and reliability, and it is defined as \text{ACCRUE}=\beta\cdot\overline{\text{CRPS}}+(1-\beta)\cdot\text{RS}, (2) where the mean continuous rank probability score ( \overline{\text{CRPS}} ) measures the average accuracy of the learned uncertainty distribution for all observation-prediction pairs. Reliability is measured by the reliability score ( RS ), which captures the mismatch between the aggregate empirical CDF and the aggregate expected CDF of the uncertainty for all observation-prediction pairs. Since accuracy and reliability tend to be competing objectives, ACCRUE balances them. The choice of the weighting parameter \beta introduces a tradeoff between competing objectives and can be interpreted as a hyperparameter tuning or bilevel optimization problem, where model parameters and calibration criteria are jointly balanced [ 16 ] . The accuracy term in ACCRUE for N observation-prediction pairs with error \bm{\varepsilon}^{o}=\left(\varepsilon^{o}_{1},\varepsilon^{o}_{2},\ldots,\varepsilon^{o}_{N}\right) is defined by \overline{\text{CRPS}}=\frac{1}{N}\sum_{k=1}^{N}\int_{-\infty}^{\infty}\left[P_{k}(\varepsilon)-H(\varepsilon-\varepsilon^{o}_{k})\right]^{2}d\varepsilon, (3) where H(x)=\begin{cases}0,\quad x<0\\ 1,\quad x\geq 0\end{cases} (4) is the Heaviside step function that jumps from zero to one at the point where \varepsilon equals the error \varepsilon^{o}_{k} , and P_{k}(\varepsilon) is the CDF of the error between observation and prediction. Since we assume the uncertainty is normally distributed, P_{k}(\varepsilon)=\frac{1}{2}\left[1+\text{erf}\left(\frac{\varepsilon}{\sqrt{2}\sigma(\bm{X}_{k})}\right)\right], (5) where \bm{X}_{k} are the inputs associated with the error \varepsilon^{o}_{k} . Substituting Equation 5 into Equation 3 yields \overline{\text{CRPS}}=\frac{1}{N}\sum_{k=1}^{N}\sigma(\bm{X}_{k})\left[\frac{\varepsilon^{o}_{k}}{\sigma(\bm{X}_{k})}\text{erf}\left(\frac{\varepsilon^{o}_{k}}{\sqrt{2}\sigma(\bm{X}_{k})}\right)+\sqrt{\frac{2}{\pi}}\exp\left(-\frac{{\left(\varepsilon^{o}_{k}\right)}^{2}}{2\sigma(\bm{X}_{k})^{2}}\right)-\frac{1}{\sqrt{\pi}}\right]. (6) The reliability term in ACCRUE measures predictions’ statistical consistency with observations, and it is defined by \text{RS}=\int_{-\infty}^{\infty}\left[\Phi(\eta)-C(\eta)\right]^{2}d\eta, (7) where \eta is the standardized error, \Phi(\eta) is the expected sample CDF, and C(\eta) is the empirical CDF. Therefore, C(\eta)=\frac{1}{N}\sum_{i=1}^{N}H\left(\eta-\eta_{i}\right), (8) where \eta_{i}=\frac{\varepsilon^{o}_{i}}{\sqrt{2}\sigma(\bm{X}_{i})} . The standardized errors are assumed to be sorted \{\eta_{1},\eta_{2},\ldots,\eta_{N}\} where (\eta_{i}\leq\eta_{i+1}) . Since we assume the uncertainty is normally distributed, \Phi(\eta)=\frac{1}{2}\left[1+\text{erf}\left(\eta\right)\right]. (9) Substituting Equation 9 into Equation 7 yields \displaystyle\text{RS} \displaystyle=\int_{-\infty}^{\infty}\left[\frac{1}{2}+\frac{1}{2}\text{erf}\left(\eta\right)-\frac{1}{N}\sum_{i=1}^{N}H(\eta-\eta_{i})\right]^{2}d\eta (10) \displaystyle=\sum_{i=1}^{N}\left[\frac{\eta_{i}}{N}\left(\text{erf}(\eta_{i})+1\right)-\frac{\eta_{i}}{N^{2}}\left(2i-1\right)+\frac{\exp(-\eta_{i}^{2})}{\sqrt{\pi}N}\right]-\frac{1}{2}\sqrt{\frac{2}{\pi}}. To weigh the accuracy and reliability terms roughly equally—even if they have different magnitudes—we use the heuristic \beta=\frac{\text{RS}_{\text{min}}}{\overline{\text{CRPS}}_{\text{min}}+\text{RS}_{\text{min}}}. (11) The minimum of \overline{\text{CRPS}} is \overline{\text{CRPS}}_{\text{min}}=\frac{\sqrt{\log 4}}{2N}\sum_{i=1}^{N}\varepsilon_{i}, (12) and the minimum RS is \text{RS}_{\text{min}}=\frac{1}{\sqrt{\pi}N}\sum_{i=1}^{N}\exp\left(-\left[\text{erf}^{-1}\left(\frac{2i-1}{N}-1\right)\right]^{2}\right)-\frac{1}{2}\sqrt{\frac{2}{\pi}}. (13) 3 Extending ACCRUE It is a strong assumption that all uncertainties are normally distributed; this may fail under complex parameter uncertainty or measurement error, and especially in the face of significant model-form error. In such cases, errors may be asymmetric for some inputs. We focus on distributions that can be left-skewed, right-skewed,

Read the complete text