Principled Input-Output-Conditioned Post-Hoc Uncertainty Estimation for Regression Networks
Lennart Bramlage, Cristóbal Curio · 2025
Read Principled Input-Output-Conditioned Post-Hoc Uncertainty Estimation for Regression Networks by Lennart Bramlage, Cristóbal Curio (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00918.txt.
Introduction
Lennart Bramlage and Cristóbal Curio offer a way to add doubt estimates to trained networks in a paper from 2025. Many fields need a network to say not just a value, but how unsure it is. Adding that feature later is hard when the model's inner parts cannot be changed. Their method trains a small helper model on the same inputs plus the frozen model's outputs.
The pair ground the method in classic statistics and state when it is valid. Tests on toy tasks and standard data sets support their claims. Feeding the helper varied training samples sharply improved spotting of odd new inputs. The method keeps a sound sense of how doubt changes with input.
Catalog record
| CREATOR | Lennart Bramlage, Cristóbal Curio |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00918
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Principled Input-Output-Conditioned Post-Hoc Uncertainty Estimation for Regression Networks Lennart Bramlage, Cristóbal Curio Abstract Uncertainty quantification is critical in safety-sensitive applications but is often omitted from off-the-shelf neural networks due to adverse effects on predictive performance. Retrofitting uncertainty estimates post-hoc typically requires access to model parameters or gradients, limiting feasibility in practice. We propose a theoretically grounded framework for post-hoc uncertainty estimation in regression tasks by fitting an auxiliary model to both original inputs and frozen model outputs. Drawing from principles of maximum likelihood estimation and sequential parameter fitting, we formalize an exact post-hoc optimization objective that recovers the canonical MLE of Gaussian parameters, without requiring sampling or approximation at inference. While prior work has used model outputs to estimate uncertainty, we explicitly characterize the conditions under which this is valid and demonstrate the extent to which structured outputs can support quasi-epistemic inference. We find that using diverse auxiliary data, such as augmented subsets of the original training data, significantly enhances OOD detection and metric performance. Our hypothesis that frozen model outputs contain generalizable latent information about model error and predictive uncertainty is tested and confirmed. Finally, we ensure that our method maintains proper estimation of input-dependent uncertainty without relying exclusively on base model forecasts. These findings are demonstrated in toy problems and adapted to both UCI and depth regression benchmarks. Code: https://github.com/biggzlar/IO-CUE . 1 Introduction Uncertainty quantification (UQ) is critical for deploying machine learning systems in safety-sensitive contexts ( Ayhan and Berens 2018 ; Begoli et al. 2019 ) , yet it remains underutilized in practice due to its negative impact on model performance ( Seitzer et al. 2022 ) . Retrofitting existing models with UQ capabilities often requires costly retraining or access to model parameters or intermediate representations – conditions frequently unmet in real-world scenarios. We propose a scalable, post-hoc framework for UQ based on sequential distribution parameter fitting ( Cawley et al. 2004 ; Yuan and Wahba 2004 ) , which circumvents these limitations. Unlike MCMC ( Neal et al. 2011 ) or Laplace-based methods ( Ritter et al. 2018 ) , our approach does not require expensive test-time sampling or access to model internals. It also only requires a fraction of the data required for training from scratch and does not impact base model performance negatively. To our knowledge, this approach has not been rigorously formalized in prior work. In addition, we examine how frozen model predictions can reveal latent structure indicative of epistemic uncertainty, enabling effective out-of-distribution (OOD) detection and uncertainty estimation. We provide both theoretical justification and empirical evidence across regression and vision-based tasks for these claims. 1.1 Contributions • We introduce a theoretically grounded and Input-Output-Conditioned Uncertainty Estimation (IO-CUE) framework based on sequential disjoint parameter estimation, applicable to frozen base models post-hoc without requiring access to parameters or internal states. • We provide a detailed analysis of how structured model outputs can implicitly encode epistemic uncertainty, enabling post-hoc estimators to approximate OOD sensitivity under black-box constraints. • We propose a principled approach to enhancing post-hoc model characterization through targeted data augmentations, significantly improving uncertainty estimation and OOD detection across both simple and complex regression domains. 2 Related Work Uncertainty quantification. A wide range of methods exist for UQ, but we focus on gradient-based optimization of the negative log-likelihood (NLL) objective for heteroscedastic regression ( Nix and Weigend 1994 ) . Under a Gaussian assumption, this corresponds to learning both the predictive mean and input-dependent variance, where the latter reflects aleatoric uncertainty ( Der Kiureghian and Ditlevsen 2009 ) . Epistemic uncertainty, which arises from model misspecification ( Lahlou et al. 2021 ) or parameter uncertainty ( Blundell et al. 2015 ) , is typically estimated via ensemble disagreement, MC dropout, or other sampling-based approaches ( Kendall and Gal 2017 ; Gal and Ghahramani 2016 ; Lakshminarayanan et al. 2017 ) . More recent methods attempt to jointly estimate both uncertainty types through higher-order distributional objectives ( Sensoy et al. 2018 ; Amini et al. 2020 ; Sale et al. 2023 ; Bramlage et al. 2023 ) , though this has been criticized as ill-posed since epistemic uncertainty, by definition, cannot be learned from finite data alone ( Bengs et al. 2022 ; Meinert et al. 2023 ; Jürgens et al. 2024 ) . Sequential parameter estimation. In classical settings, maximum likelihood estimation (MLE) of heteroscedastic Gaussian models proceeds sequentially: the mean is first estimated via least squares, yielding the MAP estimate under a Gaussian prior, after which the variance can be estimated conditioned on this fixed mean ( Cawley et al. 2004 ; Yuan and Wahba 2004 ) . This separation is statistically sound and exploits the conditional structure of the likelihood. However, modern approaches often optimize the negative log-likelihood (NLL) jointly over mean and variance parameters via gradient descent. This poses two challenges: first, the Gaussian NLL is not jointly convex, only conditionally convex in each parameter, leading to unstable or suboptimal convergence; second, when one of the parameters is far from its MAP estimate, the other is estimated under misspecified conditions, further compounding error. To address these issues, prior work has explored reparameterizing the Gaussian using its natural parameters ( Le et al. 2005 ; Immer et al. 2023 ) , applying regularization to stabilize training ( Seitzer et al. 2022 ) , or using alternating optimization schemes that decouple updates ( Detlefsen et al. 2019 ) . Post-hoc UQ. Given the above challenges, post-hoc UQ offers a practical alternative. Estimating the mean via least-squares yields the MAP estimate, after which variance parameters can be fit more effectively. Several methods build on this by sampling from the posterior over weights using approximate gradient-based updates ( Maddox et al. 2019 ) , MCMC ( Neal et al. 2011 ; Betancourt 2017 ; Chen et al. 2014 ) , Dropout injection ( Ledda et al. 2023 ) or Laplace approximations ( MacKay 1992 ; Ritter et al. 2018 ; Daxberger et al. 2021 ) . However, these require access to model parameters and multiple forward passes through sampled networks at inference-time. In contrast, Upadhyay et al. 2022 propose estimating uncertainty directly from base model predictions, enabling black-box applicability and single-forward pass inference. Their approach motivates our more rigorous formulation of the necessary conditions for proper post-hoc UQ, as well as the so-far unexplored capacity of structured output spaces to encode epistemic model uncertainty. 3 Methodology 3.1 Problem Statement We consider the general regression problem of mapping inputs x\in\mathbb{R}^{D} to outputs y\in\mathbb{R}^{K} , where both spaces may be of arbitrary dimensionality. We denote a base model f:\mathcal{X}\mapsto\mathbb{R}^{K} with parameters \theta and predictions \mu_{\theta}(x)=\hat{y}=f(x;\theta) without the capability to infer uncertainty. Assuming a frozen base model and no access to intermediate representations, we want to train an auxiliary model g:\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R}^{+} with parameters \phi on a probe dataset \mathcal{D}_{probe} to estimate the uncertainty of the base model’s predictions \sigma_{\phi}(x,y)=g(x,f(x);\phi) . We aim to (1) capture data uncertainty in-distribution (ID), and (2) provide relative epistemic uncertainty forecasts for, e.g., OOD detection. 3.2 Preliminaries Sources of uncertainty. For the sake of simplicity, we assume additive Gaussian noise in the following set of definitions. Heteroscedastic aleatoric uncertainty arises from noise in the data and is typically represented as \sigma^{2}(x)=\mathrm{Var}(y|x) , such