Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators
Teodor Chiaburu, Felix Bießmann, Frank Haußer · 2025
Read Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators by Teodor Chiaburu, Felix Bießmann, Frank Haußer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03736.txt.
Introduction
Teodor Chiaburu and co-authors ask how much trust to place in AI explanations in this 2025 paper. Tools that explain a model's choice can wobble when the input or the model shifts a bit. The team built one frame to measure that wobble across many such tools.
They compare two ways to gauge the spread of an explanation: a formula-based estimate and one built from repeated trials. Tests across varied data sets show where the two agree and where they part. Some widely used explainers fail to pass the uncertainty on in a sound way. The authors argue that high-stakes uses, like health or finance, need this check before trust.
Catalog record
| CREATOR | Teodor Chiaburu, Felix Bießmann, Frank Haußer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators Teodor Chiaburu, Felix Bießmann, Frank Haußer Abstract Understanding uncertainty in Explainable AI (XAI) is crucial for building trust and ensuring reliable decision-making in Machine Learning models. This paper introduces a unified framework for quantifying and interpreting Uncertainty in XAI by defining a general explanation function e θ ( x , f ) e_{\theta}(x,f) that captures the propagation of uncertainty from key sources: perturbations in input data and model parameters. By using both analytical and empirical estimates of explanation variance, we provide a systematic means of assessing the impact uncertainty on explanations. We illustrate the approach using a first-order uncertainty propagation as the analytical estimator. In a comprehensive evaluation across heterogeneous datasets, we compare analytical and empirical estimates of uncertainty propagation and evaluate their robustness. Extending previous work on inconsistencies in explanations, our experiments identify XAI methods that do not reliably capture and propagate uncertainty. Our findings underscore the importance of uncertainty-aware explanations in high-stakes applications and offer new insights into the limitations of current XAI methods. The code for the experiments can be found in our repository: https://github.com/TeodorChiaburu/UXAI . 1 Introduction Understanding uncertainty in Explainable AI (UXAI) is essential for assessing the reliability of explanations provided by Machine Learning (ML) models. Extensive research [ 15 , 12 , 11 , 1 ] has shown that many widely used XAI methods produce explanations that are unreliable, non-robust and unfaithful to both the learned predictor and the data. These properties are widely recognized as fundamental desiderata of explanations in the XAI community [ 3 , 9 , 17 ] . The absence of these properties can result in misleading attributions, which, in turn, undermines trust in model explanations. This issue is particularly problematic in tasks that require understanding how a model generalizes from data, such as debugging, feature importance analysis and identifying learned relationships between inputs and outputs. Ensuring reliability in XAI methods is especially critical in domains where human verification is difficult and explanation failures may go unnoticed, such as healthcare, finance or autonomous systems. Consequently, identifying and characterizing failure cases in XAI methods is a necessary step towards developing explanations that remain reliable across different inputs, model configurations and data distributions. Despite these concerns, the degree to which explanations are sensitive to perturbations in the input and model parameters remains rather underexplored. If explanations vary significantly under small changes to the data or the model, this raises concerns about their stability and robustness in real-world applications. To address this gap, we introduce a principled framework for quantifying and analyzing uncertainty in explanations, focusing on how perturbations propagate through explanation functions. The main contributions of our work are: 1) We systematically compare two fundamental approaches to uncertainty quantification in XAI: empirical estimation via Monte Carlo sampling and analytical estimation via first-order uncertainty propagation . As a simplified measure of uncertainty, we compute the total variance of the explanations, given by the trace of their covariance matrices. 2) We experimentally investigate the conditions under which Gaussian perturbations in the input and model parameters lead to an approximately Gaussian distribution of explanations, highlighting the regimes where uncertainty propagates in a predictable manner across different XAI methods. 2 Related Work Previous work on uncertainty in XAI (UXAI) has been focusing on developing explanation methods that inherently incorporate uncertainty estimation [ 5 ] . They are designed to generate multiple explanations for the same input and predictor, allowing for direct uncertainty quantification through variance estimation. In [ 5 ] , they are referred to as stochastic XAI methods . Here, we further differentiate between parametric stochastic (some of the parameters in \theta are random variables such as randomly generated perturbations [ 16 , 14 ] ) and parametric deterministic (the method admits parameters \theta , but they are not random variables [ 25 ] ). Either way, their variability comes from a set of XAI-method-specific parameters \theta . For instance, [ 10 , 8 , 27 , 4 ] compute concept scores via multiple random partitions of a ’non-concept’ dataset. [ 18 ] train an ensemble of UNets on previously generated explanations (from various XAI methods) and use the resulting distribution of explanations at inference time to measure uncertainty in the feature importance estimates. [ 21 , 26 ] use LIME [ 16 ] as a backbone XAI method and estimate a confidence interval around the relevance scores of the features. While these approaches enable direct uncertainty assessment within the explanation process, they are limited to XAI methods that are explicitly designed to be parameterized. This leaves open the question of how uncertainty propagates through non-parametric explanation methods, that provide point estimates of feature importance but do not naturally offer a measure of uncertainty. Some limited research looked into how sensitive explanations are with respect to perturbations in the input or the predictor’s weights. The authors of [ 12 ] investigated the reliability of saliency-based explanation methods by examining their sensitivity to transformations that do not affect model predictions. Specifically, it has been shown that many saliency methods fail to satisfy input invariance , meaning they incorrectly attribute importance when a constant shift is added to the input, even though this transformation has no effect on the model’s output. [ 1 ] examine several methods’ sensitivity to model parameters and the data-generating process. They introduce a randomization-based evaluation framework , demonstrating that certain widely used saliency methods produce similar explanations regardless of whether the model is trained or randomly initialized. In this work, we focus on the propagation of uncertainty from the model input and model parameters to the explanation itself. This perspective is, to the best of our knowledge, largely unexplored in existing literature. By leveraging both empirical and analytical first order approximations, we aim to bridge this gap and provide a structured methodology for measuring UXAI. Our framework offers insights into how sensitive explanations are to perturbations in the input and model parameters, enabling a deeper understanding of the stability and reliability of widely used XAI techniques. Our study primarily investigates uncertainty propagation in XAI by introducing controlled perturbations to inputs and model parameters. This analysis relates to the classical distinction between epistemic and aleatoric uncertainty . Epistemic uncertainty arises from limited knowledge about the model parameters due to finite training data, while aleatoric uncertainty stems from inherent randomness in the data itself. In our framework, model weight perturbations can be viewed as approximating epistemic uncertainty, as they simulate the variability that would result from training different models on the same dataset. Input perturbations, on the other hand, capture aspects of aleatoric uncertainty by modeling variations in the observed data. 3 Methods This section presents the two considered sources of propagating uncertainty over to explanations - input perturbations and model weights perturbations - as well as approaches to quantify these propagations. 3.1 Formalizing Uncertainty Propagation in XAI We are considering various explanation methods e for a number of prediction models f (classification or regression). Similar to the notations in [ 5 ] , let the explainer be a function e_{\theta}:\mathbb{R}^{n\cdot l\cdot r}\rightarrow\mathbb{R}^{m}, which maps an input sample x (be it image, text or tabular data) to an explanation e_{\theta}(x,f) , where: