Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding
Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp · 2026
Read Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding by Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25251.txt.
Introduction
Baer and co-authors test whether truer AI explanations truly help people grasp a model. Faithfulness scores are rarely checked against human outcomes. Their study gave 200 people explanations at four correctness levels, from full to 55 percent. The task was a fake time series sort where gut feel could not help. People predicted the AI's calls from feature-style explanations. Understanding fell at 70 percent and below, but 85 and 100 percent looked alike. Even perfect explanations left many users lost, which warns against trusting scores alone.
Catalog record
| CREATOR | Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47032 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25251
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp Abstract Explainable AI (XAI) methods are commonly evaluated with functional metrics such as correctness, which computationally estimate how accurately an explanation reflects the model’s reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested experimentally with controlled levels. We conducted a user study ( N = 200 N=200 ) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a time series classification task where participants could not rely on domain knowledge or visual intuition and instead predicted the AI’s decisions based on explanations (forward simulation). Correctness affected understanding, but not at every level: performance dropped at 70% and 55% correctness relative to fully correct explanations, while further degradation below 70% produced no additional loss. Rather than shifting performance uniformly, lower correctness decreased the proportion of participants who learned the decision pattern. At the same time, even fully correct explanations did not guarantee understanding, as only a subset of participants achieved high accuracy. Exploratory analyses showed that self-reported ratings correlated with demonstrated performance only when explanations were fully correct and participants had learned the pattern. These findings show that not all differences in functional correctness translate to differences in human understanding, underscoring the need to validate functional metrics against human outcomes. † † shortheadings: Does Explanation Correctness Matter? / Baer et al. † † firstpage: 1 keywords Explainable AI, Feature attributions, Explanation correctness, Faithfulness, User study, Forward simulation, Time series classification 1 Introduction Explainable AI (XAI) methods generate explanations for AI model decisions to support human understanding and oversight. The field has grown rapidly [ 38 , 43 ] , but evaluating these explanations remains an open challenge [ 35 ] . Two broad approaches exist [ 20 ] : functional evaluation scores explanations against computational criteria such as robustness or correctness, while user-centered evaluation measures human outcomes such as understanding or human-AI team performance [ 46 ] . Most work in the machine learning and computer science communities relies on functional evaluation [ 38 ] , with user studies being comparatively rare [ 43 , 46 ] , partly because they are costly and time-consuming. This prevalence of functional evaluation implies that the field optimizes XAI methods for computational metrics that may or may not relate to human outcomes, implicitly assuming that functional metrics serve as meaningful proxies for human understanding. One widely used functional property is correctness (also called faithfulness or fidelity), which estimates how accurately a post-hoc explanation reflects the model’s reasoning. The assumption is that explanations scoring poorly on correctness will mislead users or fail to support understanding. Whether higher correctness reliably improves human understanding remains unclear. However, functional correctness may vary in ways that do not affect human understanding. For example, differences may be hard to notice, or they may occur in parts of the explanation that users do not attend to or do not treat as relevant for their decisions. In addition, people may be able to learn useful decision rules from imperfect explanations as long as the explanation preserves enough useful information, with effects emerging only once inaccuracies exceed a tolerance threshold. If either mechanism holds, optimizing for small differences in functional correctness may yield limited gains in human outcomes. A small number of studies have investigated how explanation properties relate to human outcomes, but none provide a controlled experimental test of how correctness at varying levels affects understanding. Existing work has treated correctness as binary [ 13 , 37 , 56 ] , relied on correlational rather than experimental designs [ 18 , 30 , 39 ] , and all used image tasks where human intuition can compensate for poor explanations [ 13 , 18 , 30 , 37 , 39 , 56 ] . Therefore, while linking functional metrics to human outcomes has been identified as a priority for advancing XAI [ 43 ] , the question remains open. To address this question, we conducted a between-participants user study ( N=200 ) that tested whether explanation correctness, treated as a continuous rather than binary property, affected human understanding of AI decision-making. We designed a synthetic classification task with a simulated AI classifier, where the ground truth was defined by the data-generating process rather than a trained model. This allowed us to manipulate correctness at exact levels (100%, 85%, 70%, and 55% temporal overlap with the ground truth) and examine whether understanding degraded gradually or remained stable until correctness fell below a certain level. We used time series rather than images so that participants could not rely on visual intuitions or domain knowledge that might otherwise confound results [ 16 ] . Together, these design choices provide a first experimental test of whether, and to what degree, functional correctness affects human understanding of AI explanations. We found that correctness affected understanding, but not proportionally: degraded explanations reduced the proportion of participants who learned the decision pattern, suggesting a threshold effect rather than a gradual decline. At the same time, correctness alone was not sufficient for understanding, as many participants failed to learn even from fully correct explanations. Exploratory analyses indicated that self-reported ratings tracked demonstrated understanding only under narrow conditions, raising questions about when subjective measures are informative. The remainder of this paper is organized as follows. We first review related work on feature attribution evaluation, correctness metrics, and existing evidence linking correctness to human outcomes. We then describe the study design, including the synthetic time series task, the correctness manipulation, and the forward simulation measure. Next, we present results from the planned hypothesis tests alongside exploratory analyses of learning curves, self-reported ratings, and the relationship between subjective and objective measures, followed by a general discussion of the findings. We close with limitations, conclusions, and directions for future work. 2 Related Work 2.1 Feature attributions and evaluation Post-hoc XAI methods explain AI models after training. Among the various types of post-hoc explanations, including rules, concept-based explanations, and counterfactuals [ 21 , 23 ] , feature attributions are among the most common. They assign an importance score to each input feature, such as a pixel, time point, or word, indicating how much it contributed to the model’s prediction. Well-known examples include LIME [ 45 ] and SHAP [ 36 ] , alongside other gradient-based [ 5 , 52 , 54 , 55 , 57 ] and perturbation-based methods [ 22 , 61 ] . These attributions are typically visualized as heatmaps, often called saliency maps in the image domain, that highlight which parts of the input drive the prediction. The implicit promise is that showing users what the model attends to will help them understand how it decides. Evaluating explanations is difficult because, unlike prediction tasks, there is no ground truth for what the correct explanation should be. 20 distinguish three evaluation approaches: application-grounded evaluation tests explanations with domain experts on real tasks [ 14 , 15 , e.g.,] ; human-grounded evaluation uses simplified proxy tasks to measure properties like understanding, trust, or human-AI collaboration performance [ 46 , for an overview, see] ; and functionally-grounded evaluation bypasses humans entirely and scores explanations with