Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

Nick Oh, Helen Jin · 2026

Read Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models by Nick Oh, Helen Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29346.txt.

Introduction

Oh and Jin argue that explaining a scientific machine-learning model does not, by itself, yield insight into the phenomenon behind it. Post-hoc explanation methods are used as if their deliverable were understanding of the phenomenon the model was trained on. Two checks seem to secure that step: reliability, that the model's predictions match outcomes, and faithfulness, that the explanation matches the model.

Neither check, they argue, tests whether the model works the way the phenomenon works. Yet that is what a claim about the phenomenon's structure needs. Reliability compares model with outcomes; faithfulness compares explanation with model. The gap between model and mechanism stays unexamined. The chain can still support candidate hypotheses when outside evidence corroborates them. On its own, though, it cannot back claims about the phenomenon's true structure.

Catalog record

CREATORNick Oh, Helen Jin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models Nick Oh, Helen Jin Abstract Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model’s predictions match the phenomenon’s outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured. 1 Introduction Post-hoc explainable AI (XAI) methods — SHAP, LIME, attention attribution, gradient-based saliency, Layer-wise Relevance Propagation (LRP), TCAV and Concept Relevance Propagation, counterfactual explanations, and related techniques — are routinely deployed to interpret models built for scientific discovery and hypothesis generation. The practice now spans neuroscience ( 34 ; 16 ; 20 ; 13 ) , cognitive psychology ( 3 ; 43 ; 24 ) , physics ( 41 ; 9 ) , biochemistry ( 30 ; 28 ; 40 ) , economics ( 10 ; 45 ; 17 ; 22 ) , medicine ( 2 ; 32 ; 23 ) , the earth and environmental sciences ( 36 ; 18 ; 42 ) , and sociology ( 6 ) . In each case the model functions not as a black-box predictor whose only credential is empirical accuracy, but as a putative source of insight into the phenomenon it has been trained on. The intended deliverable is understanding with the model, rather than understanding of the model ( 37 ) . The transition from understanding of the model to understanding about the world is taken to be secured by two materials whose standards are themselves the subject of extensive methodological work ( 21 ; 26 ; 25 ; 35 ; 1 ; 7 ; 44 ) . The model must be reliable on the phenomenon, to a degree the relevant scientific community judges adequate; and the explanation must be faithful to the model, to a degree the methodological literature judges adequate. This paper asks a question that neither refinement addresses. Does a reliable model paired with a faithful explanation of it deliver, by composition alone, a justified claim about the structure of the phenomenon? We argue that it does not, and that the shortfall is not closed by raising either standard further. The contribution of the paper is one of clarification. The practice of explaining reliable models invites a move from a faithful account of what the model does to a claim about the structure of the world, and it is this inference we examine. It composes two assessments of different epistemic kinds, a description of what the model computes and an endorsement of the model as a tracker, into a third that is neither, a claim about the structure of the phenomenon. The composition does not go through, and its failure is structural, holding at the limit of perfect reliability and faithfulness as firmly as in any actual case. 2 The chain We begin with a formalism familiar to the ML community. Let f\colon\mathcal{X}\to\mathcal{Y} denote the true function representing the natural phenomenon of interest. A predictive model is trained to approximate f by selecting a hypothesis h^{*}\in\mathcal{H} , where \mathcal{H} is a hypothesis space such as the set of functions representable by a deep network of given architecture. We denote h^{*} a ‘reliable enough’ model 1 1 1 We adopt the locutions ‘reliable enough’ and ‘faithful enough’ in the spirit of Elgin’s ‘true enough’ ( 14 ) , on which a representation’s acceptability is not absolute but depends on whether the larger body of discourse it figures in, such as argument, explanation, or theory. . For an opaque model h^{*} , a post-hoc explanation method E produces an explanatory representation g^{*}=E(h^{*})\in\mathcal{P} , drawn from a space of possible explanations \mathcal{P} such as the SHAP-value space, the saliency-map space, or the counterfactual space. We denote g^{*} a ‘faithful enough’ explanation in the same contextual sense. The relationship under analysis is the relationship between f and the composition that connects it to g^{*} . A model h^{*} approximates f through training, and an explanation g^{*} approximates h^{*} through some XAI method. We make the composition explicit by writing f\overset{\phi}{\longleftarrow}h^{*}\overset{E}{\longleftarrow}g^{*} with arrows running in the direction of approximation: g^{*} approximates h^{*} , which approximates f . We follow Sullivan in calling \phi the model–world link ( 39 ) , since it captures whether h^{*} tracks f . For the second link we depart from Sullivan’s terminology. Sullivan calls the relation between h^{*} and g^{*} a model–model link, but the word model then covers two different roles. h^{*} is a model of the phenomenon; g^{*} is a model of h^{*} , a representation of a representation, one remove further from the world. Naming both relata ‘model’ puts them on a level and so invites the reading that g^{*} , like h^{*} , is a candidate representation of the phenomenon. That reading is precisely what this paper denies. We adopt instead 15 ’s term and call E the model-of-model link, with g^{*} a model of h^{*} , where the ‘of’ keeps the asymmetry in view. 3 Two verdicts of different epistemic kinds The chain f\leftarrow h^{*}\leftarrow g^{*} is assembled from two per-link assessments. The argument of this paper is that the two are verdicts of different epistemic kinds, that what the scientist wants is a verdict of a third kind, and that no composition of the first two yields the third. This section sets out the two verdicts and the third thing wanted. \lx@sectionsign 4 shows the composition cannot supply it. The verdict on g^{*} as a model of h^{*} is descriptive . A faithful explanation correctly reports which features the model relies on, or more generally what the model computes, in whatever idiom the explanation employs, feature attribution, counterfactual, or concept activation. This is a fact about the computational artefact in \mathcal{H} , recovered by the map E:\mathcal{H}\to\mathcal{P} . It answers a question about what the model does, and answering it well is what the discipline means by faithfulness. Where faithfulness is checked by comparing recovered features against external annotations, the check confirms the explanation only on the assumption that h^{*} already tracks f , and so tests the descriptive verdict indirectly through the reliability verdict rather than appealing to the world directly. The verdict on h^{*} as a tracker of f is partly justificatory . When h^{*} predicts f accurately across the population the surrounding practice has validated against, the endorsement does more than record a regularity in the outputs; it licenses using them as evidence about f . A prediction then gives the scientist some reason, evidential and not merely behavioural, to believe the corresponding claim about the phenomenon. This is the standing that reliabilist accounts of model-based belief pick up ( 12 ) , and that Sullivan’s appeal to suitable model-target linking secures ( 37 ) .

Read the complete text