In Defence of Post-hoc Explainability
Nick Oh · 2024
Read In Defence of Post-hoc Explainability by Nick Oh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17883.txt.
Introduction
Oh defends post-hoc explainability in machine learning in a 2024 position paper. Critics doubt that explanations built after training are sound, or that they count as real knowledge. Oh argues that they can be valid tools for science when they are used with care.
His case rests on mediated understanding and bounded factivity. In his view, insight can come from a structured reading of model behavior, without full mechanistic transparency. Such explanations must own that they are approximate, and they must face strict tests. Cases from biomedical machine learning show how these methods can raise new hypotheses and deepen understanding of phenomena.
Catalog record
| CREATOR | Nick Oh |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In Defence of Post-hoc Explainability Nick Oh Abstract This position paper defends post-hoc explainability methods as legitimate tools for scientific knowledge production in machine learning. Addressing criticism of these methods’ reliability and epistemic status, we develop a philosophical framework grounded in mediated understanding and bounded factivity. We argue that scientific insights can emerge through structured interpretation of model behaviour without requiring complete mechanistic transparency, provided explanations acknowledge their approximative nature and undergo rigorous empirical validation. Through analysis of recent biomedical ML applications, we demonstrate how post-hoc methods, when properly integrated into scientific practice, generate novel hypotheses and advance phenomenal understanding. 1 Introduction Is explainable AI (XAI) in its peril ( Weber et al., 2024 ) ? Perhaps. However, this view has yet to gain mainstream acceptance. Driven by the increasing complexity and black-box nature of machine learning models, the field has witnessed an unprecedented surge in post-hoc explanation methods since 2017 ( Saeed and Omlin, 2023 ; Nagahisarchoghaei et al., 2023 ; Retzlaff et al., 2024 ) , with bibliometric analyses documenting exponential growth in XAI approaches ( Sharma et al., 2024 ) . This momentum manifests in substantial industry adoption exemplified by tools such as Captum ( Kokhlikyan et al., 2020 ) . However, this popularity-driven surge cannot obscure the substantive criticisms that challenge the fundamental foundations of post-hoc explainability methods. Challenging both the theoretical premises and practical reliability of post-hoc explainability methods, these criticisms include: • Fidelity Guarantees : Critics such as Rudin (2019) argue that post-hoc explanation methods cannot provide genuine scientific understanding because they only approximate model behaviour rather than granting direct access to underlying mechanisms. This critique has gained formal computational support: recent complexity research demonstrates that generating explanations that are both interpretable and faithfully replicate model predictions is computationally intractable for most non-trivial models. Specifically, Bassan et al. (2024) and Adolfi et al. (2025) prove that local and global interpretability tasks, as well as discovering concise logical circuits for inner interpretability, are typically NP-hard or worse, precluding strong fidelity guarantees except in highly restricted cases. • Empirical Unreliability : A growing body of empirical investigations ( Hooker et al., 2019 ; Laugel et al., 2019b ; Laugel et al., 2019a ; Slack et al., 2020 ; Adebayo et al., 2022 ; Bilodeau et al., 2024 ; Wei et al., 2024 ) has revealed pervasive limitations across the entire spectrum of post-hoc interpretability methods – spanning local and global approaches, model-agnostic and model-specific techniques alike. For instance, ( Wei et al., 2024 , pp. 7–10) addressed the issue of robustness in post-hoc methods, demonstrating that many popular methods fail to provide stable explanations. Most notably, Bilodeau et al. (2024) demonstrate that widely-adopted feature attribution methods (e.g., SHAP) can provably fail to outperform random guessing when inferring model behaviour in neural networks. • Scientific Methodology Concerns : Scientific domains remain particularly reluctant to accept post-hoc explanations as legitimate epistemic tools, with domain experts questioning their reliability for knowledge generation ( Klein et al., 2024 ) . Ghassemi et al. (2021) warn that post-hoc methods create ‘false hope’ in critical applications by appearing to provide understanding without reliable epistemic foundations. This critique fundamentally questions whether these methods can generate legitimate scientific knowledge rather than merely plausible-sounding narratives that satisfy human cognitive biases. (add Madsen et al. "empty argument") • Epistemological Imperative : Unlike other computational research domains that can rely on predictive accuracy or measurable outcomes for validation, interpretability researchers lack the empirical validation mechanisms that typically legitimate scientific claims – we rarely have the ground truth for benchmark evaluation in on the quality of generated explanations ( Doshi-Velez and Kim, 2017 ; Yang et al., 2019 ) , and the existence of a ground truth explanation is still highly debated ( Bommer et al., 2024 ) . This structural absence is compounded by the field’s methodological fragmentation, where no universal consensus exists on what methodology should be applied in evaluating XAI methods ( Ehsan and Riedl, 2024 ) , leading to repeated cycles where we take faithfulness for granted across different paradigms, later proving inadequate ( Madsen et al., 2024 ) . Yet we argue that post-hoc interpretability methods can provide epistemically justified scientific understanding despite offering imperfect approximations of model behaviour. Rather than demanding complete, factive explanations – which may be unattainable for complex AI systems – we propose that scientific knowledge emerges through (1) structured interpretation that mediates between model behaviour and phenomenal understanding, and (2) approximative explanations that acknowledge their limitations whilst maintaining empirical accountability. This position recognises that scientific progress often proceeds through strategic simplifications, and post-hoc methods, when properly validated, can serve this same epistemic function. The remaining paper is structured as follows: Section 2 examines the concept of interpretability in ML systems, contrasting intrinsic and post-hoc approaches. Through distinguishing between model understanding and phenomenon understanding, we establish the key theoretical relationships and assumptions that underpin our analysis. Section 3 introduces contemporary philosophical discussions relevant to ML interpretability, making these insights accessible to the broader ML community. We examine three interconnected themes: reliability and justifiability, scientific understanding and explanation, and scientific practice in the age of AI. Section 4 presents our integrated philosophical framework to defend post-hoc explainability. Through empirical case study, we also demonstrate how this framework translates theoretical principles into practical scientific discovery. Section 5 addresses and re-examines key criticisms of post-hoc methods. We offer constructive responses to concerns about approximation, fidelity, and confirmation bias. Section 6 concludes by outlining future research directions, identifying critical relationships in need of deeper theoretical examination and empirical validation. Throughout the paper, we minimise philosophical jargon where possible, with key concepts further explained in Appendix C for readers seeking additional clarification. 2 On Interpretability Interpretability in ML systems is fundamentally pluralistic ( Zednik, 2021 ) , encompassing multiple distinct concepts and serving diverse stakeholder needs. As Lipton (2016) and Beisbart and Räz (2022) emphasise, the very notion of interpretability shifts and adapts based on stakeholder expectations, posed questions, and the explanatory virtues deemed most relevant within specific contexts. This plurality in interpretation manifests most clearly in the distinct approaches taken by different stakeholders. Computer scientists typically gravitate towards what we might term model understanding – a focus on comprehending the properties of the model itself, including its input-output mappings, learned representations, and decision boundaries (e.g., seeking to understand model grokking through mechanistic interpretability ( Nanda et al., 2023 ) ). In contrast, domain experts in scientific fields often seek to understand how model outputs inform or align with real-world phenomena – what we might term phenomenon understanding . This distinction between model and phenomenon understanding is particularly salient when examining the scientific applications of ML interpretability in detail. For scientific applications, which are the focus of this position, interpretability encompasses our epistemic capacity to understand and articulate how AI systems generate insights about natural phenomena in ways that advance scientific understanding. This relationship begins with a true function f(X) representing the natural phenomenon of interest, where the model learns an approximation h^{*}(X) within a hypothesis space \mathcal{H} that represents the set of possible functions. This broader conception is particularly important when considering post-hoc methods, whose criticisms often stem from evaluating them solely on