Explanation Beyond Intuition: A Testable Criterion for Inherent Explainability
Mike Merry, Pat Riddle, Jim Warren · 2025
Read Explanation Beyond Intuition: A Testable Criterion for Inherent Explainability by Mike Merry, Pat Riddle, Jim Warren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17316.txt.
Introduction
Michael Merry, Pat Riddle, and Jim Warren propose a test for whether an AI model is truly explainable by design. The field calls that gold standard inherent explainability, yet no shared test exists; most work leans on gut feeling. Their test turns gut feeling into something a regulator could check.
A model passes if it can be split, using graph theory, into small parts that each carry a clear local explanation. Those local notes must then join back into one global account, in a hypothesis-and-evidence form others can verify. The team works the test in full on PREDICT, a heart risk model used in New Zealand clinics. A large tangled model, they argue, would fail where a sparse network passes.
Catalog record
| CREATOR | Mike Merry, Pat Riddle, Jim Warren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explanation Beyond Intuition: A Testable Criterion for Inherent Explainability Mike Merry, Pat Riddle, Jim Warren Abstract Inherent explainability is the gold standard in Explainable Artificial Intelligence (XAI). However, there is not a consistent definition or test to demonstrate inherent explainability. Work to date either characterises explainability through metrics, or appeals to intuition - ”we know it when we see it”. We propose a globally applicable criterion for inherent explainability. The criterion uses graph theory for representing and decomposing models for structure-local explanation , and recomposing them into global explanations. We form the structure-local explanations as annotations , a verifiable hypothesis-evidence structure that allows for a range of explanatory methods to be used. This criterion matches existing intuitions on inherent explainability, and provides justifications why a large regression model may not be explainable but a sparse neural network could be. We differentiate explainable - a model that allows for explanation - and explained - one that has a verified explanation. Finally, we provide a full explanation of PREDICT - a Cox proportional hazards model of cardiovascular disease risk, which is in active clinical use in New Zealand. It follows that PREDICT is inherently explainable. This work provides structure to formalise other work on explainability, and allows regulators a flexible but rigorous test that can be used in compliance frameworks. 1 Introduction The value of Explainable AI (XAI) in sensitive fields remains high, but there are still evident challenges defining explanation, and formal tests for explainability. For inherent explainability, where whole models—not just localised predictions—are fully explainable, the test remains predominantly intuitive - ”we know it when we see it” 3 . This creates challenges for regulators and practitioners seeking safety and transparency in health, law, and insurance. Without an objective standard, any framework or regulation requiring explainability has a weak foundation. ”We know it when we see it” is not auditable. Here, we describe the mechanics of formalised explainability - functional graph decomposition of models, hypothesis-evidence structures called annotations, and composition of subgraphs with annotations. These mechanics are then used to formalise the test for inherent explainability. We demonstrate that this test aligns with existing claims and intuitions, and we demonstrate its application on a cardiovascular disease risk prediction model in use in clinical care. 2 Background There is little consensus on what explainability in machine learning is and how to evaluate it for benchmarking 3 . It is referred to ambiguously as interpretability, comprehensibility, understandability and explainability 5 . Although there is substantial work characterising and describing ways in which we approach it, even formalising them in taxonomies and similar, there is no definitive standard for inherent explainability 3 ; 1 ; 6 ; 5 ; 17 ; 7 ; 18 ; 9 . The foundation of many techniques rely on a claim of inherent explainability, where some model classes, especially regression models and decision trees, are considered explainable axiomatically 3 . Proxy techniques including LIME 16 and LORE 6 create explanations of local predictions by reducing the complex model to a proxy model using inherently-explainable models. This class of approach naturally relies on the acceptance of the inherent explainability of the proxy models. Other approaches include attention 20 or attribution based methods 19 . These methods provide the relative importance or criticality of differing features - the what of predictions - but they do not address specifically how the methods use these features to produce the result. This can help address some risks, such as identifying models that have been inadvertantly trained on incorrect features 2 , so help reject classes of poor models, but they do not demonstrate how a model is behaving. Much of the motivation for XAI is related to practical uses in sensitive contexts in real-world applications such as healthcare. Human-centric metrics like comprehension and trust have become increasingly important 15 ; 11 , as have user studies. The importance of structures within (especially neural-network type) models, is increasing in prominence in XAI. Rago et al identify substructures within neural networks to explain through argumentation 15 . Olca et al. have explained transformer behaviour regarding substructures called ”circuits” with methods such as ablation or manipulation of the model to determine the importance of the structure 13 . GNNExplainer 21 and similar work uses explanatory subgraphs to create instance-local explanations. These approaches share an intuition that model structure is explanatorily relevant, but lack a unifying framework for when structural decomposition constitutes a complete explanation The work of miller 12 and Le et al 10 have advocated for hypothesis-driven approaches to XAI, where systems provide evidence for and against hypotheses rather than direct recommendations. Despite this work, no formal criterion exists for testing inherent explainability claims—no standard that creators can demonstrate and regulators verify. 3 A Formal Framework for Explanation Here, we build a formalisation of the mechanics needed for testing explainability. We start from a formal definition of explainable that addresses what needs to be demonstrated. We then build the mechanics of explanation in our framework: the atomic syntax of explanations - a hypothesis-evidence structure called annotation; the form in which they are applied to computational models - a computational graph formalisation from automatic differentiation; the construction method to create global explanations from structure-local explanations - composition of subgraphs; and finally the semantics of explanation - the methods by which evidence can be verified. 3.1 Explainable Definition 1 (Explanation) . An explanation of a model M is a representation of M in a language L , for an audience A , and for a purpose P . The triple (A,L,P) constitutes the context of the explanation. Definition 2 (Explainable) . A model is explainable if there exists an explanation where the representation is accurate to the underlying model, the language is understandable by the audience, and the purpose of the explanation is met Merry et al. established that explanation must be understood relative to context (purpose, audience, language), arguing that existing definitions failed by treating explanation as context-independent 11 . 3.2 Verifiable Hypotheses Definition 3 (Verifiability) . A claim about a model is verifiable if there exists a method, accepted by the intended audience as epistemically adequate, to determine whether the claim accurately characterises the model’s behaviour. A central requirement of our framework is that explanatory claims be verifiable - that there exist methods by which claims can be assessed for accuracy. We deliberately adopt a pluralistic stance on verification. Different communities accept different forms of evidence and different standards of proof. This extends Definition 2 ’s position that explanations are context dependent. This definition encompasses: • Mathematical derivation (proof that a claim follows from axioms) • Empirical testing (experimental observation confirming predictions) • Logical inference (deduction from established premises) • Computational verification (algorithmic confirmation of properties) The key constraint is that verification methods must be epistemically acceptable to the intended audience, just as the language must be understandable. This mirrors scientific peer review: different communities apply different standards, yet the process remains sound because standards are established and consistently applied. Definition 4 (Hypothesis-Evidence Structure) . An explanation consists of: • A hypothesis : A claim about what a model component does • Evidence : Verification that the hypothesis accurately characterises the component’s behaviour Explanations in