A Mathematical Philosophy of Explanations in Mechanistic Interpretability The Strange Science: Part I.i
Kola Ayonrinde UK AI Security Institute, Louis Jaburi · 2025
Read A Mathematical Philosophy of Explanations in Mechanistic Interpretability The Strange Science: Part I.i by Kola Ayonrinde UK AI Security Institute, Louis Jaburi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00808.txt.
Introduction
Kola Ayonrinde and Louis Jaburi set out a mathematical philosophy of explanation for the study of neural networks, in a 2025 paper. That study, called mechanistic interpretability, tries to explain a network's behavior through its inner causal workings. The pair argue for a guiding hypothesis: networks hold implicit explanations of themselves, which research can extract and make readable.
On that basis they define what a good explanation must be. It should describe the whole model, state what really exists inside it, trace causes, and be open to being proven wrong. They also state a principle of explanatory optimism, the bet that such extraction can succeed, which they treat as a precondition for the field. The paper uses the definitions to separate this work from other ways of interpreting models.
Catalog record
| CREATOR | Kola Ayonrinde UK AI Security Institute, Louis Jaburi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Mathematical Philosophy of Explanations in Mechanistic Interpretability The Strange Science: Part I.i Kola Ayonrinde UK AI Security Institute, Louis Jaburi Abstract Mechanistic Interpretability aims to understand neural networks through causal explanations. We argue for the Explanatory View Hypothesis : that Mechanistic Interpretability research is a principled approach to understanding models because neural networks contain implicit explanations which can be extracted and understood. We hence show that Explanatory Faithfulness, an assessment of how well an explanation fits a model, is well-defined. We propose a definition of Mechanistic Interpretability (MI) as the practice of producing Model-level , Ontic , Causal-Mechanistic , and Falsifiable explanations of neural networks, allowing us to distinguish MI from other interpretability paradigms and detail MI’s inherent limits. We formulate the Principle of Explanatory Optimism , a conjecture which we argue is a necessary precondition for the success of Mechanistic Interpretability. 1 Introduction ML artifacts are strange objects. ML researchers have produced models with a wide range of cognitive capabilities that no human knows how to program a machine to do, from playing Go and poker at a superhuman level ( Schrittwieser et al. 2020 ; Brown & Sandholm 2019 ) to folding proteins ( Jumper et al. 2021 ) , and solving advanced mathematical problems ( Glazer et al. 2024 ) 2 2 2 to deceiving humans in games ( Golechha & Garriga-Alonso 2025 ; Bakhtin et al. 2022 ) . However, we did not design these systems. No human wrote the blueprint for how AI systems ought to perform a given task. Instead, neural networks organically learn to solve problems via gradient descent, given large quantities of data. Neural networks aren’t built; they’re grown. Because we don’t design neural networks, ML researchers typically do not know how their models perform a given task. Additionally, neural networks often solve problems in unintuitive ways, relying on concepts that are not obvious to humans ( Widdicombe et al. 2018 ; Hosseini et al. 2018 ; Goodfellow et al. 2014 ; Ilyas et al. 2019 ) . This situation of relative ignorance about the processes that give rise to a neural network’s capabilities leaves us with a scientific problem analogous to the natural sciences. A physicist might observe some natural dynamical system, like the weather, and seek an explanation allowing them to understand, predict, and possibly even steer the system. Similarly, neural network interpretability (henceforth just interpretability ) is the process of understanding artificial neural networks using the scientific method. In this way, we characterise Interpretability as The Strange Science : Interpretability is the science of understanding artificial neural phenomena, just as the natural sciences seek to understand natural phenomena. Interpretability researchers study formal systems using empirical methods — making observations, generating conjectures, and refuting those conjectures — to understand complex neural systems. Since interpretability is analogous to the natural sciences, a Philosophy of Interpretability should be inspired by our best understanding of the philosophy of science. Recent works like Bereska & Gavves 2024 , Sharkey et al. 2025 , and Geiger et al. 2023 have explored the methods and assumptions of Mechanistic Interpretability (MI). Other works have explored the philosophically relevant components of MI ( Millière & Buckner 2025 ; Harding 2023 ; Kästner & Crook 2024 ) . In this work, we foreground the philosophical role of explanation in Mechanistic Interpretability specifically, and how this differs from previous interpretability paradigms. 3 3 3 Lipton 2018 ; Gilpin et al. 2018 ; Fleisher 2022 ; Doshi-Velez & Kim 2017 ; Leavitt & Morcos 2020 ; Erasmus et al. 2021 provide an overview of classical (pre-Mechanistic) Interpretability. In particular, inspired by the Information-Theoretic perspective, we can understand explanations in terms of their compressive power and ability to communicate understanding which generalises . Understanding neural networks can provide affordances for interventions which are important for AI Safety, AI Ethics, and AI Cognitive Science ( Bengio et al. 2025 ; Anwar et al. 2024 ; Chalmers 2025 ; Olah et al. 2020 ; Amodei 2025 ) . Such understanding also allows us to improve the performance of neural networks and debug their failings ( Lindsay & Bau 2023 ; Sharkey et al. 2025 ; Amodei 2025 ) . Contributions. Our contributions are as follows: • Firstly, we show that producing compressive explanations frames a potential solution to the Interpretability Problem. We hence define explanatorily faithfulness as the goal of Mechanistic Interpretability. • Secondly, we provide a technical definition of MI and leverage this definition to highlight both the possibilities and limitations of MI. • Thirdly, we formulate the Principle of Explanatory Optimism , a conjecture at the heart of MI which states that the algorithmic structure of generalising neural networks is human-understandable. We show that without the Principle of Explanatory Optimism the project of MI is intractable. Series Structure This paper is the first in a series titled The Strange Science of Mechanistic Interpretability , concerning the Philosophy of Mechanistic Interpretability. See later papers in this series for an evaluation of methods in MI through the lens of Explanatory Virtues ( Ayonrinde & Jaburi 2025 ) (Part I.ii). See also Ayonrinde 2025 (Part I.iii) which proposes methods to empower humans by teaching humans Machine Concepts. 2 The Strange Science Is Interpretability a natural science or a formal science? Natural sciences like physics, biology, and earth sciences seek to explain physical phenomena, creating hypotheses and running experiments. Formal sciences like algebra, decision theory, linguistics, and music theory are concerned with abstract systems and structures, using deductive methods to construct proofs from axioms. The defining strange property of neural network interpretability is that it is a natural science whose objects of study are both artificial (constructed by humans rather than naturally occurring) and formal (inherently abstract rather than physical systems). Neural networks are mathematical objects; they are deterministic functions from the input embedding space to the output embedding space. Yet, the strange science of interpretability is not a formal science. Interpretability researchers are not primarily interested in proving theorems about neural networks, but in understanding the empirical properties of neural networks. They seek to understand “why” questions about generalisation: why does the network generalise in this specific way? or what are the reusable representations that a neural network is using? and so on. We a priori know the model’s weights, architecture and formal specification and we can compute the output behaviour corresponding to any given input. Where other sciences might be limited by the precision of their measuring tools or the fidelity of observations, in Interpretability our observations are exactly precise and our experiments perfectly reproducible. Furthermore, we can intervene on the network at any point and any time to arbitrarily high precision. However, despite this formal knowledge and potential for intervention, understanding neural networks remains elusive. Here we find a peculiar reversal of the sciences: in natural science, we pursue mathematical formalism to describe empirically observed phenomena; yet in interpretability, we pursue empirical methods to understand formalism. To make progress in understanding ML models, we approach them as naturalists studying a complex system. We can see neural networks as an exemplar of the pitfalls of reductionism: we know all the material and formal causes of the network, and we understand each individual part;