Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstraction
Li Puyin, Jiyuan Tan, Ahmad Jabbar, Thomas Icard, Atticus Geiger, [6pt] Stanford University · 2026
Read Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstraction by Li Puyin, Jiyuan Tan, Ahmad Jabbar, Thomas Icard, Atticus Geiger, [6pt] Stanford University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02234.txt.
Introduction
Li Puyin and colleagues present a diagnostic method for causal abstraction, a way to test whether a simple causal story explains a neural network's behaviour. Such stories are usually judged by interchange interventions, where swapping an internal part should swap the output as the story predicts. A single global accuracy score hides where the story holds and where it fails.
Their method partitions inputs into well-interpreted and under-interpreted regions by pairwise intervention behaviour. That turns a global verdict into a diagnosis of regions. Comparing the two regions can reveal missing distinctions in the story. The authors also give heuristics for improving an interpretation from that regional structure, a practice they frame as bucketing the good apples.
Catalog record
| CREATOR | Li Puyin, Jiyuan Tan, Ahmad Jabbar, Thomas Icard, Atticus Geiger, [6pt] Stanford University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02234
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstraction Li Puyin, Jiyuan Tan, Ahmad Jabbar, Thomas Icard, Atticus Geiger, [6pt] Stanford University Abstract We present a method for diagnosing interpretation in neural networks by identifying an input subspace where a proposed interpretation is highly faithful. Our method is particularly useful for causal-abstraction-style interpretability, where a high-level causal hypothesis is evaluated by interchange interventions. Rather than treating interchange intervention accuracy as a single global summary, we refine this framework by partitioning the input space into well-interpreted and under-interpreted regions according to pairwise interchange-intervention behavior. This turns causal abstraction from a purely global evaluation into a more diagnostic tool: it not only measures whether an interpretation works, but also reveals where it works, where it fails, and what distinguishes the two cases. This diagnostic view also provides practical heuristics for improving interpretations. By analyzing the structure of the well-interpreted and under-interpreted regions, we can identify missing distinctions in a high-level hypothesis, discover previously unmodeled intermediate variables, and combine complementary partial interpretations into a stronger one. We instantiate this idea as a simple four-step recipe and show that it yields informative error analyses across multiple causal abstraction settings. In a toy logic task, recursively applying the recipe recovers a high-level hypothesis from scratch. More broadly, our results suggest that partitioning the input space is a useful step toward more precise, constructive, and scalable mechanistic interpretability. 1 1 1 We provide the code base for this paper at https://github.com/Paulineli/apple-bucket . 1 Introduction As Language Models (LMs) have scaled in complexity, the mechanistic interpretability community has developed a broad set of tools for studying the representations and algorithms underlying model behavior, including gradient-based attribution ( 31 ) , activation-based localization ( 24 ; 25 ) , and feature decomposition with Sparse Autoencoders (SAEs) ( 3 ; 6 ) . Among these paradigms, causal abstraction provides a particularly rigorous framework: given a task, a high-level causal model and an alignment between high-level variables and internal neural representations, it asks whether the model and the hypothesis agree under counterfactual interventions ( 10 ; 9 ) . This framework has been shown to be suitable for a wide variety of tasks ( 32 ; 1 ; 19 ; 2 ) . In practice, causal abstraction is usually evaluated through interchange interventions , summarized by a single scalar metric, interchange intervention accuracy (IIA) ( 10 ; 9 ) . The IIA score provides a simple metric to measure how well hypotheses align with neural models, but it says little about where a hypothesis is faithful, and intermediate scores are especially hard to interpret ( 22 ; 33 ; 23 ) . Moreover, although methods such as DAS make it easier to find candidate alignments, the overall workflow remains largely evaluative rather than constructive: it can tell us that a hypothesis is imperfect, but not how to improve it. We address this gap by shifting the unit of analysis from a global score to the structure of the input space. Given a low-level model \mathcal{L} , a high-level hypothesis \mathcal{H} , and an alignment \Pi , we partition the input space into well-interpreted target buckets with high IIA and a complementary bucket that captures the remaining failure modes. The key question is no longer just whether an abstraction works on average, but for which inputs it is actually faithful. This yields a more informative diagnosis and turns abstraction failures into evidence for refinement. Similar ideas of exploiting structure in observational or interventional partitions have appeared in causal feature learning ( 5 ; 4 ) . Our main contribution is a practical four-step pipeline for diagnosing and improving causal abstractions: specify a reliable task and input space, obtain a candidate alignment, partition the input space by pairwise interchangeability to identify nearly interchange-consistent subsets, and train a classifier to generalize this diagnosis beyond the analyzed sample. Across fine-tuned and pretrained models of different sizes, and across alignment methods including full-vector patching, DAS, and MDAS, we show that this bucketing procedure is broadly useful for diagnosing causal abstractions. Empirically, we validate it on logic, entity binding ( 16 ) , and factual recall ( 15 ; 18 ; 14 ; 20 ) . In the toy logic task, recursively applying the recipe supports iterative refinement of the high-level hypothesis itself, showing how causal abstraction can move from post hoc evaluation toward constructive hypothesis discovery. Figure 1: Four-step interpretation diagnosis pipeline. Given a causal abstraction for a task-performing model, we focus on task-correct inputs, identify an alignment, partition the input space by pairwise interchangeability, and train a classifier to generalize the diagnosis. The rest of the paper is organized as follows. Section 2 reviews the causal abstraction framework and the alignment and feature-learning tools used in this work. Section 3 formalizes interchange-consistent subsets and introduces our diagnosis pipeline. Section 4 evaluates the method on three settings of increasing complexity, and Section 5 concludes with limitations and future directions. 2 Preliminaries Causally abstracting an LM involves hypothesizing and verifying that a higher-level causal model is an abstraction of the lower-level LM. Causal models—familiar from the literature on causal inference (e.g., 27 )—comprise variables, on which interventions predict counterfactual behavior. After hypothesizing alignments of the causal variables within LM representations and verifying the predicted counterfactual behavior for all such representations of an LM with appropriate interventions, the higher-level model can be taken to causally abstract the LM, if the corresponding interventions on the corresponding causal variables predict the same counterfactual output. We adopt notation from 8 . Causal Abstraction. Specifically, we use \displaystyle\mathcal{H},\mathcal{L} to represent a high-level and a low-level causal model, respectively. For each variable \displaystyle X , we notate its domain as \displaystyle\mathbb{V}_{X} . For \displaystyle X in \displaystyle\mathcal{H} , \displaystyle(\pi_{X},\tau_{X}) is an alignment that pairs a high-level variable \displaystyle X with a set of low-level variables \displaystyle\pi_{X} and maps variable values of \Pi_{X} to variable values of X , i.e., \displaystyle\tau_{X}:\prod_{Z\in\pi_{X}}\mathbb{V}_{Z}\rightarrow\mathbb{V}_{X} . Moreover, we define an alignment \displaystyle\Pi of two models \displaystyle\mathcal{H},\mathcal{L} as a collection of variable alignments, i.e., \displaystyle\Pi=(\{\pi_{X}\},\{\tau_{X}\}) . Given a causal model \displaystyle\mathcal{M} , we define \displaystyle\mathcal{M}_{V\leftarrow v}(x) to be the output of \displaystyle\mathcal{M} with input \displaystyle x after setting \displaystyle V=v . For \displaystyle\mathcal{M} , where \{s_{k}\} is a source input setting and \{V_{k}\} is a set of intermediate variables, the interchange intervention ( II ) is defined as: \displaystyle\textit{II}(\mathcal{M},\{s_{k}\},\{V_{k}\})(b) \displaystyle=\mathcal{M}_{V_{k}\leftarrow\text{GetVals}_{V_{k}}(\mathcal{M}(s_{k}))}(b), where \displaystyle\text{GetVals}_{V_{k}}(\mathcal{M}(s_{k})) is the value of variables \displaystyle V in model \displaystyle\mathcal{M}(s) . Similarly, one can define II with multiple sources as in 8 ; 13 . Ideally, we want the alignment \displaystyle\Pi to satisfy \displaystyle\tau(\textit{II}(\mathcal{L},s,\pi_{X})(b)) \displaystyle=\textit{II}(\mathcal{H},\tau(s),X)(\tau(b)). Distributed Alignment Search (DAS). Standard-basis interventions implicitly assume that a high-level variable is localized in a particular set of neurons, but neural representations are often distributed across overlapping directions ( 30 ; 26 ; 29 ; 13 ) . Distributed Alignment Search (DAS) addresses this by replacing “localist” alignment search with gradient-based search over distributed linear subspaces. Concretely, for a high-level variable X and a candidate low-level representation \mathbf{h}_{\pi_{X}}\in\mathbb{R}^{n} , DAS learns an orthogonal matrix R_{\theta} and aligns X with a k -dimensional subspace of the rotated representation R_{\theta}(\mathbf{h}_{\pi_{X}}) , rather than with coordinates in the standard basis