Position: Causality is Key for Interpretability Claims to Generalise Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, Dhanya Sridhar Abstract Interpretability research on large language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl’s causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (e.g., average change in token probabilities) over a set of prompts. However, counterfactual claims—i.e., asking what the model ouput would have been for the same prompt under an unobserved intervention—remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise. † † affiliationnotice: * Equal advising; authors listed in alphabetical order. 1 Introduction Interpretability research on LLMs has produced a growing toolkit for linking model behaviour to internal structure, e.g. circuits trace task computations ( 115 ; 42 ; 154 ) , activation patching localises contributions of specific model components ( 150 ) , sparse autoencoders (SAEs) surface human-readable features ( 33 ) . Yet practitioners frequently encounter a gap between local success and reliable deployment, e.g., a linear probe achieves high accuracy, but steering the same direction fails to produce reliable behavioural change ( 144 ) , or an ablation suppresses a behaviour on the test set, yet the same circuit proves brittle under distribution shift ( 106 ) . These patterns raise a general epistemological challenge: high predictive accuracy does not, by itself, establish the existence of manipulable mechanisms ( 74 ) . To move beyond purely correlational accounts, recent interpretability work has focused on grounding model behaviour in internal structure. This shift has been framed as a turn toward mechanistic interpretability. In its original usage, mechanistic explanation characterised how causes produce effects ( 93 ; 156 ; 155 ) , asking not just what a system computes but how ( 99 ) , and which events count as the cause(s) ( 53 ; 54 ) . When interpretability researchers set out to “reverse engineer the detailed computations performed by transformers” ( 42 ) , they implicitly inherited these causal commitments. Claims that a circuit computes a function, that an attention head mediates a behaviour, or that an SAE feature controls a capability are claims about causal influence within the model. Causal inference provides vocabulary needed to make these claims precise: What variables are we positing? What interventions define the causal relationships? What alternative explanations remain compatible with our evidence? These questions have well-developed answers in causal inference. To formalize the variables and claims, we can specify an estimand , the precise quantity a method targets. For instance, a probe targets decodability of a concept (an associational estimand), which is distinct from asking how the output changes if the decoded concept is perturbed (an intervention-effect estimand). An intervention class then specifies the manipulations that would be needed to estimate the intervention-effect estimand, i.e., the change in a chosen output metric induced by perturbing the representation of the decoded concept. To obtain a unique estimate, we must determine which perturbations are indistinguishable given our measurements, which is captured by an equivalence class . Finally, we must ask when conclusions drawn from these measurements are transferable, a question addressed by causal transportability ( 121 ; 10 ) . Without this scaffolding, interpretability claims become ambiguous, where the same word—“mechanism”, “feature”, “circuit”—can refer to different estimands and intervention classes, making results hard to compare across papers ( 136 ; 111 ) . More practically, methods can succeed locally yet fail when deployed because the evidence answers a different question than the one implied by the claim ( 15 ; 139 ) . For instance, decodability (an association) may be used to justify control (an intervention effect). Importantly, the interpretability community has itself identified many of these challenges: recent works advocate for necessity and sufficiency testing ( 56 ) , document how entanglement limits single-feature interventions ( 110 ) , and call for more rigorous evaluation methodology ( 94 ; 139 ) . Our contribution is not to discover these issues but to provide a unified causal framework that makes them precise. We use Pearl’s causal hierarchy ( 121 ) to diagnose potential claim-evidence mismatches, and causal representation learning (CRL) to specify the assumptions required for common claims in interpretability research. One response to these limitations is the emerging shift to pragmatic interpretability ( 125 ) , which advocates for iterating against measurable proxies rather than “using model internals to understand or explain behaviour”. This posture is not new ( 11 ; 36 ; 24 ; 124 ) and has precursors in explainable machine learning ( 114 ) . where interpretability is treated as purpose-relative and evaluated by downstream use rather than discovering a fixed set of concepts ( 37 ; 88 ; 108 ) . We take this shift as a useful point of comparison and argue that causal inference provides complementary tools for specifying when proxy-based success should generalise, and when it may not ( 32 ; 67 ; 86 ) . Our position is that interpretability claims should be stated in the language of causal inference and identifiability : specify the estimand, the intervention class, and the equivalence class implied by the available evidence. The payoff is practical : this framing helps practitioners choose methods that actually answer the question of interest, diagnose mismatch failures between method and goal, and state the conditions under which the conclusions will transfer. A shared causal vocabulary also helps verify and compare claims across methods and applications. 2 Position: Interpretability Requires Identifiable Causal Quantities Interpretability research aims to make precise claims about model internals, yet—as in any application of the scientific method—such claims require committing to a well-defined target of inference, which often remains implicit in questions like “What does this head do?” or “Is this the honesty feature?” Without a precise target, even well-designed metrics risk validating structure that is not meaningfully different from random baselines ( 55 ; 102 ) . Notably, this is a known problem when testing many hypotheses in high-dimensional data ( 14 ) , not one unique to interpretability. We argue that making interpretability claims more reliable requires three steps, formalised with the language of causality, for which we briefly introduce informal meanings of key terms: Estimands and Identifiability Estimand: A quantity that would answer the question of interest, if it can be computed exactly. Estimator: A procedure approximating the estimand from data. Equivalence class: Hypotheses that are indistinguishable given interactions with the model. Identifiability: An estimand is identifiable up to an equivalence class if there exists an estimator that provably computes that estimand. The causality recipe. First, we use the causal ladder (see box below) to provide a taxonomy for distinguishing associational, interventional, or counterfactual questions ( Section 2.1 ). This makes the target question explicit and mathematically precise. Second, we characterize the sufficient evidence and assumptions to identify the answer to the causal question ( Section 2.2 ). Third, we investigate the plausibility of the theoretical insights under practical constraints and identify potential gaps ( Section 2.3 ). Pearl’s Causal Ladder L1 \cdot Associational. Statistics from observed data. Query: Does A correlate with B ? L2 \cdot Interventional. Effects of controlled modifications. Query: Does changing A change B ? L3 \cdot Counterfactual (Instance-level). Alternative outcome for the same instance under an unobserved intervention. Query: For this input, would changing A have changed B ? Different rungs answer different questions. Choosing the right one depends on your goal. Higher rungs require stronger evidence: L( k ) evidence does not license L (k{+}1) claims ( 121 ; 9 ) . Notation. Details are in Appendix A . Consider a pretrained model with L layers and parameters \theta . For input \mathbf{x}\in\mathbb{R}^{d_{x}} , define layerwise activations recursively as \mathbf{a}^{(0)}:=\mathbf{x} and \mathbf{a}^{(l)}:=f^{(l)}_{\theta}(\mathbf{a}^{(l-1)}) for l=1,\dots,L , where \mathbf{a}^{(l)}\in\mathbb{R}^{d_{a}^{(l)}} denotes the raw activations at layer l . The model output is given by \mathbf{y}:=\mathbf{a}^{(L)} . We distinguish activations from representations \mathbf{h}^{(l)}:=\phi^{(l)}(\mathbf{a}^{(l)}) , where \phi^{(l)}:\mathbb{R}^{d_{a}^{(l)}}\rightarrow\mathbb{R}^{d_{h}^{(l)}} is a (possibly learned) map 1 1 1 Since our arguments apply to any fixed layer, we drop the layer index and write \mathbf{a} , \mathbf{h} , \phi . to a basis where structure may be more apparent. A feature is simply a subspace S\subset\mathbb{R}^{d_{h}^{(l)}} . 2 2 2 Different connotations exist including those requiring semantic consistency or human interpretability ( 41 ; 20 ) , but we adopt a purely geometric notion following 44 ; 109 . See Section A.2 for other usages of the term. Whether a feature is interpretable or meaningful is an empirical question requiring additional evidence. 2.1 Interpretability Questions Are Causal Questions We argue that the questions of interpretability research can be mapped to the three rungs of Pearl’s causal ladder and we use this classification to align each claim with the evidence required to support it. This perspective clarifies a recurring pattern in the literature ( 6 ; 16 ; 20 ; 128 ) : purely associational evidence (L1) is often reported in the language of counterfactual or interventional claims (L3/L2), as illustrated in Table 1 . The resulting rung mismatch has concrete consequences for reliability and safety, as interventions that appear effective on benchmark evaluations frequently fail under distribution shift. We illustrate the implications at each rung using a running example of refusal , expressed in terms of conditional probabilities and potential outcomes. Table 1 : Interpretability methods typically produce associational or interventional evidence (L1–L2), yet the interpretations we’d like to draw often implicitly require counterfactual reasoning (L3) . Recognising this rung gap clarifies what additional evidence is needed to justify stronger claims. Method Aim What the Evidence Supports Sparse autoencoders L3 The learned features correspond to a unique set of concepts. Identifiability claim. L1 A sparse basis that minimises reconstruction error on the training data. Describes a basis, without yet establishing uniqueness. Auto-interpretability Explanations L3 This feature corresponds to the underlying concept named by the description. Semantic assignment. L1 The description predicts when the feature activates on held-out text. Distinguishes activating from non-activating contexts, without confirming the feature’s causal role. Circuit discovery L3 This circuit is a key mediator of the behaviour. Causal attribution. L2 Ablating this circuit changes model behaviour on evaluated prompts. An intervention effect for the chosen ablation, not yet a unique localisation. Worked Example: Causing Refusal L1 Do certain activations correlate with refusal behaviour? Given prompts \mathbf{x} labeled by whether the model refuses ( \mathbf{y}=1 ) or complies ( \mathbf{y}=0 ), an associational query asks whether a representation \mathbf{h} is predictive of refusal under the observational distribution: p(\mathbf{y}\mid\mathbf{h})\neq p(\mathbf{y}) . Licenses: Refusal is decodable from this layer. Does not license: The model represents refusal here, or this layer computes refusal. L2 Can we induce refusal by manipulating activations? Let \mathrm{do}(\mathbf{h}:=\widetilde{\mathbf{h}}) denote an intervention that replaces the representation at a chosen layer with a fixed value \widetilde{\mathbf{h}} , leaving the remainder of the computation intact. An interventional query compares p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}:=\widetilde{\mathbf{h}})) against the baseline p(\mathbf{y}) or against alternative interventions. Licenses: Intervening here changes refusal under tested conditions. Does not license: This is the refusal mechanism, or that it generalises to novel jailbreaks. L3 For a specific prompt on which the model was jailbroken, what activation change would have caused refusal? Given an observed triple (\mathbf{x}_{0},\mathbf{h}_{0},\mathbf{y}_{0}) where \mathbf{y}_{0} represents harmful compliance, a counterfactual query seeks \tilde{\mathbf{h}} such that \mathbf{y}_{\mathbf{h}\leftarrow\widetilde{\mathbf{h}}}(\mathbf{x}_{0}) corresponds to refusal. Licenses: For this forward pass, the intervention would have caused refusal. Does not license: Generalises to other inputs without a structural model. A comprehensive reference with the full ladder taxonomy and worked examples is provided in Appendix D ; Table 1 illustrates the mapping for common methods. The ladder specifies the question’s rung, and the estimand specifies the quantity needed to answer the question of interest. A core concern, to which we now turn, is identifiability : whether the available evidence is sufficient to uniquely determine the chosen estimand. For instance, observing that A correlates with B is consistent with A causing B , but equally consistent with a confounder causing both. Interventional evidence can distinguish these cases; associational evidence cannot. Thus, Pearl’s causal ladder classifies causal questions by the type of evidence (associational, interventional, counterfactual) required to answer them, but it does not ensure that the evidence pins down a unique answer. 2.2 The Right Question Is Not Enough. Answers Must Be Identifiable After specifying an estimand, we must establish its identifiability, i.e., whether the available evidence rules out alternative values of the estimand. Identifiability is therefore best understood as a statement about what structure or quantity is invariantly recoverable, not about what entities exist . It characterises an equivalence class of explanations consistent with the evidence, not a unique ground truth (see Section B.1 for a philosophical discussion). Consider the interventional (L2) estimand p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}^{(l)}:=\widetilde{\mathbf{h}}^{(l)})) that measures whether the activations at a layer causally influence a model’s rate of refusal beyond some baseline rate p(\mathbf{y}) . Suppose we only observe that \mathbf{h}^{(l)} accurately predicts refusal. The estimand is not identifiable from the evidence we have, i.e., the available evidence could be equally well explained by earlier layers encoding an alternate concept that affects both the activations \mathbf{h}^{(l)} and refusal. Thus, locating an estimand on the causal ladder helps determine the strength of the evidence needed for identifiability. However, there are practical considerations for identifiability. For example, interventional evidence (L2) is costly: perturbing neurons and measuring behavioural changes requires many experiments and human annotation of outputs with interpretable concepts. Such practical considerations have paved the way for unsupervised methods like sparse autoencoders (SAEs), which learn a change of basis \mathbf{h}^{(l)}:=\phi^{(l)}(\mathbf{a}^{(l)}) regularised toward sparsity, with the hypothesis that sparse coordinates are interpretable. However, even if SAE features are interpretable, that implies neither uniqueness nor identifiability—at least not automatically. The use of either auto-interpretability explanations (L1 evidence; 16 ; 118 ) or interventions (L2 evidence) cannot pin down the meaning of a feature precisely, which requires L3 evidence (as seen in Section 2.1 ). In fact, auto-interp scores can attain high values even on random structure ( 55 ) . These problems, we argue, can be characterised and understood with an identifiability lens. Namely, identifiability results prove that without additional structure, unsupervised recovery of latents is impossible ( 65 ; 89 ) . The central question, then, is not whether unsupervised methods can discover structure, but under what conditions the structure they recover is causally meaningful . Identifiability in CRL. The field of causal representation learning (CRL, 137 ) focuses on identifiable learning of causal variables. Concretely, CRL generally assumes that some ground-truth human-interpretable latent variables {\mathbf{c}}\in{\mathbb{R}}^{d_{c}} produce observations {\mathbf{x}} via an unknown generative process g:{\mathbb{R}}^{d_{c}}\rightarrow{\mathbb{R}}^{d_{x}} . The underlying variables {\mathbf{c}} are considered to be causal variables since we can imagine intervening on them as humans. The goal of CRL is to develop assumptions about the observations or generative function g to render the unmixing function {\mathbf{c}}=g^{-1}({\mathbf{x}}) identifiable up to a simple equivalence class (e.g. permutation and rescaling of {\mathbf{c}} ). Typically, identifiability requires observations gathered under multiple interventions on the latents {\mathbf{c}} to learn both the latent factors and their causal structure. Applying CRL to pretrained LLM activations. CRL methods offer a way to learn the map \mathbf{h}^{(l)}:=\phi^{(l)}(\mathbf{a}^{(l)}) to reinterpret activations \mathbf{a}^{(l)} in a coordinate system where the axes have causal semantics. Indeed, recent papers ( 73 ; 44 ; 50 ; 95 ; 129 ; 142 ) have already begun developing sufficient assumptions to identify some concept classes from observations reflecting natural concept variation. The practical question is whether the assumptions guaranteeing identifiability are guaranteed hold in practice, and whether the resulting equivalence class suffices for the task at hand. While making these assumptions explicit and testing design choices against them may require extra effort upfront, they provide a practical payoff: identifiability guarantees can reduce experimentation costs by specifying how a claim is supported, independent of model size, data scale, or compute. Concrete Example. Sparse shift autoencoders (SSAEs) 73 leverage paired samples that reflect diverse perturbations to an unknown number of concepts. The identifiability result guarantees the perturbed latents are identified up to permutations and scaling. If SSAEs are trained on data reflecting changes to human notions of target concepts (e.g., sentiment), such concepts are provably recovered. However, if the naturally occurring perturbations always jointly affect, e.g., sentiment and topic, the identified quantities will not reflect sentiment or topic separately. This interpretation of CRL results can be linked to the idea of affordances, which we describe next. Concepts as Affordances While classical CRL assumes ground-truth latents {\mathbf{c}} and a ground-truth generative process g , interpretability lacks true concept labels. Therefore, we ask not whether a representation recovers true latents, but what it affords for specific interactions. Affordance: What an interpreter can discover about a model depends on the interactions available to them ( 49 ) . Different probing methods—like different tools—reveal different structure. This may include novel structure or mechanisms, potentially requiring neologisms to describe. Example: Consider two iron filings—one magnetised, and one not. They are indistinguishable under interactions such as: looking, weighing, or picking up. Only interaction with a magnet makes the distinction observable. Likewise, a representation may encode structure that only becomes visible under the right interaction . Identifiability as Interaction-Relative. We cannot test whether sentiment varies independently of topic unless we observe settings where one changes without the other. What can be identified is therefore bounded by the variations the system affords. As in first-contact 3 3 3 The challenge of interpreting an unknown system when no shared reference frame exists to verify meaning, analogous to cryptography without a Rosetta Stone, or encountering an alien civilisation , behaviour alone admits multiple interpretations—“cold” may reflect temperature, discomfort, or a metaphor—so evidence identifies only an equivalence class consistent with the interactions performed ( 126 ; 34 ) . Interpretation is thus bidirectional ( 7 ) , and may surface structure for which humans lack vocabulary, requiring the creation of new words (neologisms) ( 138 ; 59 ) . Anthropomorphic pattern-seeking ( 22 ; 98 ) risks projecting human concepts onto “alien” structure, as seen in topic modeling where labels impose meaning on arbitrary clusters ( 25 ) . Why require identification guarantees if a method works in practice? There’s remarkable evidence showing that interpreted concepts can improve chess grandmasters ( 138 ) , steering vectors can induce refusal ( 6 ) , and probes can accurately predict sentiment ( 31 ) or truthfulness ( 97 ) . Then one might ask: why do we need to recover latent variables that correspond to a uniquely identifiable or causally meaningful structure? This is because the evidence is not guaranteed to generalise. Model edits revert on paraphrases ( 60 ) ; steering vectors miscalibrate OOD ( 147 ) ; probes fail under distribution shift ( 91 ) . This might suffice for exploration. But we know that for safety-critical deployment, asking why something works, and when it will stop working is unavoidable. Identifiability makes this question answerable: it specifies which assumption to verify (e.g. concept variation) and what failure to expect if it breaks. This provides means to rule out failure modes pre-deployment, rather than post-hoc. Relocating supervision. There is no free lunch in interpretability: supervision enters both through the affordances a method is designed to leverage (rather than labelled datasets), and through downstream interpretation; unsupervised objectives and evaluations do not, by themselves, distinguish representations that uncover meaningful variables from those that do not. Thus, without identifiability, fitting a latent variable model recovers some latent coordinates that reproduce the observed distribution, such that any invertible remixing of these coordinates (such as, a rotation, or even a nonlinear reparameterisation) yields the same solution under the designed objective 4 4 4 For instance, consider the reconstruction objective ||{\mathbf{x}}-q(r({\mathbf{x}}))||^{2}_{2} where ( q , r ) form an autoencoder pair. Any invertible transformation a would give the exact same value of the objective: ||{\mathbf{x}}-(q\circ a)(a^{-1}\circ r({\mathbf{x}}))||^{2}_{2} for a different autoencoder pair (q\circ a,a^{-1}\circ w) thus resulting in latent coordinates a^{-1}\circ r({\mathbf{x}}) instead of r({\mathbf{x}}) . . Consequently, assigning individual semantics (such as, “latent i is speech”, and “latent j is music”) is not stable: another equally good solution can blend speech and music across multiple latent components. Identifiability pins down this meaning, as an invariance across the equivalence class of observationally equivalent solutions, i.e., if the latent coordinates are the same in every solution (up to elementwise indeterminacies that still do not mix them), then their meaning gets fixed, enabling us to interpret them and intervene on them. Conversely, if we find the latent coordinates not to be identifiable, the theory guides us to what to add in terms of more diversity of data, or additional modelling constraints. 2.3 A Unified Causal Lens Predicts Interpretability Failure Modes Our framework provides language to characterise the relationship between interpretability claims and their supporting evidence. The key question is the gap, if any, between the asked-for estimand (what the narrative implies) and the identified estimand (what the method actually recovers): asked-for \displaystyle\tau^{\star} but identified \displaystyle\hat{\tau} . Two factors (often compounding) help structure this analysis: 1. Rung mismatch : The evidence lives on a lower rung of Pearl’s ladder than the claim requires (e.g. associational evidence for an interventional claim). 2. Identification gap : Even at the correct rung, the target is identified up to an equivalence class of models, conditional on datasets assumptions and modelling choices. To measure how well claim language in interpretability papers tracks the evidential strength of the reported methods (i.e., whether a phrase’s common interpretation matches its formal causal meaning), we conducted a pilot study described below. Calibrating Claim Language to Evidential Strength We annotated 50 papers (186 claims; protocol in Appendix G ) on interpreting model internals, and assigned each claim a method rung (what the reported procedure establishes) and a claim rung (what the surrounding language asserts). Since most papers do not state a causal estimand explicitly, we operationalise claim rungs by mapping common verbs (e.g. “encodes,” “mediates”) onto the ladder. These verbs often admit causal readings—though they may also reflect disciplinary convention—so the same sentence can be interpreted as making a stronger claim than the reported evidence supports. We therefore mark a potential claim–evidence gap when claim rung > method rung. We provide a practitioners’ checklist in Section G.4 to help align claims with evidentiary support. Result. Roughly half of claims can admit a stronger interpretation than the evidence rung licenses—reflecting the field’s nascent terminological infrastructure rather than unreliable findings (exact figures in Appendix G ). Common patterns include: probing results (L1) narrated as causal mechanisms (L2); patching evidence (L2) narrated as counterfactual necessity (L3); and single-distribution findings generalised beyond their empirical scope. Annotator confidence drops monotonically with gap size (mean 4.9 for gap-free claims vs. 4.0 for two-rung gaps), confirming that larger rung distances involve genuinely harder judgments. We use causality to analyse some widely-used interpretability techniques for potential estimand-evidence gaps in Table 2 . We also acknowledge prior interpretability work raising similar concerns. Next, we analyse representative examples that translate interpetability workflows into a common causal template: the implicit estimand and the evidence along with additional tests that would connect the two (see Appendix F for more). Our goal is to formalise known caveats to diagnose an estimand–evidence gap and the minimal additional evidence that would resolve such that claims will generalise. Case Study I: Sufficient \neq Necessary. Activation patching shows that intervening on component set S changes behaviour; the result is often narrated as “ S is the mechanism for B .” Evidence (L2). The experiment establishes that intervening on S is sufficient to, on average, shift the output distribution away from the unmodified model: identified \displaystyle\mathbb{E}_{\mathbf{x}\sim p}\big[p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}_{S}:=\tilde{\mathbf{h}}_{S}),\mathbf{x})\big] \displaystyle\neq\mathbb{E}_{\mathbf{x}\sim p}\big[p(\mathbf{y}\mid\mathbf{x})\big]. Gap. The mechanism asserts necessity and uniqueness, but the evidence establishes neither. Other pathways S^{\prime}\neq S may be equally sufficient ( 154 ; 101 ) , and sufficiency (intervention changes output) does not entail necessity (without S , output would have been different). What would resolve it (L3). Necessity is a counterfactual claim—for a specific input, ablating S would have changed the output: asked-for \displaystyle\mathbf{y}_{\mathbf{h}_{S}\leftarrow 0}(\mathbf{x}_{0})\neq\mathbf{y}(\mathbf{x}_{0}). This requires a structural model specifying what is held fixed. Uniqueness requires additionally ruling out alternative pathways within a defined mechanism class. Takeaway. Patching identifies only a sufficient (but not necessary) control handle; necessity requires L3 evidence. Establishing uniqueness (ruling out alternatives) is a distinct requirement, and neither is established by L2 evidence. Testing for both denoising (sufficiency) and noising (necessity), as advocated by 56 , is one practical step toward closing this gap—their recommendation has a direct theoretical grounding. Case Study II: Proxy Gaming An SAE yields sparse features with high reconstruction accuracy and strong auto-interpretability scores; the result is reported as discovering disentangled or concept-aligned representations. Evidence (L0). The method identifies a representation that satisfies the training objective for encoder \phi –decoder \psi pair: identified \displaystyle\arg\min_{\phi,\psi}\ \mathbb{E}[\|\mathbf{a}-\psi(\phi(\mathbf{a}))\|^{2}]+\lambda\|\phi(\mathbf{a})\|_{1} The solution would minimise reconstruction error while being sparse (as encouraged by the \ell_{1} sparsity penalty). Gap. Multiple sparse factorisations can achieve similar objective values. Thus, while sparsity is a useful inductive bias, it does not guarantee identification. asked-for \displaystyle\exists\hskip 5.69054pt{\mathbf{c}}\ \text{s.t.}\ \phi(\mathbf{a})=\mathbf{P}\mathbf{D}{\mathbf{c}} where \mathbf{P} and \mathbf{D} are permutation and diagonal matrices such that the desired representation identifies the concepts up to permutation and scaling. What would resolve it. Sufficient identifiability conditions under which the recovered representation is unique up to trivial ambiguities (permutation and scaling). For example, 84 show that if there is sufficient variability across latent variables’ contexts, then the representation is identifiable. In practice, this means verifying that the data exhibits such variability and not just that the objective value is low. 73 achieve this with SSAEs in practice by uniformly sampling pairs of contexts. Takeaway. Optimising the reconstruction objective and achieving a low \ell_{1}- norm are necessary but not sufficient for identifiability of a concept. It is known that proxies such as reconstruction error or the \ell_{0}- norm cannot be used for model selection ( 89 ) . Case Study III: Steering a Concept A steering vector \mathbf{v} reliably shifts behaviour toward refusal or politeness. Usually, this iis formulated as the model represents honesty or has a refusal variable . Evidence (L2). Adding a scaled direction \alpha\mathbf{v} to the activation controllably shifts the output distribution: identified \displaystyle p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}:=\mathbf{h}+\alpha\mathbf{v}))\ \text{varies with }\alpha, where \alpha\in{\mathbb{R}} denotes steering strength. This establishes that \mathbf{v} affords behavioural control. Gap. Interpreting controllability as encoding implicitly asserts that \mathbf{v} corresponds to an internal causal variable mediating the model’s computation for a specific concept: asked-for \displaystyle\mathbf{v}\text{ corresponds to a causal variable} mediating the computation of {\mathbf{y}} for a specific concept. However, \mathbf{v} may entangle multiple concepts, may not be unique, and may exploit distributional shortcuts rather than the model’s natural computation ( 147 ; 72 ; 157 ; 110 ) . What would resolve it. Evidence that \mathbf{v} is a stable, reusable causal variable—e.g. transfer across distributions ( 146 ) , specificity to a constrained subspace ( 96 ) , and ideally a causal abstraction in which interventions on \mathbf{v} compose appropriately with other model computations ( 45 ) . Concurrent work by 107 demonstrated that an orthogonality regulariser, inspired by the Independent Causal Mechanisms (ICM) principle ( 69 ) , decreased the interference between features under interventions. Takeaway. Steering identifies a control handle on a particular concept, but claiming that the model represents that concept asserts that it is an internal causal variable. The former is supported by L2 evidence, while the latter requires additional structural assumptions beyond controllability. These patterns recur across the interpretability literature ( 154 ; 101 ; 55 ; 147 ; 157 ; 110 ) . In each case, the gap between evidence and claim follows predictably from the rung of the causal hierarchy at which the method operates. Making this explicit clarifies what additional assumptions or experiments are needed to support stronger claims. 3 Alternative Views We briefly situate our framework relative to other prominent views in current interpretability research, highlighting points of alignment and how our framework can contribute additional insights to these perspectives. Table 2 : Common estimand-evidence gaps in mechanistic interpretability. Each row contrasts the Intended Scope of a claim with the Actual Scope supported by the reported evidence. The third column points to work that documents the gap or demonstrates ways to address it. Our aim is to offer shared terminology that helps unify these efforts and make progress comparable. Inferential Gap Implied Claim Actual Scope Existence \to Uniqueness This is the circuit or factor causing a behaviour. A circuit that produces output aligned with the behaviour has been found, but the solution is generically non-unique. Other circuits can implement equivalent input–output behaviour ( 154 ; 101 ) . Correlation \to Causation Feature {\mathbf{h}}^{(l)}_{S} causally mediates behaviour related to concept c . {\mathbf{h}}_{S} and c are statistically correlated. Without targeted interventions (eg. interchange interventions 45 or causal scrubbing 23 ), the correlation may reflect confounding rather than a causal pathway. Decodability \to Model Use The model represents and computes with concept c . A probe can decode c from {\mathbf{h}}_{S} , but decodability does not entail that the model’s computations depend on c . Probes can extract information that the model never uses ( 13 ; 40 ; 57 ; 130 ) . Local Sensitivity \to Global Causal Role The mechanism generalises beyond tested prompts. Sensitivity to c has been verified on specific inputs. Local attributions and patching can be input dependent and may not generalise to held-out distribution ( 1 ; 17 ) . Subspace-level \to Coordinate-level claim A probe or PCA direction that correlates with concept c identifies a privileged coordinate-level encoding of c . A probe recovers c from a linear combination of many neurons; this does not mean any single neuron encodes c , it is only recoverable from a subspace. The finding reduces to a coordinate-level claim only when the probe direction is axis-aligned ( 41 ; 117 ; 38 ) . Sufficiency \to Necessity This component or mechanism is necessary for observed behaviour. A sufficient pathway under specific interventions may be sufficient to produce the desired behaviour, but sufficiency does not entail necessity. Alternative pathways may exist which sustain the behaviour when the pathway is ablated ( 154 ; 56 ) . Relative Evidence \to Absolute Identification The canonical circuit or factor has been recovered. The method identifies a solution only up to an equivalence class. Without additional structural constraints, unsupervised methods cannot pin down a unique factorisation ( 89 ; 65 ) . 3.1 Interpretability Can Be Wholly Pragmatic Position. 112 argue for a pragmatic criterion: choose proxy tasks such that success would enable progress on an ultimate goal . Behavioural outcomes on well-chosen proxies, not internal metrics, determine progress, directly addressing a known failure mode where unsupervised metrics (reconstruction, sparsity, auto-interp scores) provide little evidence about the structure learned by the model ( 89 ) . Rebuttal. The criterion implicitly assumes what it seeks to avoid: proxy-task success is informative only insofar as the proxy identifies structure that is relevant beyond the proxy distribution. Put differently, a proxy can justify stronger conclusions only when it rules out alternative internal explanations that would also solve the proxy yet fail on the target objective. This is an identifiability requirement: the available evidence (here, proxy performance, interventions, and stress tests) must constrain the recovered representations such that recovered features are empirically indistinguishable. Recent evidence exposes the gap: SAE probes fail under distribution shift ( 76 ) , which is not a failure of SAEs per se, but of identification 5 5 5 This is a reason to improve how SAEs are evaluated and constrained, not to discard them. Identifiability conditions offer one solution —the training distribution did not sufficiently constrain the representation to fix which features should generalise. Identifiability makes this failure predictable rather than surprising, along with additional sufficient conditions to mitigate it. 3.2 Symmetries are Sufficient to Formalise Interpretability Position. Concurrent work by 8 define interpretability through symmetry constraints: i.e., a specific set of transformations under which an explanation preserves its meaning. Concretely, if two internal descriptions that are transformations of each other induce the same model behaviour, then any change in the produced explanation under the transformation is attributable to the interpreter’s choice of representation rather than to model-intrinsic structure. This lens complements identifiability: symmetry constraints prescribe which transformations the explanations should be invariant to, and identifiability characterises the equivalence classes of representations that are empirically indistinguishable given available evidence. Rebuttal. Two tensions remain. First, while the framework specifies desirable invariance principles, it leaves underspecified how violations translate into concrete failure modes of existing interpretability methods; making the path from symmetry violation to empirical breakdown explicit would strengthen its operational value. Second, privileging human-interpretable symmetries risks a streetlight effect: model-relevant structure that does not respect these symmetries may be systematically overlooked, even when it materially affects behaviour (see discussion in Section 2.2 ). 4 Call to Action. The failure modes catalogued in Section 2.3 can be addressed by recognizing that identifiable causal representation learning (CRL) and interpretability are mutually beneficial. CRL provides formal tools for reasoning about equivalence classes, causal levels, and transportability, while interpretability supplies empirical phenomena and safety-relevant evaluation criteria. Below we outline research directions that leverage this complementarity, specifying what each field contributes and what concrete work would result. 4.1 Counterfactual Semantics for Safety Safety verification asks counterfactual questions: “Would this output have been harmful had we not intervened?” Activation patching provides interventional (L2) evidence; counterfactuals (L3) additionally require specifying which variables are held fixed. CRL offers: Structural models formalising exogenous variables and counterfactual semantics, clarifying when counterfactual quantities are well-defined under explicit abstraction assumptions, c.f., e.g., 45 . Interpretability offers: Realistic architectures (residual streams, attention) that stress-test whether standard structural assumptions hold. Research questions: While counterfactual guarantees (L3) are the ideal target, they are exceptionally difficult to obtain in practice. However, interpretability provides a useful testbed in practice to operationalise where we can ask: 1. When does activation patching coincide with counterfactual conditioning in transformers? 2. For circuits where it does not (e.g., residual-stream coupling), what minimal exogenous annotations restore counterfactual semantics? 3. Can we construct tasks where L2 and L3 answers provably diverge? 4.2 Task-Relative Equivalence Classes. Different tasks require different strengths of identification: Task asked-for Minimal equivalence class Binary classification Affine (sign suffices) Steering Block-wise (direction suffices) Knowledge editing Elementwise (magnitude matters) CRL offers: A formal vocabulary for equivalence classes induced by symmetries and data constraints. Interpretability offers: Empirical evidence (e.g., steering succeeds where editing fails) revealing which equivalence classes are sufficient in practice. Research questions: 1. What are the weakest equivalence classes empirically sufficient for reliable steering or knowledge editing? 2. Can representation learning objectives be designed to target task-specific equivalence classes directly? 4.3 Compositional Control Compositionality varies across intervention types: model edits do not compose ( 60 ) , while task vectors compose linearly ( 66 ) . Establishing whether and when steering vectors compose is essential for claiming they can be used to control variables of interest. CRL offers: Algebraic structure for intervention families (closure, associativity) clarifying when composition is even well-defined. Interpretability offers: Empirical compositional failures that reveal violations of these assumptions. Research questions: 1. Under what conditions do representation-level interventions compose additively? 2. Can compositional failures be predicted from representation geometry (e.g. feature overlap or shared support)? 3. Does stronger identifiability correspond to more reliable compositional control? (e.g. 110 ) 4.4 Transportability as a Theory of Edit Generalisation Failures of model editing ( 28 ; 60 ) are predictable consequences of unexamined transportability assumptions ( 9 ) . CRL offers: Necessary and sufficient conditions for when causal effects transfer across distributions. Interpretability offers: Systematic edit failures as natural experiments revealing transportability violations. Research questions: 1. Can transportability criteria predict, prior to editing , which prompts or distributions an edit should generalise to? 2. Can editing objectives be designed to optimise for transportable effects rather than in-distribution success? 5 Conclusion Our position paper has argued that mechanistic interpretability and identifiable causal representation learning are mutually beneficial. First, we propose that the formal definitions of claims and estimands from causal inference and identifiability ( Section 2 ) can help clarify the warranted scope of interpretability methods and improve the comparability of their claims. Potential misinterpretations of claims arise from rung mismatch, identification gap, or both ( Section 2.3 )—and the diagnostic checklist ( Section G.4 ) helps practitioners identify which. Second, the causal lens extends rather than dismisses alternative views; for instance, extensive empirical robustness testing partially addresses deployment reliability, and our framework complements such testing by characterizing which distribution shifts are likely to cause failure, though prospective validation remains future work. Finally, we outline four concrete research directions where CRL and mechanistic interpretability can aid each other ( Section 4 ). Our hope is that grounding interpretability in causal inference will improve our understanding of what our methods can and cannot tell us. This will help bridge the gap to globally reliable control that safe deployment demands. Acknowledgements The authors thank Thomas Klein, Abhinav Menon for their valuable feedback on the manuscript. Dhanya Sridhar acknowledges support from NSERC Discovery Grant RGPIN-2023-04869, and a Canada-CIFAR AI Chair. Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program and thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for its support. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. This work was also supported by a grant from Coefficient Giving to Aaron Mueller. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. Lastly, we thank the organisers and participants of the Fourth Bellairs Workshop on Causality (McGill University Bellairs Research Institute, 14–21 February 2025) for facilitating connections among collaborators. Appendix A Notation and Glossary A.1 Formal Definitions: Affordances, Estimands, and Identifiability This appendix provides formal definitions for the key concepts introduced intuitively in Section 2 . A.1.1 Interventions Interventions are one important class of affordance. We consider interventions targeting inputs \mathbf{x} , activations \mathbf{a}^{(l)} , representations \mathbf{h}^{(l)} , or model parameters \theta ; denote the set of admissible targets v by \mathcal{V}\;:=\;\{\mathbf{x}\}\,\cup\,\{\mathbf{a}^{(l)}\}_{l=1}^{L}\,\cup\,\{\mathbf{h}^{(l)}\}_{l=1}^{L}\,\cup\,\{\theta\} denote the set of admissible intervention targets. For any such v\in\mathcal{V} , we write \mathrm{do}(v=v^{\prime}) to denote setting v to v^{\prime} ; e.g. clamping, activation patching, zero-ablating. More generally, \mathrm{do}(\mathcal{I}_{v}) would represent modifications such as steering, scaling, fine-tuning, which don’t fix the target to a specific value. This notation unifies input-, representation-, activation-, and parameter-level manipulations within a single calculus. Table 3 : Examples of intervention classes in LLMs and representative methods. Intervenable object Representative methods Inputs prompts, context Minimal pairs [ 100 ] \cdot Prompt perturbations [ 133 ] \cdot Jailbreak suffixes [ 160 ] \cdot Persona modulation [ 159 ] Activations layer states Activation patching [ 154 ] \cdot Causal mediation analysis [ 150 ] \cdot Ablations [ 48 ] Representations learned features SAE feature steering [ 145 ] \cdot Interchange interventions [ 46 ] Parameters weights ROME [ 103 ] \cdot MEMIT [ 104 ] \cdot ReFT–r1 [ 157 ] \cdot LoRA [ 61 ] A.1.2 Counterfactuals Counterfactual queries ask how the output would have differed under an alternative intervention, given what was actually observed. For an observed input–output pair (\mathbf{x},\mathbf{y}) and intervention target v\in\mathcal{V} , we write \mathbf{y}_{v\leftarrow v^{\prime}} (or, \mathbf{y}_{\mathcal{I}_{v}} ) to denote the counterfactual output that would have been produced had the intervention \mathrm{do}(v=v^{\prime}) (or, induced by \mathcal{I}_{v} ) been applied, while holding all other external factors fixed. A.2 Features This section consolidates the various meanings of “feature” used in mechanistic interpretability and causal representation learning, clarifying terminological ambiguities that can lead to confusion when bridging these communities. A.2.1 Two Notions of Feature Contemporary research on neural network interpretability operates with distinct notions of what constitutes a feature . Mechanistic interpretability typically considers a feature as any semantically meaningful, extractable piece of information encoded in model activations [ 41 , 20 ] . This operational definition carries no strong ontological commitment regarding the existence of latent generative factors or an underlying data-generating process (DGP). A.2.2 Feature Operationalisations in Mechanistic Interpretability In mechanistic interpretability, a feature may refer to several different objects, with the common theme being they are all internal intervenable units of the model. This terminological ambiguity reflects genuine disagreement about the right level of analysis for understanding neural networks: 1. Individual neurons : Early work studied when “individual neurons correspond to natural ‘features’ in the input” [ 70 ] , treating single units a_{i}^{(l)} as the atomic interpretable components. However, the prevalence of polysemantic neurons—where single neurons respond to multiple unrelated concepts—has challenged this view [ 41 ] . 2. Linear directions : The linear representation hypothesis formalises features as directions {\mathbf{v}} in activation space such that {\mathbf{v}}^{T}{\mathbf{a}} varies systematically with some property of interest [ 117 ] . This view underlies concept activation vectors [ 79 ] and activation steering [ 147 ] . 3. Dictionary elements : The dominant current usage refers to basis directions in the learned encoding space of a sparse autoencoder (SAE), motivated by the superposition hypothesis—that models represent more features than they have dimensions [ 41 , 20 , 33 ] . 4. Circuit components : In circuit-based analysis, features may refer to the nodes in sparse feature circuits—causally implicated subnetworks that explain model behaviours [ 96 ] . These different operationalisations carry different assumptions about what makes a unit interpretable. In contrast, causal representation learning (CRL) presupposes the existence of ground-truth latent factors \mathbf{z}\in{\mathbb{R}}^{d_{z}} that causally generate observations \mathbf{x}\in{\mathbb{R}}^{d_{x}} through some mixing function g:{\mathbb{R}}^{d_{z}}\to{\mathbb{R}}^{d_{x}} [ 137 , 89 ] . The central question in CRL is whether these latent factors can be identified —that is, recovered up to well-characterised equivalence classes—from observations alone or with auxiliary information such as temporal structure [ 63 , 64 ] , paired samples [ 90 ] , or multi-environment data [ 3 ] . A.3 Three Sources of Identifying Structure The strategies outlined in the main text—interventions, natural variation, and inductive biases—correspond to three recurring sources of identifying structure in the representation learning literature: Interventional data. Paired samples before and after perturbations, temporally resolved actions, or multiple environments with unknown targets can force competing explanations apart, identifying latent variables up to equivalence classes determined by the intervention family [ 39 , 121 , 19 , 87 , 153 , 149 ] . Natural variation. Natural variation provides similar leverage without explicit intervention: nonstationarity enables nonlinear ICA through segment discrimination [ 63 , 64 , 77 ] , while self-supervision and weak supervision exploit contrasts across timesteps, augmentations, or contexts [ 151 , 2 ] . Inductive biases. Inductive biases further constrain the equivalence class by ruling out families of solutions: independence assumptions [ 29 , 62 , 51 ] , sparsity constraints [ 84 ] , and functional restrictions on the mixing process [ 78 , 134 ] . A.3.1 Unifying Perspectives The Identifiable Exchangeable Mechanisms (IEM) framework [ 131 ] provides a unifying perspective on these sources of variation, distinguishing two complementary forms: cause variability , where the distribution of inputs or latent sources varies across contexts while the mechanism mapping them to outputs remains fixed, and mechanism variability , where inputs are held constant while the generative mechanism changes. In practice, identifiability results typically combine these sources—e.g. temporal structure plus sparsity [ 64 , 87 ] , weak supervision plus independence [ 90 , 2 ] , or interventions plus mechanism constraints [ 19 , 84 ] —and the resulting equivalence class reflects the joint constraints imposed by all available affordances. Appendix B Philosophical Commitments B.1 The Metaphysics of Affordance-based Identifiability A common (often implicit) premise in identifiable (causal) representation learning is that there exists a privileged latent description of the world, an “ideal reality” given by a true set of generative factors and that a learned representation should recover it, up to a narrow equivalence class (typically permutation and rescaling). Such a representation is called platonic , harking back to Plato’s theory of “forms” [ 122 , 82 ] —eternal and changeless entities paradigmatic of the nature of the perceived world. Structuralist traditions motivate a different stance. Rather than treating latent variables as privileged ontological objects whose coordinates we only imperfectly recover, structuralism holds that scientific knowledge concerns relations, invariants, and transformations , not things-in-themselves. On this view, the epistemic content of a theory lies in the structure it preserves across admissible representations, while the nature of the underlying entities remains either inaccessible or underdetermined. This position has appeared in multiple guises. In the philosophy of science, 123 argues that while ontological posits may change across scientific revolutions, certain relations (expressed through equations, symmetries, and invariants) persist; this privileges epistemology over ontology, and suggests that stable knowledge resides in structure rather than in a unique set of underlying entities. In mathematics, the Bourbaki programme [ 18 ] formalised structuralism by defining objects only up to isomorphism: groups, spaces, and algebras are not collections of elements with intrinsic identity. In philosophy, Kant’s critical realism [ 75 ] similarly denies access to noumena, or the world as it is., while affirming that stable structure can nevertheless be known through forms of interaction. Read through this lens, one can argue that the practice of identifiability already aligns more closely with structural realism than with a strong Platonic metaphysics. Identifiability results do not recover a unique latent state of the world; they recover an equivalence class under a transformation group that preserves specified relations. Permutation, rescaling, and invertible reparameterisations are not technical pathologies but explicit acknowledgements that only certain aspects of a representation are empirically meaningful. The identifiable object is not a latent variable per se , but the structure defined by its invariants. The apparent Platonic commitment enters only through interpretation. Latent variable models are often narrated as approximations to a true generative process, giving rise to the impression that identifiability concerns proximity to an underlying set of hidden entities. A structuralist reading weakens this ontological claim without weakening the theory. The equivalence class need not correspond to “the true ground-truth generative factors”; it need only faithfully represent the relational structure that is recoverable from data and stable across environments. In this sense, identifiability is best understood as a statement about what structure is invariantly identifiable, not about what entities exist . Affordances [ 49 ] provide a concrete operationalization of this structuralist view. In Gibson’s ecological psychology, an affordance is not a property of an object in isolation but a relation jointly defined by an agent and its environment: what actions the pairing makes possible. What an object is , functionally speaking, is inseparable from what it affords . Translating this to representation learning: a latent dimension is meaningful insofar as it parameterises stable affordances—predictable, reproducible changes in observations under manipulation. Identifiability, then, concerns the recovery of coordinates that preserve these affordances, not the recovery of hidden objects independent of any interaction. Under this view, the goal of identifiable representation learning shifts. Rather than asking how closely a learned representation approximates presumed ground-truth factors, we ask whether it preserves the same affordance structure under admissible transformations. The key insight is threefold: (i) affordances define which variations are observable or inducible; (ii) these variations break equivalence classes among candidate representations such that two representations that behave identically under all afforded interactions are, for our purposes, equivalent;(iii) identifiability characterises precisely which structure survives this winnowing. Hence a representation is structurally sufficient if it supports all affordance-based distinctions the analyst can induce, even without recovering any unique “true” latent variables. Identifiability thus becomes a theory of structural sufficiency—it characterises the maximal structure that can be stably recovered while remaining agnostic about deeper ontological commitments. This reframing dissolves a persistent confusion. Latent variables are not the necessary content of ICA or related models; they are convenient coordinates for representing the recovered structure. Just as Cartesian and polar coordinates describe the same geometric relations, different latent parameterizations may describe the same affordance structure. This addresses a central question: what does identifiability mean when privileged ground truth is never accessible? It specifies which structural distinctions are recoverable from the interactions available. From this perspective, the affordance set is induced by the experimental conditions under which data are generated: auxiliary variables, multiple environments, temporal structure, or admissible interventions. We need not presuppose a ground-truth generative process; the affordances themselves define what can be learned. This motivates affordance-based identifiability : assess a representation not by axis-aligned recovery of latent coordinates, but by the invariances and interventional queries it supports. B.2 Pragmatism Core idea. Pragmatism is a philosophical tradition—developed by 68 and 36 —that evaluates ideas primarily by their practical consequences rather than by correspondence to abstract truth. On this view, the meaning of a concept is inseparable from the difference it makes in practice: if two theories yield identical predictions and interventions, they are pragmatically equivalent, regardless of their metaphysical commitments. ML context. When interpretability researchers ask “does this explanation help us predict model failures?” or “can we use this feature to steer behavior?”, they are implicitly adopting a pragmatist stance. A circuit explanation that enables reliable intervention is pragmatically valuable even if we cannot prove it reflects the model’s “true” computation. The challenge is that pragmatism without discipline can become an excuse for vagueness—claiming success on easy proxies while avoiding harder questions about generalization. B.3 Constructive Empiricism Core idea. Constructive empiricism, articulated by 11 , holds that science aims to produce theories that are empirically adequate —that “save the phenomena” by correctly predicting observable outcomes—without necessarily claiming to describe unobservable reality as it truly is. We can accept a theory as useful without believing it literally describes hidden mechanisms. ML context. Consider a sparse autoencoder (SAE) that decomposes activations into interpretable features. A constructive empiricist would say: we can use these features for prediction and control without claiming they represent the model’s “real” internal concepts. The SAE is a useful instrument for organizing our observations, not necessarily a window into ground truth. This perspective is liberating (we need not solve the “what are concepts really?” question) but also demands honesty about what our tools actually deliver. B.4 Underdetermination Core idea. Underdetermination refers to situations where multiple, mutually incompatible explanations are equally consistent with all available evidence [ 127 ] . No amount of data can uniquely determine which explanation is correct—additional assumptions or constraints are required to break the tie. ML context. This problem pervades interpretability. A linear probe achieving 95% accuracy on sentiment classification does not uniquely identify “the sentiment direction”—many directions may achieve similar performance, and the probe may exploit correlates rather than causes. Similarly, multiple circuit hypotheses may explain the same input-output behavior. Underdetermination is not a failure of method but a structural feature of inference from finite evidence. The remedy is not to ignore it but to explicitly characterise the equivalence class of solutions and understand what additional evidence (e.g. interventions, distribution shifts) could narrow it. B.5 Ontological Faithfulness vs. Instrumental Adequacy Core idea. These terms distinguish two standards for evaluating explanations. Ontological faithfulness asks whether an explanation accurately describes what is “really there”—the true structure, mechanisms, or entities underlying a phenomenon. Instrumental adequacy asks only whether the explanation serves its intended purpose—enabling prediction, control, or communication—without commitment to ontological accuracy [ 124 ] . ML context. When we identify a “deception feature” in an LLM, are we discovering something the model genuinely represents, or constructing a useful fiction that helps us predict and steer behavior? Ontological faithfulness would require the feature to correspond to some real computational primitive; instrumental adequacy requires only that interventions on this feature reliably produce the desired effects. Much interpretability research implicitly claims ontological faithfulness (“the model has this concept”) while only demonstrating instrumental adequacy (“this probe predicts this label”). Being explicit about which standard we are meeting helps calibrate the strength of our claims. Literature evidence. The gap between these standards is well-documented. 57 introduced “control tasks”—random label assignments that probes can nonetheless fit—demonstrating that high probe accuracy does not entail that representations encode the probed property; the probe may simply memorise. 40 showed that probing accuracy is not correlated with task importance: properties easily extracted by probes may not be used by the model, calling for “increased scrutiny of claims that draw behavioral or causal conclusions from probing results.” More recent work on sparse autoencoders (SAEs) reveals analogous concerns: 43 find substantial “dark matter”—unexplained variance that SAE features fail to capture—questioning whether SAE features constitute the model’s “true” computational primitives. 26 show that SAEs trained on LLMs suffer from “feature hedging,” merging correlated features in ways that destroy the monosemanticity they are meant to provide. Example. Consider the common claim that “BERT encodes part-of-speech information.” A linear probe trained on BERT’s hidden states achieves > 97% accuracy at predicting POS tags [ 58 ] . This is often interpreted ontologically—as if BERT internally represents parts-of-speech as a computational primitive. However, 57 showed that probes with similar capacity can achieve high accuracy on random control labels that carry no linguistic content, revealing substantial probe memorization. More strikingly, 40 demonstrated that removing the information exploited by POS probes via causal intervention (“amnesic probing”) had minimal effect on BERT’s language modeling performance—suggesting that while the information is extractable (instrumental adequacy), it may not be used by the model’s actual computation (calling into question ontological faithfulness). The probe succeeds, but claiming BERT “has” POS representations in any strong sense overstates what the evidence supports. B.6 Transportability Core idea. Transportability, formalised in causal inference by 119 and 10 , concerns when causal relationships learned in one setting (population, environment, distribution) remain valid in another. A causal effect is transportable if we can formally justify applying it to a new context, accounting for differences between source and target. ML context. Suppose we discover a steering vector that induces honesty in GPT-4 on a particular distribution of prompts. Does this vector work on different prompt types? On other models? In deployment conditions not seen during development? Transportability formalises these questions. A steering intervention that only works in-distribution has limited practical value; one that transports across contexts provides robust control. The causal inference literature provides tools for reasoning about when and why transport succeeds or fails—tools that interpretability research has largely not yet adopted. Appendix C Technical Review C.1 Background: Identifiability Results from Representation Learning This section provides technical background on identifiability results. These results formalise how different sources of variation—interventions, natural distribution shifts, and inductive biases—constrain the equivalence class of recoverable latent structures. C.1.1 Reinterpreting Classical Identifiability Classically, identifiability results are stated with respect to latent “ground-truth” variables that generate observed data. We adopt a different reading: ground-truth variables are not the ontologically privileged entities that exist independently in the observations. Rather, they are formal reference variables used to specify which distinctions are, in principle, recoverable from a given interface—i.e. from the affordances (variations and assumptions) available to an interpreter. On this reading, identifiability characterises an equivalence class of internal structures [ 51 , 84 , 2 , 52 , 131 ] consistent with those affordances, not a unique ontology of concepts. C.2 Background: Reconciling Mechanistic Interpretability and Causal Representation Learning The term “feature” carries different connotations in mechanistic interpretability and causal representation learning; see Section A.2 for a detailed discussion of these two notions and how they can be reconciled. C.2.1 The DGP as a Specification of Interestingness One reconciliation of these perspectives, following 83 , is to view the DGP assumption not as a metaphysical claim about platonic reality, but as a mathematical specification of what makes features interesting —a position that aligns with the structuralist reading of identifiability developed in Section B.1 . Different choices of DGP correspond to different criteria for feature quality: • Independence: Classical ICA [ 62 ] defines interesting directions as those maximizing statistical independence of the recovered components. • Variation across contexts: Paired-sample and multi-view approaches [ 90 , 152 ] identify features that vary across paired observations while others remain fixed. • Intervention-sensitivity: Causal approaches [ 3 , 21 ] privilege factors whose distributions shift under interventions or across environments. Under this view, identifiability theory provides mathematical precision to the otherwise vague notion of “semantically meaningful features” commonly invoked in mechanistic interpretability. C.2.2 ICA Without Latent Variables: A Philosophical Note An important observation, due to 62 , is that ICA admits formulations that do not posit latent variables at all. One can define the ICA problem purely in terms of finding “maximally independent” projections of observed data, without reference to an underlying generative process. In the linear case, the equivalence between the generative and projection-based formulations is well-established; in the nonlinear case, analogous results hold under suitable definitions of independence [ 77 , Appendix F] . This distinction parallels a broader divide between latent variable models—which assume unobserved generative factors—and energy-based models, which make no such commitments and are, in a sense, “assumption-free” [ 85 ] . However, the latent variable formulation naturally admits a causal interpretation: the standard ICA equation \mathbf{x}=A\mathbf{s} should be read as an assignment \mathbf{x}:=A\mathbf{s} (in the sense of structural causal models), indicating that latent sources causally generate observations rather than merely being correlated with them [ 121 ] . C.2.3 Implications for Mechanistic Interpretability Grounding interpretability methods in identifiability theory offers several potential benefits: Consistency across methods. Even without commitment to a “true” DGP, identifiability results can address whether different interpretability methods—e.g. sparse autoencoders with varying architectures [ 33 ] , linear probes [ 4 ] , or distributed alignment search [ 47 ] —converge on equivalent feature representations. This question of inter-method consistency is distinct from, and arguably more tractable than, the question of whether any method recovers ground-truth factors [ 134 ] . Operationalising semantic meaning . Mechanistic interpretability literature frequently appeals to features being “semantically meaningful” or “human-interpretable” without formal criteria for these properties. Identifiability theory can provide such criteria—statistical independence, intervention-sensitivity, or invariance across domains—that operationalise interestingness in falsifiable terms. Downstream guarantees. A causal DGP assumption may yield guarantees that purely statistical notions cannot provide. Recent empirical work has documented failures of sparse autoencoder features to transfer across distribution shifts [ 94 ] or to support reliable steering interventions [ 35 ] , suggesting that features identified without causal grounding may lack the robustness required for safety-critical applications. C.2.4 Abstraction and Selection in Learned Representations Both the specification of a DGP and the learning process itself can be understood as performing two complementary operations [ 135 , 12 ] : 1. Abstraction: Selecting the level of detail at which to represent the world (e.g. modeling objects as point masses versus textured 3D entities). 2. Selection: Discarding factors deemed irrelevant to the task at hand (e.g. ignoring friction in a mechanics problem). Under this framing, mechanistic interpretability can be viewed as the problem of reconstructing the learned DGP implicit in a trained model’s representations. Crucially, this learned DGP may differ from any “true” external DGP due to inductive biases, architectural constraints, or training dynamics—a form of model misspecification that interpretability methods must contend with. C.3 The causal ladder for interpretability queries Pearl’s causal ladder [ 121 ] distinguishes three rungs of evidence: association (L1), intervention (L2), and counterfactual (L3). This hierarchy provides a principled framework for classifying what interpretability methods can actually establish. In practice, interventions such as activation patching or steering do not simply “change a variable”, but also implicitly change how the rest of the network processes it. Consequently, the observed effect may reflect properties of the intervention procedure itself (e.g. where the patch is taken from, how it is injected, or how strongly it is applied), rather than the effect of the variable in isolation. Appendix D provides a comprehensive reference that maps interpretability queries to their ladder rungs ( Table 4 ), with detailed examples illustrating how to locate your analysis on the hierarchy. Appendix D The Causal Ladder: A Unified Reference This section provides a comprehensive reference for Pearl’s causal ladder [ 121 ] as applied to mechanistic interpretability. For technical background on identifiability results and the reconciliation of mechanistic interpretability with causal representation learning, see Section C.1 . We first present a systematic mapping of interpretability queries to their ladder rungs ( Table 4 ), then provide detailed guidance for locating your own analyses on this hierarchy. D.1 Interpretability Queries Across the Ladder Table 4 maps common interpretability questions onto the three rungs of Pearl’s ladder. Each rung corresponds to a distinct type of evidence: association (L1), intervention (L2), and counterfactual (L3). The table organises queries by what is being examined—inputs, activations, representations, or parameters—and illustrates representative methods at each level. Table 4: Interpretability queries across Pearl’s causal ladder, expressed in terms of association, intervention, and counterfactuals. Rung of the Ladder Query Distribution Practical Query (with Examples) 1: Association p({\mathbf{y}}\mid{\mathbf{x}}) Prediction: Given an input prompt {\mathbf{x}} , what output tokens {\mathbf{y}} are likely? (Standard next-token prediction, logit inspection) p({\mathbf{h}}\mid{\mathbf{x}}) Encoding: Given an input {\mathbf{x}} , what internal representations or features {\mathbf{h}} are typically activated? (Activation logging, SAE feature attribution, probing) p({\mathbf{y}}\mid{\mathbf{h}}) Decoding: Given an observed internal trace {\mathbf{h}} , what outputs {\mathbf{y}} are likely? (Linear probes, sparse decoding, feature-to-token analyses) 2: Intervention p({\mathbf{y}}\mid\mathrm{do}({\mathbf{x}}={\mathbf{x}}^{\prime})) Input intervention: If I change the prompt from {\mathbf{x}} to {\mathbf{x}}^{\prime} , how does the output change? (Minimal pairs, prompt perturbations, jailbreak tests) p({\mathbf{y}}\mid\mathrm{do}({\mathbf{a}}^{(l)}={\mathbf{a}}^{\prime})) Activation intervention: If I overwrite or patch activations at layer l , how does the output change? (Activation patching, causal tracing, ablations) p({\mathbf{y}}\mid\mathrm{do}({\mathbf{h}}={\mathbf{h}}^{\prime})) Representation intervention: If I manipulate a feature or direction in representation space, what behavior changes? (SAE feature steering/ablation, DAS interchange interventions) p({\mathbf{y}}\mid\mathrm{do}(\theta=\theta^{\prime})) Parameter intervention: If I edit the model’s weights, does the model’s behavior change as intended? (LoRA etc) 3: Counterfactual {\mathbf{y}}_{{\mathbf{x}}\leftarrow{\mathbf{x}}^{\prime}} Input counterfactual: Given this specific run, would the model’s output have differed had the prompt been {\mathbf{x}}^{\prime} instead of {\mathbf{x}} ? (Prompt counterfactual analysis, contrastive decoding) {\mathbf{y}}_{{\mathbf{a}}^{(l)}\leftarrow{\mathbf{a}}^{\prime}} Activation counterfactual: For this exact generation, would the output have changed if activations at layer l had been different? (Counterfactual activation patching, causal scrubbing-style analyses) {\mathbf{y}}_{{\mathbf{h}}\leftarrow{\mathbf{h}}^{\prime}} Representation counterfactual: Would this behavior still have occurred if a specific internal feature had been absent or altered? (Feature-level necessity tests) {\mathbf{y}}_{\theta\leftarrow\theta^{\prime}} Model counterfactual: Would this same input have produced a different output if the model’s parameters had encoded different knowledge? D.2 Locating an Interpretability Method on the Causal Ladder The following diagnostic questions help practitioners identify what rung their evidence actually occupies. Each rung is illustrated with a concrete example, specifying what the evidence licenses and what it does not. L1: Association—Is the system manipulated, or only observed? If the method only computes statistics from forward passes—e.g. probing, PCA, feature attribution—the evidence is associational (L1) evidence, even if the summary is interpreted causally. Example: Linear probing for sentiment. Suppose we train a linear classifier on layer-12 activations to predict sentiment labels. The probe achieves 92% accuracy on held-out data. (i) Setup: Collect activations \{\mathbf{a}^{(12)}_{i}\} from forward passes on sentiment-labeled text. (ii) Method: Fit logistic regression \hat{y}=\sigma(\mathbf{w}^{\top}\mathbf{a}^{(12)}+b) to predict positive/negative. (iii) Result: High accuracy shows sentiment information is decodable —an external classifier can extract it. What L1 licenses: “Sentiment is linearly decodable from layer 12.” What L1 does not license: “The model represents sentiment at layer 12” or “Layer 12 computes sentiment.” Decodability \neq encoding; the information may be present but unused by the model’s own computation. L2: Intervention—Is the manipulation externally controlled? If the method modifies the forward pass—clamping an activation, patching from a different input, ablating a component, adding a steering vector—and measures the downstream effect, the evidence is interventional (L2). The key is that the modification is set by the experimenter, not determined by the natural computation. Example: Activation patching for indirect object identification. We investigate whether a specific attention head mediates the IOI task (completing “Mary gave a book to…” with the indirect object). (i) Setup: Run the model on prompt A (“Mary gave a book to John. Mary gave a pencil to…”) and record activations. Run on corrupted prompt B where names are swapped. (ii) Intervention: At head 9.6, replace the activation from run A with the activation from run B : \mathbf{a}^{(9.6)}_{A}\leftarrow\mathbf{a}^{(9.6)}_{B} . (iii) Measurement: Observe whether the output changes from “John” toward “Mary.” (iv) Result: If patching head 9.6 substantially changes the output, we have evidence that this head causally contributes to the IOI behavior under this intervention. What L2 licenses: “Intervening on head 9.6 changes IOI behavior under these tested conditions.” What L2 does not license: “Head 9.6 is the IOI mechanism” (sufficiency \neq necessity; other paths may exist) or “This finding will hold on all IOI-like prompts” (requires transportability testing). L3: Counterfactual—Is the query anchored to a specific observation? Counterfactual claims reason about the same instance under an alternative intervention: “for this input that produced {\mathbf{y}} , what would the output have been had {\mathbf{a}}^{(l)} taken value {\mathbf{a}}^{\prime} ?” This demands not just an intervention, but a model of how latent variables would have responded—requiring structural assumptions beyond what patching alone provides. Example: The Eiffel Tower counterfactual. An LLM processes “The Eiffel Tower is in” and outputs “Paris.” We observe activations \{\mathbf{a}^{(l)}\} throughout. The counterfactual question is: what would the model have output if \mathbf{a}^{(5)} had taken the value it takes when processing “The Colosseum is in”? Operationally, counterfactual reasoning follows the abduction–action–prediction recipe [ 121 ] : (i) Abduction: Condition on the observed run—input, output, and the full trace \{\mathbf{a}^{(l)}\} —to infer what latent or exogenous factors explain this particular forward pass. In our example: what about the model’s internal state, beyond the activations we recorded, determined that “The Eiffel Tower is in” \to “Paris”? (ii) Action: Intervene on the target variable. Here, set \mathbf{a}^{(5)}\leftarrow\mathbf{a}^{\prime(5)} , the activation from the Colosseum prompt. (iii) Prediction: Propagate forward under the intervention, holding fixed the exogenous factors inferred in step (i), to compute the counterfactual output \mathbf{y}_{{\mathbf{a}}^{(5)}\leftarrow{\mathbf{a}}^{\prime(5)}} . What L3 licenses: “For this specific forward pass, had \mathbf{a}^{(5)} taken the Colosseum value, the model would have output ‘Rome.’ ” What L3 does not license: Generalisations to other inputs (requires separate analysis or transportability assumptions) or mechanism-level claims like “layer 5 encodes location.” Crucially, L3 requires a structural causal model specifying how latent variables interact—not merely the ability to intervene. Standard activation patching performs step (ii) but lacks the structural model needed for steps (i) and (iii). This is why most “counterfactual” claims in interpretability are actually L2 claims narrated counterfactually. Table 1 maps the method families from Appendix E onto this hierarchy, contrasting the question each method is typically recruited to answer with the question its evidence can actually support. A recurring pattern emerges: interpretability claims often climb the ladder without the requisite evidence. Most interpretability methods that manipulate internal states license L2 evidence [ 150 , 42 , 23 , 148 ] . Achieving L3 generally requires stronger modeling assumptions about latent variables and mechanisms [ 45 ] . As a result, claims framed counterfactually—e.g. “this feature caused the output” or “the model would not have produced y without this component”—often rest on L1 or L2 evidence, thereby inflating the rungs. Appendix E Implicit Estimands in Interpretability Methods We examine five method families, identifying for each: the question the method is typically recruited to answer, the quantity it actually estimates, and the gap between the two. We have also used identifiability as a diagnostic lens, asking whether a method’s evidence pins down its claimed quantity. Table 5 provides a summary; we elaborate below. Table 5 : Interpretability method families and the implicit estimands they target. Method family Representative methods Implicit estimand Evidence type Feature attribution saliency / scores Saliency \cdot Grad \times Input \cdot Integrated Gradients \cdot SHAP \cdot LIME Local input–output sensitivity (at a given input and parameterization); marginal contribution relative to an analyst-defined baseline or perturbation distribution Gradients / synthetic perturbations Diagnostics / probing readout-based Linear probes \cdot CAVs \cdot Logit lens Existence of a decision function in a restricted probe class that predicts a concept from internal states; correlational alignment with external labels Supervised labels / correlations Causal / circuit-based interventions Activation patching \cdot Causal tracing \cdot Circuit analysis Interventional effect of manipulating internal states (e.g. replacing \mathbf{a}^{(l)} ) on downstream outputs Controlled interventions Structured decomposition basis learning SAEs \cdot PCA \cdot Dictionary learning Factors induced by an optimization objective (e.g. reconstruction with sparsity or low-rank constraints); structure defined up to objective-invariances Reconstruction objective Theory-grounded identifiable structure DAS \cdot SSAEs Recovery up to an equivalence class under stated identifiability assumptions (from interventions, variation, and inductive bias) Intervention + inductive bias Feature Attribution. Saliency maps [ 141 ] , Gradient \times Input [ 140 ] , Integrated Gradients [ 143 ] , SHAP [ 92 ] , and LIME [ 132 ] are framed as answering a counterfactual question: which input features are responsible for this output? [ 17 ] . In practice, these methods estimate local sensitivity—the rate of change of output under input perturbations—via gradients [ 141 , 5 ] or surrogate fits [ 132 ] . SHAP targets Shapley values over feature coalitions, a distinct quantity requiring marginalization under analyst-defined baselines. The evidence is associational throughout: no intervention on the data-generating process occurs. Failure modes follow predictably: saliency maps remain stable under parameter randomization [ 1 ] , attributions shift under semantics-preserving transformations [ 80 ] , and SHAP values distribute arbitrarily among correlated features [ 105 ] . Probing and Diagnostic Methods. Linear probes [ 58 , 4 ] , concept activation vectors [ 79 , TCAVs;] , and the logit lens [ 113 ] ask whether a concept is encoded in the model’s representations. What they estimate is narrower: the existence of a linear decision boundary separating concept–positive from concept–negative examples in activation space. Thus, two models are indistinguishable if they admit the same linear separator, regardless of whether either model actually uses the concept downstream. The equivalence class thus contains both models that genuinely encode the concept (it plays a causal role in computation) and models where the concept is merely recoverable as a correlate of structure the model uses for other purposes. The claimed quantity—whether the concept is represented—is not identified; only decodability is. Breaking this equivalence class requires interventional evidence: if we intervene on the concept representation and observe its downstream effects, we move from asking “can we decode this concept?” to “does the model use it?”. Causal and Circuit-Based Methods. Activation patching [ 42 ] , causal tracing [ 103 ] , and circuit analysis [ 115 , 30 ] aim to identify which components uniquely mediate behavior. The estimand—the causal effect of replacing an activation under \mathrm{do}(\mathbf{a}^{(l)}=\mathbf{a}^{\prime(l)}) —is genuinely interventional [ 150 , 120 ] . The affordance family \mathcal{U} now includes controlled interventions. However, multiple circuits can produce identical patching effects: the observation that patching component C restores behavior does not rule out alternative components C^{\prime} that would do the same. Under \mathcal{U} , models with different “true” mediating structure are indistinguishable if they respond identically to the interventions performed. Existence of a sufficient path is identified; uniqueness and necessity are not [ 158 , 23 ] . Structured Decomposition. Sparse autoencoders [ 33 , 20 , SAEs;] , PCA [ 71 ] , and dictionary learning [ 116 ] ask: what are the fundamental units of representation? The estimand is a factorisation minimising reconstruction error under structural constraints—sparsity, orthogonality, low rank. Thus any such factorisation satisfying the constraint lies in the same equivalence class, often evaluated by assessing the value of the reconstruction loss. The “true features” of the representation are not identified—we recover what the objective and its inductive biases reward [ 89 ] . Sparse factors need not be disentangled, causally operative, or concept-aligned; they are simply one valid solution among many equivalent alternatives. Theoretically Grounded Methods. Distributed alignment search [ 47 , DAS;] and identifiable sparse autoencoders [ 73 , SSAEs;] differ from prior families by reasoning about equivalence classes explicitly. DAS posits a hypothesis consisting of a high-level causal model and an alignment from its variables to distributed activation subspaces, and tests whether this hypothesis reproduces interventional behavior under a specified intervention family (often interchange interventions). [ 47 , 46 ] SSAEs impose a sparse generative model on representation differences and derive identifiability guarantees under sufficient support-variability conditions, thereby shrinking the admissible equivalence class to the benign indeterminacies characterized by the theorem (e.g. permutations and related symmetries) [ 73 ] . These methods represent genuine progress: the estimands are stated, and the residual equivalence class within which recovery holds is characterized. Crucially, however, these frameworks remain interventional (L2) rather than counterfactual (L3). Although DAS introduces a structural hypothesis and SSAEs identify coordinates that are interpretable as factors, neither supplies the full abduction–action–prediction semantics required for counterfactual queries—specifically, a justified notion of which exogenous quantities are held fixed for an individual unit. [ 121 ] Consequently, their guarantees should be read as strengthening L2 claims (interventional invariances and compatibility tests) by reducing underdetermination, not as identifying counterfactual outcomes. Appendix F Taxonomy for Characterising Intepretability Claims Decodability \neq Encoding. A linear probe g predicts a concept c from \mathbf{h} with high accuracy; the finding is reported as “the model encodes c .” Evidence (L1). Probe success establishes that some direction in \mathbf{h} predicts \mathbf{c} : identified \displaystyle\exists g\in\mathcal{G}:\ \mathbb{E}_{p}[\mathbf{1}\{g(\mathbf{h})=c\}]\geq 1-\epsilon. Gap. Many probes \in\mathcal{G} can achieve the same accuracy while picking out different directions; predictive success alone does not tell us which direction (if any) the model actually uses [ 130 , 40 ] . What would resolve it (L2). Intervene on candidate directions and measure behavioral change: asked-for \displaystyle p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}_{\parallel c}:=\tilde{\mathbf{h}}))\neq p(\mathbf{y}), where \mathbf{h}_{\parallel c} denotes the component of {\mathbf{h}} aligned with c . Pinning down which direction requires additional constraints such as sparsity, independence, etc. Takeaway. Probing shows c is in \mathbf{h} ; encoding requires knowing where , and that the model uses it. Local Sensitivity \neq Global Importance. A saliency map highlights token i as “important” at input \mathbf{x}_{0} ; the result is narrated as a claim about how the model generally behaves. Evidence (L1, pointwise). Gradient methods measure sensitivity at a single input for a chosen parameterisation and baseline: identified \displaystyle\nabla_{\mathbf{e}_{i}}f(\mathbf{x})\big|_{\mathbf{x}_{0}} or \displaystyle f(\mathbf{x}_{0})-f(\mathbf{x}_{0}^{\setminus i}). Gap. The quantity depends on arbitrary choices—baseline, parameterisation, perturbation scheme—that are not properties of the model; different choices yield different attributions [ 1 , 143 , 5 ] . Even if local sensitivity were uniquely defined, pointwise evidence does not establish distributional claims. What would resolve it (L2, distributional). Specify an intervention family and aggregate over a reference distribution: asked-for \displaystyle\mathbb{E}_{\mathbf{x}\sim p}[f(\mathbf{x})-f(\mathbf{x}^{\setminus i})], with robustness checks over intervention and baseline choices. Takeaway. Saliency measures local, choice-dependent sensitivity; global importance requires distributional aggregation over explicit interventions , and acknowledging dependence on these choices. Existence \neq Uniqueness. A circuit discovery method finds a subgraph S whose intervention flips a behavior; the result is reported as “ the mechanism.” Evidence (L2). The method shows that some sufficient subgraph exists: identified \displaystyle\exists\,S:\ p(\mathbf{y}\mid\mathrm{do}(\mathbf{h}_{S}:=\tilde{\mathbf{h}}_{S}))\neq p(\mathbf{y}). Gap. Calling it “ the mechanism” implies uniqueness (up to a stated equivalence \sim ): asked-for \displaystyle\exists!\,S\ \text{(mod }\sim) . This is an existence result, not an identification result. Multiple subgraphs may produce the same effect—redundant pathways, distributed implementations, or equivalent parameterisations. The method identifies one member of an equivalence class of sufficient circuits; which one it returns may depend on initialisation, optimisation path, or arbitrary tie-breaking. (Contrast: sufficiency \neq necessity is a rung error—necessity is a counterfactual L3 claim; here we remain at L2 but lack uniqueness.) What would resolve it. Characterise the equivalence class itself, or introduce constraints that select a unique representative: multi-site interventions that discriminate between candidates, minimality criteria, or a causal abstraction specifying primitives. Takeaway. Circuit discovery finds a sufficient subgraph; “the mechanism” implies uniqueness. Without ruling out alternatives, treat discoveries as intervention handles, not unique explanations. Alignment \neq Ontological Identity. A feature is labelled “honesty” because it aligns with annotations under a contrast distribution \pi ; under a different contrast, the same feature aligns with a different label. Evidence. Alignment is established relative to a contrast set: identified \displaystyle z\text{ aligns with }c\text{ under }\pi, operationalised via predictability, separability, or correlated intervention effects. Gap. Treating the label as ontological implies contrast-invariant identity: asked-for \displaystyle z\text{ represents }c\text{ independently of }\pi. The label is not intrinsic to z —it is a property of the pair (z,\pi) . When concepts co-vary under \pi , the data are consistent with multiple labellings: “honesty” and “formality” may be indistinguishable if they correlate on the contrast set. Change \pi , and the same z may align with a different concept [ 25 ] . What would resolve it. Test alignment under multiple contrast sets to check invariance; use supervision that breaks co-variation; or specify an explicit structural model defining which latent variable the label targets. Takeaway. Alignment is contrast-indexed, not ontological. Report (z,c,\pi) together, and treat labels as hypotheses unless stable across distributional variation. Subspace \neq Coordinates. An analysis identifies a low-dimensional subspace \mathcal{S} associated with a behaviour; the writeup assigns semantics to individual directions within it. Evidence. The method identifies the subspace as an equivalence class (invariant under rotations within \mathcal{S} ), not a particular basis. Writing \Pi_{\mathcal{S}} for projection onto \mathcal{S} : identified \displaystyle\exists\mathcal{S}\subset\mathbb{R}^{d_{h}}\ \text{s.t.}\ \Pi_{\mathcal{S}}(\mathbf{h})\ \text{predicts behaviour}. Gap. The estimand is identifiable only up to the subspace —i.e. up to the symmetry group that leaves \Pi_{\mathcal{S}} unchanged. Coordinates within \mathcal{S} are not identified: any rotated basis also yields identical subspace-level evidence. Assigning semantics to directions therefore assumes coordinate-level identification: asked-for \displaystyle\text{the axes in }\mathcal{S}\text{ are identifiable} What would resolve it. Constraints that select a unique basis: sparsity, statistical independence, non-negativity, or supervised anchors tying specific directions to labeled concepts. Takeaway. Report the level at which structure is identified. Subspace claims are licensed; coordinate semantics require axis-identifying assumptions. Appendix G Pilot Study: Calibrating Claim Language to Evidential Strength We conducted a pilot study to measure the distance between claim language and method strength in mechanistic interpretability papers, using Pearl’s causal ladder as a shared ruler. This appendix summarizes the methodology, findings, and implications for a larger study. G.1 Methodology Paper sampling. We annotated 50 papers comprising 186 claims, drawn from major ML and NLP venues (NeurIPS, ICLR, ACL, EMNLP, AAAI, TMLR) and their workshops, plus arXiv preprints (2021–2026). Papers were selected to span major mechanistic-interpretability method types, including circuit discovery, knowledge localisation, SAE analysis, steering vectors, and evaluation benchmarks; selection was not formally stratified. We treat author attribution with care: the full list of annotated papers is not published, and the annotation dataset will be shared upon request after a review period allowing authors to examine their paper’s annotations. We name two papers as positive calibration anchors in Section G.4 because their tight claim–method alignment offers concrete models for the field; in all other cases, papers are identified only by aggregate statistics or anonymous pattern descriptions. LLM-driven annotation. Annotations were performed using an LLM (Claude Opus 4.5, Anthropic) with human oversight. The LLM operated within an interactive coding environment (Claude Code) with access to arXiv API tools for paper retrieval and search. For each paper, the LLM: (1) read the full paper text via arXiv API, (2) extracted verbatim claim text and identified its location, (3) classified method and claim rungs following the codebook criteria described below, and (4) computed gap scores. The annotation was guided by a structured codebook that the LLM accessed during each session. The codebook specified: • Field definitions for each annotation column (claim text, location, prominence, method rung, claim rung, gap score, confidence, replication status) • Method-to-rung mappings as listed below • Linguistic markers for claim rung classification (see below) • Edge-case decision rules : hedged claims (“may encode”) were coded at the underlying claim’s rung with reduced confidence; implicit claims from narrative framing were coded but weighted lower; for multi-method papers, each claim was coded against the method that directly supports it • Confidence scoring on a 1–5 scale reflecting annotator certainty in rung assignments Calibration. Five papers served as calibration anchors, spanning circuit discovery, knowledge localisation, algorithm analysis, SAE evaluation [ 27 ] , and production probing [ 81 ] . For each calibration paper, detailed rationales documented the reasoning behind method and claim rung assignments, including common rung-elevation patterns (e.g., definite articles implying uniqueness beyond what patching establishes). These worked examples were available to the LLM during annotation of subsequent papers to promote consistency. The two anchors with tight claim–method alignment are named in Section G.4 ; the three exhibiting rung-elevation patterns are presented as anonymous field-level illustrations. Human oversight and fact-checking. Human involvement comprised: (1) rung definitions; (2) review of calibration rationales; (3) setting up fact-verification of 12 of 50 papers (43 of 186 claims) against original arXiv sources. Of the verified claims, 84% required no corrections. The 16% that required corrections were primarily claim location errors (e.g., a claim marked as “body” that appeared in the abstract) and two method misclassifications where interventional components were initially overlooked (e.g., ablation-based progress measures coded as purely observational). No inter-rater reliability metrics are available; a larger study would require multiple independent human annotators to assess LLM annotation quality. Note that claims are nested within papers (mean 3.7 per paper), so reported percentages should not be interpreted as independent observations. Additionally, systematic LLM annotator biases cannot be ruled out—the single-annotator design means any tendency of the LLM to over- or under-flag gaps would propagate uniformly across the dataset. More broadly, the rung classification treats linguistic form at face value: a paper that writes “THE circuit” in its abstract but qualifies “other circuits may exist” in its discussion receives the same gap score as one that asserts uniqueness without qualification. The binary scheme thus conflates convention with commitment, and a finer-grained annotation (e.g., a within-paper qualification flag) would be needed to separate them. A conservative sensitivity check—reclassifying all claims whose gap rests solely on definite-article or conventional-verb patterns (12 of 99 gap {}>0 claims)—reduces the gap rate from 53.5% to 47.0%, suggesting the headline finding is not driven by this ambiguity. Measuring potential ambiguity, not intent. A key methodological choice is that hedged claims (“may encode,” “partially reverse-engineer”) are coded at the underlying claim’s rung, with hedging reflected only in reduced confidence scores. This means the study measures surface-level interpretive risk —what a reader skimming the abstract or a policy-maker citing the paper could take away—rather than the authors’ epistemic state. In practice, many papers use strong language in abstracts (where space compresses nuance) while hedging in the body. Of the 99 claims with gap >0 , only 3 (3.0%) had annotator notes flagging an explicit hedge in the source paper, and even then the structural rung distance persisted. Similarly, definite articles (“THE circuit”) may function as naming conventions—referring to the circuit the authors found—rather than as uniqueness claims; the binary rung scheme cannot distinguish these readings. This design choice is deliberate: as interpretability claims increasingly inform safety-relevant decisions, the potential for misinterpretation by readers outside domain-specific conventions is precisely the risk that a calibration framework should surface. Annotator confidence corroborates the interpretive difficulty: claims scored with a gap had mean confidence 4.3 (62.6% at level 4), compared with 4.9 (95.3% at level 5) for gap-free claims, reflecting the genuine judgment involved in distinguishing mechanistic language from mechanistic evidence. The full codebook and calibration rationales are reproduced in Sections G.3 and G.4 . Annotation scheme. For each empirical claim, we annotated: • Method rung : What the method can establish (1=observational, 2=interventional, 3=counterfactual) • Claim rung : What the paper claims, based on linguistic markers • Gap score : \max(0,\text{claim\_rung}-\text{method\_rung}) Method-to-rung classification: Rung Methods 1 (Observational) e.g. linear probing, SAE attribution, attention visualisation 2 (Interventional) e.g. activation patching, causal tracing, ablation, steering 3 (Counterfactual) e.g. causal scrubbing, necessity tests, uniqueness proofs Linguistic markers for claim classification: Rung Markers 1 e.g. “correlates with,” “predicts,” “is decodable,” “activates on” 2 e.g. “causally affects,” “mediates,” “is sufficient for” 3 e.g. “encodes,” “represents,” “THE circuit,” “computes,” “stores” Replication evidence. We searched Google Scholar citations, GitHub issues, Alignment Forum posts, and author retrospectives to assess replication status: 0=replicated, 0.5=partial, 1=failed, NA=no evidence. G.2 Key Findings Rung-elevated language is prevalent. Of 185 claims with valid scores (one conceptual-analysis claim excluded), roughly half (53.5%) had gap scores >0 , meaning the claim language can admit a stronger causal reading than the reported method licenses. This reflects the field’s lack of shared terminological infrastructure for stating causal estimands—not a finding about the reliability of the underlying empirical results. Terms like “encodes” are often used informally to mean “is decodable from” rather than to assert counterfactual dependence. Most gaps were +1 (Rung 2 method \to Rung 3 claim), typically from activation patching narrated in mechanistic language (“performs,” “encodes,” “THE circuit”). Gaps of +2 (R1 \to R3) accounted for 12.4%. Gap Score Count % Pattern 0 (no gap) 86 46.5% Claim matches method rung +1 (one-rung gap) 76 41.1% R2 \to R3 typical +2 (two-rung gap) 23 12.4% R1 \to R3 (probing \to encodes) Gap rates are uniform across claim locations. Rung-elevated language is not confined to abstracts, where space pressure might excuse compressed phrasing: abstract claims show a 53.3% gap rate (72/135), while non-abstract claims (body, results, introduction) show 52.9% (27/51). This uniformity suggests a field-wide linguistic convention rather than abstract-specific compression. Paper type and gap frequency. Paper Type Papers Mean Gap Rate Circuit discovery 16 0.64 Knowledge localisation 3 0.50 Other 19 0.47 Evaluation/benchmark 5 0.45 Applied/production 7 0.35 Annotator confidence tracks gap difficulty. Confidence scores (1–5 scale) are strongly skewed toward high values (overall mean 4.6), but show a clear monotonic decrease with gap size, consistent with larger rung distances involving harder judgments. Confidence Level (%) Subset 3 4 5 Mean n All claims 2.2 35.5 62.4 4.60 186 Gap =0 0.0 4.7 95.3 4.94 86 Gap =1 1.3 57.9 40.8 4.39 76 Gap =2 8.7 78.3 13.0 4.04 23 Method Rung 1 5.6 55.6 38.9 4.33 36 Method Rung 2 1.3 24.0 74.7 4.73 150 The gap between confidence for gap-free claims (mean 4.94, with 95.3% at level 5) and gap-positive claims (mean 4.31) reflects genuine interpretive difficulty: distinguishing mechanistic language from mechanistic evidence requires judgment about linguistic convention. Rung 1 methods (probing, SAE attribution) also show lower annotator confidence, likely because mapping observational evidence to claim language involves more ambiguity than for interventional methods where the causal semantics are more explicit. Because confidence scores were produced by the same LLM annotator that assigned gap scores, the monotonic relationship may partly reflect the annotator’s internal uncertainty calibration rather than an independent measure of judgment difficulty; independent human confidence ratings would resolve this ambiguity. Scope and intended use. This pilot study measures surface-level interpretive risk : whether claim language, taken at face value, admits a stronger causal reading than the reported method licenses. It does not measure whether authors intended the stronger reading, whether the underlying findings are correct, or whether the field’s empirical contributions are unreliable. The gap rate should not be cited as evidence that mechanistic interpretability findings are untrustworthy—it is evidence that the field’s linguistic conventions have not yet converged on vocabulary that precisely tracks causal evidence strength. We offer the practitioners’ checklist in Section G.4 as a constructive tool for aligning language with evidence, not as a scorecard for evaluating past work. G.3 Annotation Codebook The complete codebook was loaded into the LLM annotator’s context at the start of each annotation session. This section reproduces the material that extends the summary tables in the methodology above. Detailed method rung classification. Each method was assigned to the highest rung of causal evidence it can establish. Method Description Example Evidence Rung 1 (Observational): correlational evidence only, no model intervention Linear probing Classifier on frozen activations “Probe accuracy of 85%” Activation logging Record activations without intervention “Feature X activates on…” SAE feature attribution Identify which SAE features activate “Feature 4123 fires on…” Attention visualisation Inspect attention weights “Attention concentrates on…” PCA / SVD Dimensionality reduction analysis “First PC correlates with…” Correlation analysis Statistical associations “ r{=}0.7 between activation and…” Rung 2 (Interventional): causal effects under specific interventions Activation patching Replace activation, measure effect “Patching head 9.1 restores 80%…” Causal tracing Systematic patching across positions “Layer 15 shows highest causal effect” Ablation Zero/mean out components “Ablating heads reduces accuracy by 40%” Steering vectors Add direction, observe output change “Adding \mathbf{v} shifts sentiment…” DAS interchange Swap aligned subspaces “IIA of 0.92 on agreement task” ROME/MEMIT edits Modify weights, observe change “After edit, model outputs…” Rung 3 (Counterfactual): what would have happened, unique mechanisms, or necessity Counterfactual patching Per-instance counterfactual “For THIS prompt, had activation been \mathbf{x} …” Causal scrubbing Test if mechanism fully explains “Scrubbing preserves behaviour” Necessity tests Show component is necessary “No alternative achieves same behaviour” Uniqueness proofs Demonstrate unique structure “This is THE circuit” Claim rung linguistic markers. The following markers and worked examples anchored claim rung assignment. Rung Markers Example Sentences 1 (Associational) “correlates with,” “is associated with,” “predicts,” “co-occurs with,” “information is present,” “is decodable from,” “can be extracted,” “activates on,” “fires when” “Sentiment information is linearly decodable from layer 6”; “The feature correlates with Python code inputs”; “Probe accuracy predicts model behaviour” 2 (Causal) “causally affects,” “has causal effect on,” “mediates,” “influences,” “is sufficient for,” “can produce,” “enables,” “intervening on X changes Y,” “ablating X degrades Y” “Head 9.1 causally affects the output”; “This component is sufficient for the behaviour”; “Ablating these heads degrades performance” 3 (Mechanistic) “encodes,” “represents,” “computes,” “performs,” “THE circuit / mechanism / feature,” “controls,” “is responsible for,” “underlies,” “this head DOES X,” “uses X to do Y” “The model encodes subject-verb agreement in this subspace”; “These heads perform the IOI task”; “ The circuit moves names from subject to output”; “This feature represents the concept of deception” Recurring rung-elevation patterns. The codebook identified six recurring patterns where claim language implies a higher rung than the method establishes: Pattern Method (Rung) Typical Claim (Rung) Gap Probing \to “encodes” Linear probe (R1) “Model encodes X” (R3) +2 SAE \to “represents” SAE attribution (R1) “Model represents X” (R3) +2 Attention \to “performs” Attention vis. (R1) “Head performs X” (R3) +2 Patching \to “THE circuit” Activ. patching (R2) “This is the circuit” (R3) +1 Steering \to “controls” Steering vectors (R2) “Controls concept X” (R3) +1 Ablation \to “necessary” Ablation (R2) “Necessary for behaviour” (R3) +1 Edge-case decision rules. • Hedged claims (“may encode,” “suggests that”): coded at the underlying claim’s rung with reduced confidence; the hedge was noted but did not lower the assigned rung. • Multiple methods : each claim was coded against the method that directly supports it. If a paper uses both probing (R1) and patching (R2), a claim supported by patching evidence was assigned R2. • Implicit claims : mechanistic narratives (“the model uses X to do Y”) were coded as R3 even without an explicit framing as a claim. Such cases received lower confidence scores. • Review/survey papers : coded as NA for replication status (not empirical). Confidence and replication coding. Annotator confidence was rated on a 1–5 scale (5 = very confident, 1 = very uncertain). Replication status used a three-point scale: 0 = successfully replicated, 0.5 = partially replicated, 1 = failed replication, NA = no replication attempt found. Evidence was sourced from (in priority order): published replication studies, replication sections in subsequent papers, GitHub issues, author corrections, and reproducibility tracks. G.4 Calibration Rationales Five papers served as internal calibration anchors during annotation, with documented rationales that the LLM accessed when annotating subsequent papers to promote consistency. We name the two anchors with tight claim–method alignment below; the three anchors that exhibit rung-elevation patterns are presented as anonymous field-level illustrations, because the patterns they contain appear across many papers in our sample—not as specific shortcomings of those (widely cited, empirically rigorous) works. What well-calibrated language looks like. Two calibration anchors demonstrate tight claim–method alignment. These papers share key features: empirical-performance framing, appropriate hedging, and the absence of mechanistic narrative where no mechanistic evidence is presented. SAE Evaluation [ 27 ] — calibration anchor. Method: mixed Rung 1–2 (SAE feature attribution evaluated via interchange intervention). This paper’s claim language consistently tracks its method rung, offering a model of well-calibrated scientific prose: Claim Rung Why It Works “SAEs struggle to reach baseline” R2 (interventional) Reports intervention outcome without asserting mechanism “features that mediate knowledge” R2 (interventional) “Mediate” precisely describes an interventional finding “useful for causal analysis” R2 (interventional) Utility claim scoped to method capability, not internal mechanism Replication: replicated. Consistent with independent SAE evaluations. Calibration lesson: Evaluation papers that compare against baselines and frame findings in terms of method utility—rather than internal mechanism—tend to have tight claim–method alignment. The key linguistic signature: verbs describe what the method does (“struggles,” “mediates”) rather than what the model does internally. Production Probing [ 81 ] — calibration anchor. Method: Rung 1 (linear probing in a production monitoring context). Claim–method alignment is tight throughout, with consistent hedging and an empirical-performance focus: Claim Rung Why It Works “probes may be promising” R1 (observational) Hedged; reports correlation without causal assertion “probes fail to generalise” R1 (observational) Empirical observation, appropriately scoped “successful deployment” R1 (observational) Outcome claim about system performance, not mechanism Replication: not applicable (production deployment context). Calibration lesson: Production and applied papers with engineering framing—where success is measured by system-level outcomes rather than mechanistic explanation—tend toward appropriate claim levels. The absence of mechanistic vocabulary (“encodes,” “represents,” “computes”) removes the primary source of rung elevation. Common rung-elevation patterns in practice. Three additional calibration anchors—all widely cited, empirically strong papers—exhibited rung-elevation patterns characteristic of the broader field. We present these as anonymous illustrations because the patterns they exemplify appear across many papers in our dataset; attaching them to individual works would misrepresent what is a field-wide linguistic phenomenon. Pattern 1: Circuit discovery \to mechanistic narrative. A paper using activation patching (Rung 2) to identify task-relevant components narrates its findings with functional verbs (“performs,” “moves,” “inhibits”), definite articles (“THE circuit”), and engineering metaphors (“reverse-engineering”). Each of these linguistic choices admits a Rung 3 reading—implying uniqueness, necessity, or complete mechanistic understanding—while the underlying method establishes causal sufficiency under specific interventions. Lesson: Definite articles and functional verbs are inherited from neuroscience and engineering, where they serve as naming conventions. In the context of mechanistic interpretability, these conventions can inadvertently imply counterfactual claims (uniqueness, necessity) that the method does not establish. This pattern accounts for the majority of one-rung gaps in our dataset. Pattern 2: Knowledge localisation \to storage language. A paper using causal tracing and weight editing (Rung 2) describes its findings using storage and memory vocabulary: “stores,” “contains,” “localised computations.” The method establishes that intervening on specific components changes the model’s output (mediation), but storage language implies an internal representation that is the fact—a counterfactual claim about necessity and specificity. Notably, the same paper uses appropriate Rung 2 language elsewhere (e.g., “mediates factual predictions”), illustrating that rung-appropriate vocabulary is available and sometimes used within the same work. Lesson: Storage and memory metaphors (“stores,” “encodes,” “contains”) typically imply Rung 3; causal tracing and editing establish mediation (Rung 2), not storage mechanisms. The gap is subtle because “stores” feels like a natural description of what editing reveals, but it implies a representational commitment beyond what the intervention licenses. Pattern 3: Algorithm reverse-engineering \to completeness claims. A paper using ablation and weight analysis (Rung 1–2) to characterise a learned algorithm describes its findings as “fully reverse-engineering” “the algorithm,” with claims that the model “uses [specific computation] to convert” inputs. The rung distance ranges from one rung (ablation results narrated as complete mechanisms) to two rungs (weight-analysis results narrated as encoding claims). Crucially, this paper’s findings have been independently replicated by multiple groups, and the constrained mathematical setting supports strong conclusions. Lesson: Rung-elevated language can coexist with strong empirical support. The rung distance here reflects linguistic convention, not evidential weakness. Mitigating factors—multiple converging methods, a mathematically constrained setting, and specific falsifiable predictions—substantially reduce interpretive risk even where the surface language admits a stronger reading. Interpretability Claims Checklist 1. Estimand: I am measuring e.g., “probe accuracy for is_plural on layer 8 residual stream” or “logit difference change when patching MLP outputs” 2. My method provides evidence at rung: \square Associational \square Interventional \square Counterfactual Probing \rightarrow association; activation patching \rightarrow intervention; interchange interventions \rightarrow counterfactual 3. My claim requires evidence at rung: \square Associational \square Interventional \square Counterfactual “X encodes Y” \rightarrow association; “X causes Y” \rightarrow intervention; “if X had been different, Y would change” \rightarrow counterfactual 4. Rung check: Method rung \geq Claim rung? \square Yes \square No \rightarrow revise If No: either strengthen method (add interventions) or weaken claim (“correlates with” instead of “implements”) 5. Alternatives not ruled out: e.g., “probe may exploit token length, not the concept” or “other circuits may achieve same patching effect” 6. I tested robustness by varying: e.g., “different prompt templates, random seeds, ablation strategies (zero/mean/resample)” 7. This applies to: (and not beyond) e.g., “GPT-2 small, English subject-verb agreement” \rightarrow not other languages, larger models, or different tasks Common patterns to check: \triangleright Claiming causation from probing alone (L1 \to L2) \triangleright Claiming “the” circuit without testing alternatives (sufficiency \to uniqueness) \triangleright Claiming generality from single distribution (local \to global) This checklist is not meant to discourage exploratory work or to raise publication barriers. A paper that states its rung, names its equivalence class, and bounds its scope contributes more than one whose language implies stronger evidence than reported—even if the former’s claims are narrower.