Tracking Equivalent Mechanistic Interpretations Across Neural Networks Alan Sun, Mariya Toneva Abstract Mechanistic interpretability (MI) is an emerging framework for interpreting neural networks. Given a task and model, MI aims to discover a succinct algorithmic process, an interpretation , that explains the model’s decision process on that task. However, MI is difficult to scale and generalize. This stems in part from two key challenges: there is no precise notion of a valid interpretation; and, generating interpretations is often an ad hoc process. In this paper, we address these challenges by defining and studying the problem of interpretive equivalence : determining whether two different models share a common interpretation, without requiring an explicit description of what that interpretation is. At the core of our approach, we propose and formalize the principle that two interpretations of a model are equivalent if all of their possible implementations are also equivalent. We develop an algorithm to estimate interpretive equivalence and case study its use on Transformer-based models. To analyze our algorithm, we introduce necessary and sufficient conditions for interpretive equivalence based on models’ representation similarity. We provide guarantees that simultaneously relate a model’s algorithmic interpretations, circuits, and representations. Our framework lays a foundation for the development of more rigorous evaluation methods of MI and automated, generalizable interpretation discovery methods. 1 1 1 Our codebase can be found at https://github.com/alansun17904/interp-equiv 1 Introduction Ensuring the interpretability of deep neural networks is central to concerns around AI safety and trustworthiness. Among many proposed interpretation methods, mechanistic interpretability (MI) has recently emerged as a promising post-hoc interpretability framework 2 2 2 An interpretability method that does not require any retraining. These methods can be applied in parallel with inference and do not compromise the model’s performance. ( 45 ; 14 ; 73 , inter alia ) . MI generally operates in two stages: (a) identifying a minimal subset of the model’s computational graph that drives functional behavior; and (b) attaching algorithmic interpretations to each of the recovered mechanisms. In contrast to other attribution-based interpretability methods, MI yields a concrete, human-interpretable algorithmic process that faithfully describes model behavior ( 7 ; 31 ) . The resulting processes can be used to better understand a model’s inductive biases which may inform further improvements ( 19 ; 38 ; 3 , inter alia ) . For a fixed task and network, MI approaches can be broadly categorized as either top-down or bottom-up ( 72 ) . Top-down methods (b → a) propose a set of high-level candidate algorithms for the task, and then attempt to align those algorithms to the network. While the alignment step can be automated and made statistically rigorous, proposing candidate algorithms is highly ad hoc. For complex tasks, it is often unclear how to even formulate plausible candidates. Additionally, alignment with a proposed algorithm is only a necessary condition for interpretability, and does not guarantee that the model truly implements the intended algorithm ( 76 ; 18 ; 16 ; 62 ) . In contrast, bottom-up methods (a → b) first isolate mechanisms-of-interest within the model (called a circuit ) then assign interpretations to components within that circuit ( 45 ) . Although circuit discovery can be rigorously formulated as an optimization problem ( 8 ) , creating and assigning meaningful interpretations to circuits typically demand significant manual effort. Moreover, 37 has recently shown that neither top-down nor bottom-up approaches are identifiable: there exists a many-to-many relationship between high-level algorithms and circuits. This ambiguity makes it hard to verify whether a given interpretation is complete and faithful to the model ( 26 ) . We present a detailed discussion of the related literature in Appendix B . In this paper, we define and study a relaxed, subproblem of MI: interpretive equivalence . Concretely, we seek to determine whether two models implement the same high-level algorithm, without requiring an explicit description of what that algorithm is. Understanding interpretive equivalence can bridge the gap between bottom-up and top-down approaches through reductions . Consider two examples: Example 1.1 (Reduction to Simpler Models) . MI is computationally prohibitive on larger models ( 1 ) . To mitigate this, if we could show that a small model is interpretively equivalent to a large one, then MI analyses on the small model could interpret the larger model. Example 1.2 (Reduction to Simpler Tasks) . MI’s scope is limited because interpreting models that solve complex tasks is at least as hard as manually designing an algorithm to solve them ( 42 ; 79 ) . Interpretive equivalence provides a criterion to decompose complex tasks into simpler, interpretively equivalent ones. Figure 1: A high-level overview of our algorithmic approach to interpretive equivalence. Consider models h_{1},h_{2} that correspond to possibly unknown interpretations A_{1},A_{2} (Left) . To determine whether models h_{1} and h_{2} are interpretively equivalent, we propose a two-step procedure. First, we sample another model h^{\star} that also has interpretation A_{1} (Center) . Second, we compare the representation similarity ( d_{\mathrm{repr}} ) between h_{1},h^{\star} and h^{\star},h_{2} (Right) . If A_{1},A_{2} are equivalent, then averaged over all implementations h^{\star} , we should not be able to differentiate d_{\mathrm{repr}}(h_{1},h^{\star}) and d_{\mathrm{repr}}(h^{\star},h_{2}) . At the core of our approach (Figure 1 ), we propose the principle that two high-level algorithms (henceforth termed interpretations 3 3 3 We distinguish between an algorithm and an interpretation. Informally, an algorithm is independent of any computational abstraction. However, we show in Appendix E that this abstraction-free notion results in pathological equivalence definitions. Thus, we use the terminology interpretation to signify a dependence on a chosen level of computational abstraction. ) are equivalent if their implementations are equivalent. Based on this principle, we design an algorithm to detect equivalence by measuring representation similarity. We prove that representation similarity is necessary and sufficient to characterize interpretive equivalence. Overall, our contributions span both theory and practice: • We propose an algorithm to compute interpretive equivalence between two models without interpreting them (Algorithm 1 ). • We demonstrate that Algorithm 1 is well-calibrated on a simple task where the ground truth is known. Then, we show its potential to find reductions (Examples 1.1 and 1.2 ). • We specialize the theory of causal abstraction to define interpretations, circuits, and representations (Sections 4 and 5 ). As byproducts of this formality, we obtain a measure of interpretation quality and an alternative characterization of interpretations as interventions (Section 5 and Appendix H ). • We prove necessary and sufficient conditions for models’ interpretive equivalence based on their representation similarity. To our knowledge, these guarantees are the first to relate a model’s interpretations, circuits, and representations (Section 6 ). Throughout our paper, we use examples from language processing and Transformers. However, our framework does not depend on any properties of the input modality. 2 Interpretive Equivalence Through Congruent Representations We define mechanistic interpretations A_{1},A_{2} as equivalent if they have equivalent implementations. That is, any model which can be interpreted by A_{1} must also be interpreted by A_{2} and vice versa. We justify this principle later, but first we offer an algorithm to approximate this equivalence. This requires clarifying two procedures: (1) how do we enumerate all implementations of A_{1},A_{2} ? (2) How do we measure the distance between these sets of implementations? Enumerating Implementations through Interventions. We call any model h that has a mechanistic interpretation A an implementation of A (Definition 5.1 ). We take a bottom-up approach to generating implementations. Using causal interventions, we identify components in the model that are causally unrelated to the model’s behavior ( 12 ; 21 ) (i.e. they are not a part of the model’s circuit). A model’s mechanistic interpretations should be invariant to perturbation or ablation of these components. Dually, each time we perturb or ablate such a component, we yield a “new” model that is causally equivalent to the original one (and as a corollary, shares the same interpretation). Intuitively, every implementation of A should be reachable through a causal intervention applied onto h . We take this dual perspective to generating implementations by adding, removing, or modifying these unimportant components ( 18 ; 22 ) . This procedure is denoted as GetImpl (Algorithm 1 ; detailed in Appendix C ). 1 procedure Congruity ( h_{1},h_{2},n ) 2 \text{s}\leftarrow 0 3 for i\leftarrow 1\ldots n do 4 h_{1},h_{1}^{\star}\leftarrow\textsc{{\color[rgb]{0.72,0,0}{{GetImpl}}}}(h_{1}) 5 h_{2},h_{2}^{\star}\leftarrow\textsc{{\color[rgb]{0.72,0,0}{{GetImpl}}}}(h_{2}) 6 \text{s}\leftarrow\text{s}+\textsc{\color[rgb]{0.12,0.98,0.64}{{ReprDist}}}(h_{1},h_{1}^{\star},h_{2}) 7 \text{s}\leftarrow\text{s}+\textsc{\color[rgb]{0.12,0.98,0.64}{{ReprDist}}}(h_{2},h_{2}^{\star},h_{1}) return 1-|s/n-1| 8 procedure ReprDist ( h_{1},h_{2},h_{3} ) 9 for i\leftarrow 1,2,3 do 10 R_{i}\leftarrow\textsc{{\color[rgb]{0,0.88,0}{GetReprs}}}(h_{i}) 11 if d_{\mathrm{repr}}(R_{1},R_{2})\leq d_{\mathrm{repr}}(R_{1},R_{3}) then 12 return 1 13 return 0 Algorithm 1 Two models are congruent if representation similarity alone cannot differentiate them. d_{\mathrm{repr}}(R_{1},R_{2}) measures representation similarity between representations of h_{1},h_{2} . GetReprs , GetImpl retrieve the hidden representations of a given model and its implementations, respectively. Representation Similarity. We identify each implementation with their hidden representation spaces ( GetReprs in Algorithm 1 ). Given two implementations, we measure their distance using linear representation similarity ( d_{\mathrm{repr}} defined in Definition G.1 ). Informally, linear representation similarity captures the extent to which the representations of one network can be linearly transformed to the other and vice versa. We notate this as ReprDist (Algorithm 1 ). Suppose that the implementation sets of A_{1},A_{2} were exactly equal. Let h_{1},h_{1}^{\star} be implementations sampled from A_{1} and h_{2} be an implementation sampled from A_{2} . 4 4 4 For the sake of argument, suppose i.i.d. sampling from a uniform distribution over A_{1},A_{2} . By symmetry, \mathbb{P}[d_{\mathrm{repr}}(h_{1},h_{1}^{\star})d_{\mathrm{repr}}(h_{2},h_{1}^{\star})] . In this way, the implementations of A_{1},A_{2} are congruent with respect to d_{\mathrm{repr}} . Combining these concepts, we have our approximation of interpretive equivalence: Congruity . 3 Experiments Herein, we demonstrate three applications of Algorithm 1 . First, on a toy task where ground-truth interpretations are known, we show that Congruity is well-calibrated (Section 3.1 ). Next, we apply our framework to pre-trained language models of various sizes (GPT2 ( 53 ) and Pythia ( 9 ) ) and demonstrate that Congruity can distinguish between models that exhibit fine-grained algorithmic differences across various model scales (Section 3.2 ). Lastly, we show how Congruity can be used to relate a complex task like next-token prediction to a simpler one such as parts-of-speech identification (Section 3.3 ). 3.1 Calibrating Congruity We consider the task of n -Permutation Detection : determining whether a given sequence of n numbers is a permutation of the elements 1,\ldots,n . For example, when n=3 , [3, 1, 2] \to True and [1, 2, 2] \to False. The task is sufficiently rich to support different interpretations, yet simple enough that solutions can be hard-coded as Transformers. We manually devise six different interpretations to solve this task whose procedures we detail in Appendix C . Roughly, their approaches can be organized into two buckets: 1. Sorting-Based (Interpretations 1–4). First sort the list of numbers by ascending (or descending) order, then directly check whether the resulting sequence is equal to 1,2,\ldots,n (or n,n-1,\ldots,1) . 2. Counting-Based (Interpretations 5–6). Exploit the fact that the vocabulary contains exactly n numbers and use the pigeonhole principle to detect duplicates in the given sequence. For all experiments, we fix n=10 . Using the Restricted Access Sequence Processing Language (RASP), we hard-code six bidirectional Transformers (of various architectures) to implement each interpretation ( 75 ) . Then, for each hard-coded RASP Transformer, we leverage a technique of 22 ’s ( 22 ) to generate 100 model variants (with different architectures and weight configurations). These variants are constrained to maintain the same underlying interpretation as their hard-coded counterpart. So for each interpretation, we have 100 implementations. In total, we have 6\times 100 models that all achieve 96\% + accuracy on 10 -permutation detection. We now directly compute Congruity (Algorithm 1 ) between pairs of models from our generated implementations. We perform hypothesis testing by bootstrapping a 95% confidence interval over the final output. 5 5 5 Here, H_{0}: the two models do not share the same interpretation; and our alternate, H_{1}: the two models share the same interpretations. We compute a Wald confidence interval with 20 bootstrap samples. The results are shown in Figure 2 (Left) . Along the diagonals of Figure 2 (Left) , we see significantly high congruence. So when models share the same interpretation, representations across their implementations are approximately equivalent under linear transformation. On the other hand, off-diagonal entries admit low congruence. Thus, representation similarity across set of implementation as a whole can differentiate individual interpretations. These empirical findings complement our bounds in Main Results 1 and 2 because they suggest that Congruity is both necessary and sufficient to detect interpretive equivalence. Perhaps even more interesting, we observe a breaking row/column in Figure 2 , where Interpretations 1–4 (the top-left 4\times 4 square) have an average within-group congruence of 0.43 . This is larger compared to the average across-group congruence (rows/columns 5-6) with Interpretations 5–6: 0.01 . These groupings correspond to our broad algorithmic buckets above and suggest that Congruity could also be adopted as a graded notion to characterize broad-stroke interpretive differences. Figure 2: (Left) Average congruity between models associated with different interpretations. \blacksquare indicates models have different interpretations; whereas \blacksquare indicates that models have statistically indistinguishable interpretations. (Center) Congruity between GPT2 and Pythia family of models on the IOI task. groups models based on their actual interpretive differences observed by 66 ; 40 . (Right) Congruity between GPT2 on next-token prediction (for different token types: all tokens, articles, prepositions, punctuation, parentheses, and terminal punctuation) vs. GPT2 on in-context parts-of-speech identification. 3.2 Reduction to Simpler Models We now consider the task of indirect-object identification (IOI) ( 73 ) : given a sentence like “When John and Mary went to the store, John gave the drink to ” the model should complete the sentence with “Mary.” Because IOI can be solved algorithmically, it has been studied across many models: GPT2-small/medium and the Pythia class of models. While 40 find that GPT2-small and medium use the same circuit, 66 find that Pythia models across all scales use a consistent but different circuit from GPT2 models ( 40 40 , Appendix C; 66 66 : Appendix D). We use these differences as a practical testbed for Congruity and explore how it generalizes across both model families and scales (85M to 2.9B parameters). 6 6 6 Notably, 66 also find that circuits in larger models require more components , even when the underlying interpretation is constant. Thus, for Congruity to be effective in this setting, it also needs to move beyond identifying architectural similarities. For each model, we generate 10 parallel implementations by intervening on the components found not to be in the IOI circuit (identified by both 66 66 and 40 40 ). Then, we apply Congruity , each time computing representation similarity with 200 IOI sentences. The results are shown in Figure 2 (Center) . We find that the Pythia models show high within-group congruence across different scales. GPT2 models exhibit similar behavior. This supports our intuition that interpretive equivalence could be leveraged to reduce the interpretations of complex models into interpretations of smaller ones (Example 1.1 ). For example, our results affirm that on the IOI task, Pythia-2.8b is interpretively equivalent to Pythia-160M; thus, a priori , it suffices to only interpret the latter. Indeed, 66 finds that both models share the same interpretation. On average, across-group congruence between Pythia and GPT2 is significantly lower (0.13 versus 0.73 and 0.92) which supports our previous insight that representation similarity provides increased identifiability over interpretations. It is unclear why Pythia-160M and 410M show increased congruence with GPT2-medium and small, respectively; perhaps, subtle similarities in name-mover heads representations could explain this ( 66 66 , Section 3.2). 3.3 Reduction to Simpler Tasks Interpreting next-token prediction poses a challenge for both bottom-up and top-down MI approaches. For top-down approaches writing down a symbolic, end-to-end algorithm for next-token prediction is difficult. Bottom-up approaches face the opposite problem: circuit discovery may identify the entire model as important, failing to reduce the search space of interpretations. We show here that interpretive equivalence may offer some first-steps towards mechanistically understanding next-token prediction. Concretely, we identify sets of next-tokens for which GPT2’s next-token prediction process is interpretively similar to parts-of-speech identification (POS 7 7 7 Given tree:noun; run:verb; quick:adverb; fluffy: the model needs to output adjective. ). POS is a largely syntactic task since it focuses solely on the grammatical role of words rather than their meaning in-context. Thus, we expect POS to be interpretively equivalent to the prediction of “syntactic tokens.” For computational ease, we conduct all experiments on GPT2. To discover the POS circuit in GPT2, we apply a method of 68 ’s ( 68 ). We detail this process, along with the circuit we discover in Appendix C . We construct a dataset for next-token prediction by uniformly sampling 100 sentences from the C4 dataset ( 54 ) . We collate tokens from our next-token prediction sentences into disjoint groups of interest. For each group, we apply Congruity and compute representation similarity between our extracted POS circuit and the last-token hidden representations of the model. The results across different token groups are shown in Figure 2 (Right) . As a control, we first consider the group of all tokens. We observe a nonzero congruence of 0.48 with POS. Next, we consider token groups articles and prepositions . Prediction of these tokens typically appear mid-sentence and depend heavily on semantics. 8 8 8 Consider the fragment: “[context] The students are looking [mask]” the next prepositions—“at”, “into”, or “for”—are all syntactically sound, but they are semantically ambiguous without further context. Articles are similar: consider, “I need to buy [mask] car.” [mask] could be either the indefinite (“a”) or definite (“the”), both are syntactically valid but the choice requires pragmatic reasoning. Thus, we expect that predicting these tokens should yield no more interpretive equivalence to POS than the control. Indeed, we find POS’s congruity to articles and prepositions to be statistically indistinguishable from the all-token average. We now heuristically identify two sets of “syntactic tokens:” 1. Terminal Punctuation like “.”, “?”, or “!” mark the end of a sentence. Accurate prediction of these tokens intuitively requires syntactic identification of subject-verb-object relations and (in)dependent clauses. 2. Closing Brackets/Quotations like “)” can be implemented as skip-trigrams in one-layer attention Transformers ( 14 ) . So, we suspect that understanding sentence pragmatics is not needed for their prediction. We find that both groups yield significantly higher congruence compared our all-token control with medium effect size (measured through Cohen’s d ). This stands in contrast to general punctuation and opening brackets/quotations. Although these other punctuation groups admit statistically significant congruence with POS, we observe a small effect size ( d<0.3 ). We hypothesize that the placement of these tokens rely more on semantic processes. For example, commas may be placed for emphasis or to add dependent clauses rather than strictly maintain grammatical consistency. 4 Representations, Circuits, Interpretations as Causal Models We now present our framework of interpretive equivalence. This theory grounds our algorithmic contributions in Section 2 . Whenever possible, we present an informal treatment and defer the precise definitions to Appendix D , where we also have a full glossary and summary of notations. The next sections are organized as follows: 1. In Section 4 , we define circuits , representations , and interpretations (Definitions 4.2 , 4.3 , and 4.4 , respectively). 2. In Section 5 , we define a criterion for when two interpretations are equivalent. We also introduce interpretive compression , a measure of interpretation abstraction. 3. In Section 6 , we prove that representation similarity can describe interpretive equivalence (Main Results 1 and 2 ) and show that Congruity is closely related to these quantities (Main Result 3 ). Let \Sigma be an alphabet and \Sigma^{\star} be all finite strings formed using \Sigma . We define a language model as a function h:\Sigma^{\star}\to\mathbb{R}^{|\Sigma|} . We often are interested in h ’s behavior on a subset of inputs S\subset\Sigma^{\star} , rather than on all of \Sigma^{\star} . For this reason, we term S a task . While h alone serves as a sufficient functional description, interpretability requires understanding h ’s underlying computational process. For this, we rely on three behavioral characterizations of a neural network at different levels of abstraction: h ’s circuits , representations , and interpretation . 4.1 Circuits Definition 4.1 (Deterministic Causal Model, 16 16 ) . A (deterministic) causal model with m components is a quadruple ({\mathbb{V}},U,{\mathbb{F}},\succ) where {\mathbb{V}}=({\bm{v}}_{1},\ldots,{\bm{v}}_{m}) is a set of hidden variables such that |{\mathbb{V}}|=|{\mathbb{F}}|=m , U is an input variable, and \succ defines a partial ordering over {\mathbb{V}} . For each {\bm{v}}_{k}\in{\mathbb{V}} , {\bm{v}}_{k}\triangleq f_{k}(Pa({\bm{v}}_{k}),U) (4.1) where f_{k}\in{\mathbb{F}} and Pa({\bm{v}}_{k})\subset\{{\bm{v}}\in{\mathbb{V}}:{\bm{v}}\succ{\bm{v}}_{k}\} are the parents of {\bm{v}}_{k} . A deterministic causal model describes a computational graph where hidden variables {\mathbb{V}} (nodes) store latent computation results computed by the functions in {\mathbb{F}} (edges). Since a computation graph is acyclic, the data/compute flow determined by {\mathbb{F}} induces a partial ordering over {\mathbb{V}} , \succ . For a specific input setting U\leftarrow{\bm{u}} , we denote by {\mathbb{V}}^{\mathbb{F}}({\bm{u}}) to be the unique solution : a configuration of the values {\mathbb{V}} consistent with both {\bm{u}} and {\mathbb{F}} ( 51 ) . Likewise, {\bm{v}}_{i}^{\mathbb{F}}({\bm{u}}) is the solution of {\bm{v}}_{i}\in{\mathbb{V}} to U\leftarrow{\bm{u}} . In this way, {\bm{v}}_{i}^{\mathbb{F}} is a function that maps from S\to\mathbb{R}^{n_{i}} . There is a natural tradeoff between the collective complexity of {\mathbb{V}} and the complexity of any individual operator in {\mathbb{F}} . On one extreme, any blackbox model h can be expressed as a trivial causal model with one hidden variable: {\bm{v}}_{1}\triangleq h(U) . On the other, each {\bm{v}}\in{\mathbb{V}} could be a single neuron (thus |{\mathbb{V}}|\sim 10^{9} ), while elements in {\mathbb{F}} consist of primitive operations like dot-products. In this way, {\mathbb{V}} determines the level of abstraction of the causal model. We now use this formalism to define circuits. Circuits form the starting point for almost all bottom-up approaches in MI ( 73 ; 12 ; 33 , inter alia ) . Intuitively, for some task S , a circuit describes the (minimal) computational pathways used by the model to produce h(S) . Throughout our treatment, we do not require minimality as finding minimal circuits has been shown to be an intractable combinatorial optimization problem ( 1 ) . Definition 4.2 (Circuit) . An m -circuit of h on S is a causal graph ({\mathbb{V}},U,{\mathbb{F}},\succ) with m components that satisfies: (1) U is S -valued; (2) Hidden variables are real-valued; (3) There exists a path from U to a variable {\bm{v}}_{\rm{out}}\in{\mathbb{V}} such that {\bm{v}}_{\rm{out}}^{\mathbb{F}}(S)=h(S) . For any h and m\in\mathbb{Z}^{+} , an m -circuit of h exists by its trivial causal model. Since we do not require minimality, m -circuits are not unique: a joint change of bases to any {\bm{v}}_{k}\in{\mathbb{V}}\setminus{\bm{v}}_{\rm{out}} and f_{k} ; or, adding a new variable that does not contribute to {\bm{v}}_{\rm{out}} results in a new causal model that preserves faithfulness to h . We further explore some of these properties in Section 5 . 4.2 Representations Representations of a neural network are a sequence of activation spaces. Increasingly, representations have been understood to encode concepts which deep networks then iteratively refine ( 28 ) . We formally view representations as abstractions of circuits and define them through causal abstraction ( 5 ; 16 ) . A causal model K^{\star} abstracts K if there exists a surjective map from K ’s variables to K^{\star} ’s variables that preserves K ’s causal relationships. Any map that admits this abstraction is an alignment from K to K^{\star} (Definition D.3 ). Essentially, an alignment \pi maps K ’s “low-level” variables to K^{\star} ’s “high-level” variables ( 56 ) . Like m -circuits, for a given low-level and high-level causal model pair, the set of alignment maps that exist between them are not unique. Definition 4.3 (Representations) . Let K be an m -circuit of h on a task S . An L -circuit that abstracts K is an (L,\delta) -representation of K if for each {\bm{v}}_{k}\in{\mathbb{V}} , • Pa({\bm{v}}_{k+1})=\{{\bm{v}}_{k}\} , Pa({\bm{v}}_{1})=\{U\} , and {\bm{v}}_{L}={\bm{v}}_{\rm{out}} . • For each 1\leq k\leq L , there exists linear maps A_{k}:\mathbb{R}^{n_{k}}\to\mathbb{R}^{|\Sigma|},B_{k}:\mathbb{R}^{|\Sigma|}\to\mathbb{R}^{n_{k}} where \left\|A_{k}\right\|_{\mathrm{op}},\left\|B_{k}\right\|_{\mathrm{op}}\leq 1 , {\color[rgb]{0.72,0,0}\lVert{A_{k}{\bm{v}}_{k}^{\mathbb{F}}-h}\rVert}<\delta\qquad\text{and}\qquad{\color[rgb]{0.06,0.46,1}\lVert{B_{k}h-{\bm{v}}_{k}^{\mathbb{F}}}\rVert}<\delta (4.2) for some norm (to be specified). Representations have two crucial properties: they compress the complex causal model of K into a single chain of length L ; and, they have high signal and low noise . By Equation ( 4.2 ), each layer’s hidden representation can identify K ’s output behavior (independent of its children). Yet, the representations at each layer are simple enough that outputs of h can identify their value (independent of its ancestors). In MI, this latter constraint is often used as a search criterion to localize a network’s circuit ( 6 ) . For h , its set possible of circuits explode combinatorially with depth, whereas the set of possible representations only scale quadratically. Therefore, compression through abstract representation gives us a tractable staging ground to compare circuits. 4.3 Interpretations Interpretations are symbolic, human-understandable descriptions of h that faithfully captures its functional behavior on a task. We formalize them as abstractions. Definition 4.4 . Fix a task S . A causal model A\triangleq({\mathbb{V}}^{\star},U,{\mathbb{F}}^{\star},\succ) that abstracts a circuit K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) is an \eta -faithful interpretation if A ’s output approximates K ’s with error at most \eta across inputs in S . Notably, our construction does not assume there is a distance metric over A ’s variables nor do we specify the values of these variables. This generality aligns our framework with most MI literature where {\mathbb{V}}^{\star} could take on symbolic values ( 23 ; 79 , inter alia ) . For any circuit K , an interpretation always exists through the trivial interpretation : K itself is an interpretation of K (with the identity abstraction; albeit, this is not very useful). Since interpretations depend on a fixed level of abstraction ( {\mathbb{V}}^{\star} ), they must not be unique: by varying {\mathbb{V}}^{\star} and \pi we can produce different interpretations. 9 9 9 To see more details of this construction, refer to Definition D.3 and Equation ( D.2 ) As a corollary, circuits are not identifiable by their interpretations. This seems problematic, but in Appendix E we demonstrate that abstraction-dependent interpretations are necessary for a non-trivial notion of interpretive equivalence. 5 Interpretive Equivalence through Implementation Equivalence 37 shows that there exists a many-to-many relationship between circuits and interpretations. This implies that comparing individual circuits and interpretations is ill-defined. Thus, we propose that two interpretations are (approximately) equivalent if and only if their implementations are (approximately) equivalent. By examining families of circuits instead of individual ones, we can effectively quotient out the many-to-one mapping from circuits to interpretations. To start, we define implementations . Let K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) be an m -circuit of h on some task S . Let A\triangleq({\mathbb{V}}^{\star},U,{\mathbb{F}}^{\star},\succ) be an \eta -faithful interpretation of K through an alignment \pi:\mathrm{SubsetsOf}({\mathbb{V}})\to{\mathbb{V}}^{\star} (see Definition D.3 ). Notice that for fixed A,\pi , by carefully varying {\mathbb{F}} we could construct many different circuits (and computation graphs) over ({\mathbb{V}},U) that all abstract to the same interpretation A under \pi . Concretely, in our framework, {\mathbb{V}},U defines a neural architecture and {\mathbb{F}} corresponds to many weight configurations that result in the same interpretation A under \pi . We then define A ’s implementations as the set of all admissible {\mathbb{F}} under A,\pi . This construction mirrors observations that implementations are diffuse and plentiful in the weight space ( 35 ; 79 ; 22 ) . Precisely, Figure 3: Definition 5.1 (Implementation) . Let \Pi be a class of alignments. F is an implementation of A if there exists \pi\in\Pi such that A is an \eta -faithful interpretation of ({\mathbb{V}},U,F,\prec_{F}) under \pi . By a slight abuse of notation, we denote \Pi^{-1}(A) as the set of all implementations of A under alignments \Pi . Then, a non-empty intersection between \Pi_{1}^{-1} and \Pi_{2}^{-1} directly corresponds to 37 ’s ( 37 ) non-identifiability result. We assume that any circuit comes with a metric attached to its variables, i.e. d:{\mathbb{V}}\times{\mathbb{V}}\to\mathbb{R}^{\geq 0} . This is reasonable, as a circuit’s variables are real-valued (Definition 4.2 ). d naturally induces a pseudometric over \Pi^{-1}(A) , where for F,\tilde{F}\in\Pi^{-1}(A) d(F,\tilde{F})\triangleq d({\mathbb{V}}^{F},{\mathbb{V}}^{\tilde{F}})=d({\mathbb{V}}^{F}(S),{\mathbb{V}}^{\tilde{F}}(S)). (5.1) Given two sets of implementations over ({\mathbb{V}},U,\cdot,\cdot) we can leverage d to quantify (1) the Hausdorff distance between them; and, (2) their diameters. We call the former interpretive equivalence and the latter interpretive compression (see Figure 3 ). More precisely, Definition 5.2 . For i\in\{1,2\} , let A_{i}\triangleq({\mathbb{V}}_{i}^{\star},U,\cdot,\cdot) be a \eta_{i} -faithful interpretation of K\triangleq({\mathbb{V}},U,\cdot,\cdot) and \Pi_{i} be a class of alignments that map from \mathrm{SubsetsOf}({\mathbb{V}})\to{\mathbb{V}}_{i}^{\star} . Then, A_{1},A_{2} are {\epsilon} -approximate interpretively equivalent if d_{\mathrm{interp}}(A_{1},A_{2})\triangleq d_{\mathrm{H}}\left(\Pi_{1}^{-1}(A_{1}),\Pi_{2}^{-1}(A_{2})\right)<{\epsilon}, (5.2) where d_{\mathrm{H}} is the Hausdorff distance 10 10 10 The Hausdorff distance between two sets A,B is the greatest distance a point from A,B needs to travel to reach B,A , respectively: \tilde{d}(a,B)\triangleq\inf_{b\in B}d(a,b) , d_{\mathrm{H}}(A,B)\triangleq\max(\sup_{a\in A}\tilde{d}(a,B),\sup_{b\in B}\tilde{d}(b,A)) . defined by d (see Equation ( 5.1 )). Definition 5.3 . Let A be an \eta -faithful interpretation of K and \Pi be a class of alignments. Then, the interpretive compression of A is \kappa(A,K,\Pi)\triangleq\diam(\Pi^{-1}(A))=\sup_{F,F^{\prime}\in\Pi^{-1}(A)}d(F,F^{\prime}). (5.3) Interpretive equivalence and compression are tightly coupled. To explore their relationship, let us consider two extremes. On one hand, suppose A=K and |\Pi|=1 . Then, there exists exactly one implementation of A under \Pi : K . This implies that there is no interpretive compression. Further, any interpretation equivalent to A must admit the same exact same weight configuration as K . This effectively creates a bijective mapping between model weights and interpretations. On the other hand, suppose that A is the trivial causal model of h , and let \Pi be all alignments. Since A conveys no information about the computational process of K and \Pi places no constraint on alignment, any circuit by Definition 4.2 must be an implementation of A . In fact, 63 show under mild assumptions that this phenomenon persists as long as \Pi is the set of all alignments. And so, in this case, interpretive compression is maximal. These examples illustrate a duality between equivalence and compression: while more compression shrinks and simplifies the set of possible interpretations, it simultaneously enlarges the set of implementations and complicates interpretive equivalence. Interpretive equivalence, in Definition 5.2 , requires that both interpretations originate from the same circuit K . This constraint can be relaxed by augmenting \Pi_{i} such that (\pi:\mathrm{SubsetsOf}({\mathbb{V}}_{1})\to{\mathbb{V}}_{i}^{\star})\in\Pi_{i} . Through \Pi_{i}^{-1} , we only consider implementations of A_{1},A_{2} on the same neural architecture {\mathbb{V}}_{1} . This creates asymmetry in choosing which circuit to use for comparison ( {\mathbb{V}}_{1} or {\mathbb{V}}_{2} ). Our intuition is that we should select the circuit of least compression that admits at least one implementation for both interpretations. 11 11 11 Attached to each {\mathbb{V}}_{1},{\mathbb{V}}_{2} are distance metrics d_{1},d_{2} . Alternatively, one could also choose the circuit where d_{i} is most convenient to compute. Our bounds in Section 6 justify this principle. We describe critical properties of interpretive equivalence and compression in Appendix F . 6 Representational Similarity and Interpretive Equivalence Using interpretive compression and representation similarity, we now prove both upper and lower bounds on interpretive equivalence (Main Result 1 and Main Result 2 , respectively). The proofs and formalisms are deferred to Appendix G . Throughout this section, we consider the most general setup: for i\in\{1,2\} , let K_{i} be a circuit of a model h_{i} over a shared task S . Assume each circuit admits an (L_{i},\delta_{i}) -representation R_{i} through an alignment map \rho_{i} . First, we define a notion of distance between these two representations. Definition 6.1 (Representation Similarity) . Let H_{i} be the concatenation of all hidden variables in R_{i} . Then, the representation similarity of R_{1},R_{2} , d_{\mathrm{repr}}(R_{1},R_{2}) , is the bidirectional linear approximation error between H_{1},H_{2} (formally, Definition G.1 ). Defining similarity with a strict linear approximation is not necessary to develop our theory, but offers attractive computational tradeoffs. This is because we can formulate representation similarity as multi-target linear regression for which there are sharp approximations with small sample complexity. Alternatively, d_{\mathrm{repr}} could be defined with an arbitrary kernel ( 30 ) . We also make the following assumptions: (1) \rho_{i} is Lipschitz continuous with Lipschitz constant \Lip(\rho_{i}) . (2) There exists \Delta>0 such that \left\|h_{1}-h_{2}\right\|<\Delta over S . (3) The implementation distance d (see Equation ( 5.1 )) is induced by a fixed distillation function \Phi : for implementations F,\tilde{F} over the same variables {\mathbb{V}} , d(F,\tilde{F})\triangleq\lVert{\Phi{\mathbb{V}}^{F}-\Phi{\mathbb{V}}^{\tilde{F}}}\rVert . (4) There is a 1-Lipschitz map \Psi such that \lVert\Psi H_{1}-\Psi H_{2}\rVert\leq d_{\mathrm{repr}}(R_{1},R_{2}) and \lVert\Phi{\mathbb{V}}^{{\mathbb{F}}}-\Psi H\rVert<\omega . Assumption 1 stabilizes representations such that a small change in circuitry implies a proportional representation perturbation. In practice, this is reasonable since representations are usually distilled from neural networks via linear projection or by averaging the activations of several neural components ( 61 ) . Assumption 2 guarantees that functional differences between our target models are bounded, otherwise their interpretive inequivalence is trivial. Assumptions 3-4 state that our implementation metric d only measures distance using a set of observable components. Further, these components are recoverable from representations (up to \omega ). In this way, representations can serve as a surrogate for distances between circuits. For i\in\{1,2\} , suppose that A_{i} is an \eta_{i} -faithful interpretation of K_{i} . We first show that representation similarity between K_{1},K_{2} upperbounds interpretive equivalence. Main Result 1 . Let \Pi and \Pi_{\star} be alignment classes that map K_{1} ’s variables into A_{1},A_{2} ’s variables, respectively. Suppose there exists F^{\star}\in\Pi_{\star}^{-1}(A_{2}) such that its representation under \rho_{1} is an (L_{1},\delta_{1}) -representation. Then, the approximate interpretive equivalence of A_{1},A_{2} under alignments \Pi^{-1},\Pi_{\star}^{-1} is d_{\mathrm{interp}}(A_{1},A_{2})\leq\underbrace{\kappa(A_{1},K_{1},\Pi)+\kappa(A_{2},K_{1},\Pi_{\star})}_{\mathrm{(a)}}+\underbrace{2\omega+d_{\mathrm{repr}}(R_{1},R_{2})+\delta_{1}+\delta_{2}+\Delta}_{\mathrm{(b)}}. (6.1) Equation ( 6.1 ) explicits two dependencies: (a) measures interpretive compression of A_{1},A_{2} on K_{1} . When compression is low, circuits retain more information about interpretive differences. And since representations distill circuits, it is intuitive that representation similarity dominates the upperbound in this regime. In contrast, when compression is high, our examples in Section 5 illustrate that when circuits become uninformative, so should representations and their similarities; (b) jointly measures the quality of representations R_{1},R_{2} and their differences. If \delta_{1},\delta_{2}\gg 1 , representations lose meaningful linear structure even though they sufficiently abstract implementations of A_{1},A_{2} . This bound exposes a tension between the geometric richness of representations and our computational approach to their similarities. For example, representations lying on complex manifolds may not be well-described by our linear formulation of representation similarity ( 2 ; 52 ) . Thus, we hypothesize representations and their similarity criterion need to be simultaneously broadened to achieve more generality. We leave the implications of this for future work. Now, the lowerbound: Main Result 2 . Let \Pi and \Pi_{\star} be an alignment that map K_{1} ’s variables onto A_{1},A_{2} ’s variables. Suppose A_{1},A_{2} are {\epsilon} -approximate interpretively equivalent under the alignment classes \Pi_{\star},\Pi . Then, \displaystyle{\epsilon} \displaystyle\geq\frac{d_{\mathrm{repr}}(R_{1},R_{2})-(\delta_{1}+\delta_{2}+\Delta)}{\Lip(\rho_{1})+1}. (6.2) The terms in Equation ( 6.2 ) mirror the dynamics of Equation ( 6.1 ). If the implementation sets are {\epsilon} -close, then there cannot be too much discrepancy between the observed representations (see the numerator), unless it can be explained away by the slack terms (\delta_{1}+\delta_{2}+\Delta) . Along these lines, \Lip(\rho_{1}) (in the denominator) captures the intuition that when the mapping from circuits to representations is unstable, small implementation-level gaps can lead to larger representational differences. Beyond interpretive equivalence, Main Result 2 has implications for steering ( 59 ) . When the identifiability of representations is low ( \Lip(\rho_{1})\gg 1 and d_{\mathrm{repr}}(R_{1},R_{2})\ll 1 ), steering interventions are less likely to induce interpretive change, i.e. significant alterations to the representation space may see little change in model interpretation. In Main Result 3 , we show how this directly relates to Congruity . For i\in\{1,2\} , let us assume an arbitrary distribution over \Pi_{i}^{-1} and denote \mathbb{P}\triangleq\mathbb{P}_{1}\otimes\mathbb{P}_{1}\otimes\mathbb{P}_{2}\otimes\mathbb{P}_{2} the product distribution. Main Result 3 . Let p be the expected value of ReprDist in Algorithm 1 , F_{1},F_{2}\overset{\text{iid}}{\sim}\Pi_{1}^{-1}(A_{1}) , and F_{3}\sim\Pi_{2}^{-1}(A_{2}) then p\geq 1-\inf_{{\epsilon}>0}\mathbb{P}[d_{\mathrm{repr}}(F_{1},F_{2})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}]-\mathbb{P}[d_{\mathrm{repr}}(F_{1},F_{3})0 be arbitrary. If A is an \eta -interpretation of K , then A must also be an (\eta+{\epsilon}) -interpretation of K . In other words, if the faithfulness constraint is relaxed, then A could be a faithful interpretation for any circuit. This implies that any notion of interpretive equivalence must be trivial since every interpretation corresponds to every circuit. Appendix B Related Literature There has been extensive study in interpreting neural networks (specifically, language models). With respect to our paper, they can be organized into the following three buckets. Mechanistic Interpretability. Mechanistic Interpretability (MI) can be broadly broken into two distinct phases: first, identifying a minimal subset of the model’s computational graph that is responsible for a specific behavior—this process is also referred to as circuit discovery ( 46 ; 45 ; 12 ; 8 ; 24 ) ; second, assigning human-interpretable explanations to each of the extracted components—this process is generally referred to as mechanistic interpretability ( 10 ; 73 ; 79 ; 23 ; 40 ) . This paper focuses on the latter process; thus, when we invoke the term mechanistic interpretability, we are referring specifically to this second phase. For completeness, we briefly review circuit discovery as well. A model can be seen as a computational graph ( 14 ; 47 ) : G=(V,E) with vertices V and edges E\subset V\times V . A circuit is then any binary function f:E\to\{0,1\} . The optimal circuit f satisfies two properties: (1) it minimizes \sum_{e\in E}f(e) ; and (2) when edges not in the circuit e\in E,f(e)=0 are ablated, the model’s functional behavior remains unchanged. Although circuit discovery can be concretely formulated as an optimization problem, it is computationally intractable in its naive form ( 8 ; 1 ) . As a result, many relaxations and heuristics have been developed ( 12 ; 64 ; 43 ; 24 ; 8 ) . Nevertheless, there exists a solution set that can be statistically verified ( 57 ) . The process of mechanistic interpretability, which we focus on in this paper, presents different challenges. MI involves iteratively generating hypotheses about the interpretations for the interactions between different components, then testing those hypotheses through carefully crafted interventions ( 10 ; 18 ; 37 ; 62 , inter alia ) . MI methods can be further clustered into two approaches: top-down and bottom-up . Top-down approaches start by enumerating hypotheses about the possible algorithms the model could be implementing. Then, they iterate through these hypotheses and isolate those with the closest causal alignment ( 76 ; 18 ; 7 ; 72 ; 62 ) . On the other hand, bottom-up approaches are data-driven. They seek to directly find algorithmic explanations of the model’s mechanistic behaviors by analyzing activations or attention patterns ( 42 ; 79 ; 32 ; 3 ; 44 ) . Casual Abstraction. Causal abstraction seeks to formally characterize when a symbolic explanation faithfully explains a data-generating process (in our application, this data-generating process would be a model) ( 50 ; 51 ; 5 ; 4 ; 48 ) . Causal abstraction as defined in Definition D.3 seeks to align “low-level” models (neural networks) with “high-level” models (symbolic explanations). These theoretical constructions have driven the development of many top-down mechanistic interpretability methods such as 76 ; 18 ; 62 which directly verify this condition. While causal abstraction provides a necessary condition for a valid interpretation, recent works such as 37 argue that it alone is insufficient to fully characterize what constitutes a valid interpretation. Practical procedures that stem from this theory also fail to address this. This highlights the need for additional theoretical frameworks to complement causal abstraction. On the other hand, causal abstraction represents a hard equivalence: two models either are or are not causal abstractions of each other. The binary nature of this definition fails to capture our intuition that explanations can vary in quality or completeness. Some interpretations may better capture the model’s behavior than others, or may only explain a subset of the model’s functionality. The hard equivalence of causal abstraction makes it difficult to reason about these partial or imperfect explanations. As a result, framemworks such as 4 ; 36 soften this criterion by introducing distance metrics on the total settings of the “high-level” model. In practice, these approaches face two main drawbacks: 1. It is challenging to define meaningful distance metrics when high-level interpretations involve discrete symbols and symbolic reasoning. 2. Comparing interpretations between models requires fully interpreting each model first, which forces a computationally expensive top-down analysis approach. Our framework addresses these limitations by focusing on implementations rather than interpretations directly. Since neural network computations are fundamentally real-valued, we can leverage natural distance metrics in their computational space. Additionally, by analyzing families of implementations rather than requiring complete interpretations, we can compare models’ interpretive similarity without the overhead of fully interpreting each model first. Representation Similarity. Representation similarity in neural networks has emerged as a fundamental research area whcih addresses how internal representations correspond across different models, domains, and biological systems ( 30 ) . We provide a brief overview of techniques to measure representation similarity, but encourage the reader to refer to 61 ; 29 for a comprehensive survey of the techniques within this field. Representations of deep neural networks have shown to exhibit an array of powerful properties. They have given insights into training dynamics and the generalization capabilities of models ( 67 ; 77 ; 58 ) and also provide a tractable method to compare the inductive biases between different models ( 74 ; 60 ) . These representations have also shown to admit rich geometric properties ( 70 ) . In this paper, we measure representation through a learned linear regression. These techniques have been used extensively to compare different neural architecture and even human-language model similarity ( 69 ; 41 ) . Appendix C Experimental Details In this section, we detail our experimental methods. We first review constructs of RASP 75 . Then, we demonstrate that RASP provides an interface to craft interpretations, and through methods like 18 ; 22 we can enumerate implementations of these interpretations. Lastly, we describe our experimental hyperparameters. C.1 RASP Interpretations Background on RASP Programs. The R estricted A ccess S equence P rogramming (RASP) language is a functional programming model designed to capture the computational behavior of Transformer architdectures ( 75 ) . RASP programs have shown use in mechanistic interpretability both as an effective benchmarking tool for faithfulness ( 12 ; 24 ) and as a method to develop “inherently” interpretable language models ( 15 ) . Another line of work uses it (and other similar methods) as a proof technique to reason about the Transformer architecture’s generalizability on a host of tasks ( 75 ; 39 ; 20 ) . In this paper, we focus on RASP’s applications in interpretability. RASP programs operate on two primary types of variables: s-ops , representing the input sequence, and selectors , corresponding to attention matrices. These variables are manipulated through two fundamental instructions: elementwise operations and select-aggregate. Elementwise operations simulate computations performed by a multilayer perceptron (MLP), while select-aggregate combines token-level operations, modeling the functionality of attention heads. Every RASP program is equipped with two global variables tokens and indices, essentially primitive s-ops . tokens maps strings into their token representations: token("code") = ["c", "o", "d", "e"] indices("code") = [0, 1, 2, 3] On the other hand, indices map n -length strings into their indices. That is, a list of [0,1,\ldots,n-1] . Elementwise operations can be computed through composition. That is, (3 * indices)("code") = [0, 3, 6, 9] (sin(indices)) = [sin(0), sin(1), sin(2), sin(3)] Tokens and their indices can also be mixed through selection matrices which are represented through the s-op select . This operations captures the mechanism of the QK-matrix. It takes as input two sequences K,Q , representing keys and queries respectively, and a Boolean predicate p and returns a matrix S of size |K|\times|Q| such that S_{ij}=p(K_{j},Q_{i}) . Then, the OV-circuit can be computed through the select-aggregate operation, which performs an averaging over an arbitrary sequence with respect to the aforementioned selection matrix. For example, \texttt{aggregate}\left(\begin{bmatrix}1&0&0\\ 0&0&0\\ 1&1&0\end{bmatrix},\begin{bmatrix}10&20&30\end{bmatrix}\right)=\begin{bmatrix}10&0&15\end{bmatrix}. The previous example is directly lifted from 34 . Compiling RASP Programs. The power of RASP programming lies in its ability to translate any RASP program into a Transformer, a process known as compilation . As described in 34 , this involves a two-stage approach. First, a computational graph is constructed by tracing the s-ops in the program, identifying how these operations interact with and modify the residual stream. Elementwise operations are converted into MLP weights, and individual components are heuristically assigned to Transformer layers. For further details, we refer the reader to 34 . As observed by 34 , this compilation through “translation” introduces inefficiencies. Specifically, the heuristic layer-assignment of RASP components results in Transformers that often contain more layers than they need to have. Moreover, since RASP enforces the use of categorical sequences and hard attention (we only allow Boolean predicates) it requires various s-ops to lie orthogonal to each other after embedding as Transformer weights. As a result, this leads to a much larger embedding dimension that is usually observed in actual Transformers ( 13 ) . Thus, 34 proposes to compress this dimension through a learned projection matrix. The caveat is that this transformation largely not faithful to the original program (measured through cosine similarity of the outputs at individual layers). 15 takes a different approach, addressing the inherent difficulty of writing RASP programs. To overcome this challenge, the authors propose a method for directly learning RASP programs. This is achieved by constraining the space of learnable weights to those that compile into valid RASP programs, ensuring outputs with categorical variables and hard attention mechanisms. Optimizing over this constrained hypothesis class is performed through a continuous relaxation using the Gumbel distribution ( 27 ) . RASP Benchmarks. 65 is a dataset of RASP programs that have been generated by GPT-4. It contains 121 RASP programs. 22 provides 86 RASP programs and compiled Transformers. The compiled Transformers are claimed to be more realistic than Tracr compiled ones as instead of performing compression using a linear projection, they leverage strict interchange intervention training essentially aligning the intervention effects of the compressed and uncompressed model. This is similar in vein to many existing techniques on causal abstraction 48 ; 78 ; 36 . In our paper, we leverage the curated dataset 22 to craft and compose our interpretations. Figure 4: Pipeline for generating implementations for our interpretations (i.e. RASP language models). We first construct RASP programs, then using the procedure introduced in 34 , we compile these programs into Transformer models. These Transformers exclusively have hard attention. Moreover, their architecture is minimal (containing only the necessary components to fully implement the given RASP program). We then apply the procedure introduced in 22 to translate these Tracr-Compiled Transformers into “real” Transformers: ones whose weight distribution matches those trained with stochastic gradient descent; these translated models also contain more C.2 Strict Interchange Intervention Training To evaluate our methods, we need a way to verifiably ellicit different mechanisms on the same task. Let us first fix some task. Then, we proceed with the following steps: 1. Using 15 , we learn several different explicit Transformer programs (source of randomness). We can check that they are different by looking explicitly at the Transformer programs. 2. Using 22 and 18 to get different mechanistic realizations of this abstract Transformer program. To generate different mechanistic instantiations of the same interpretation across architectures, we use the following procedure: First, we take a Tracr-Compiled Transformer model and initialize a new random model with at least as many layers and attention heads. Two models share the same interpretation if the Tracr-Compiled transformer is a causal abstraction of our mechanistic instantiation. We leverage this insight by softening Definition D.3 and incorporating it directly into our objective function. For a detailed treatment of this approach, we refer readers to 18 ; 22 ; 62 . To ensure diversity in our implementations, we vary the architecture hyperparameters significantly: models contain between 2-6 layers, 2-8 attention heads, and embedding dimensions ranging from 32 to 2048. This creates a rich set of instantiations with widely varying model capacities. As a result, many mechanisms in these larger models may not contribute to the core interpretations, leading to substantial interpretive compression. C.3 Interpretations for Permutation Detection To implement this task, we proceed with the following six interpretations. As discussed in Section 3.1 , they can be roughly divided into two groups: sort- and counting-based. Interpretation 1. Sort the sequence ; compute the difference between each element and the next one; (3) Check if all elements of the sequence are equal. Interpretation 2. Sort the sequence in descending order; increment each element by its index; check if all elements of the sequence are equal. Interpretation 3. Sort the sequence ; interleave the list with the same list in reverse order; sum each number with the number next to it; (4) Check if all elements of the list are equal. Interpretation 4. Sort the sequence ; check if the list contains alternating even and odd elements. Interpretation 5. Check if at least two elements in the list are equal; sum each element with the next one in the list; check if all elements of the list are equal. Interpretation 6. Replace each element with the number of elements less than it in the sequence; check if at least two numbers are the same. For each of the subroutines of these interpretations, 22 implements RASP programs for them. Thus, we simply compose them together to create the resulting interpretations. C.4 Generating Implementations for IOI Herein, we describe how we generate implementations for the IOI task. For each model, we first identify a subset of the model’s attention heads that drive functional behavior. We do this through causal mediation analysis (which is well-documented in 73 and 66 ). For example, in GPT2-small this corresponds to attention heads (0.1, 0.10, 2.2, 4.11, 5.5, 6.9, etc., formatted as [layer index].[head index]). The complement of this set are then components that are “unused” (see Section 2 ). For any input like (a) we construct a counterfactual input like (b) see below: (a) When John went to the store with Mary, John gave a bottle of milk to (b) When Mary went to the store with John, Mary gave a bottle of milk to Then, we generate implementations by ablating the unused components. This is done by running the model on inputs like (a) and then for attention head in the ablation subset we patch in the output of the attention head as if it had been run on inputs like (b) . C.5 Function Vectors and Parts-of-Speech Identification Function vectors have been found to drive in-context learning behavior ( 68 ) . We leverage this concept to find the circuit that is responsible for in-context parts-of-speech identification. We use the Penn TreeBank dataset and consider the POS subtask. We sample n=1000 points. We create counterfactual inputs by shuffling the in-context labels. For example, Clean Input: tree:noun, run:verb, quickly:adverb, ire: Corrupted Input: tree:adverb, run:noun, quickly:verb, ire: Then, we first run the model on the clean input and store all attention head output activations on the token “ire.” We then run the model on the corrupted input and patch in the stored attention head activations from the “ire” token. Finally, we compute the difference in logit difference loss of the generated next tokens before/after this patching operation. The results for each attention head are shown in Figure 5 . It seems that attention heads in the early/middle layers are most important for POS (0.1, 0.4, 0.6, 4.11, 5.1, 5.5, 6.8, 6.9, 6.10, 7.11). Figure 5: Activation patching results for each attention head. The coloring indicates the % of performance this individual attention head contributes to POS performance. Then, we compute the function vector by \frac{1}{|A|}\sum_{a\in A}\sum_{k=1}^{n}a(x_{k}), (C.1) where A is the set of attention heads that we have deemed important and x_{k} is our dataset. Essentially, we are averaging the activations across all important attention heads. Appendix D Formal Definitions We provide a glossary (see Table 1 and Table 2 for a summary) for our technical terms and notations in the main text along with their precise, formal definitions. Table 1: Glossary of terms. Term Intuitive Explanation Formal Definition Causal/Computation Graph A graph that describes a model’s computational dependencies Definition 4.1 Variables Nodes in a model’s computation graph that store latent computation results Definition 4.1 Circuit A subset of the model’s computational graph that on its own can fully recreate the model’s input-output behavior Definition D.1 Mechanistic Explanation A symbolic algorithm, we can understand, that explains the model’s computation N/A Causal Abstraction Mapping from low-level variables to higher-level ones that preserve low-level causal relationships Definition D.3 Interpretation A causal graph much smaller than the model’s computation graph that explains the model’s computation graph Definition D.4 Representations A causal graph that is a chain that abstracts a model’s circuit Definition 4.3 Alignment A mapping between two causal graphs that preserves meaningful causal relations between variables Definition D.3 Intervention A modification to the model’s computation graph Definition D.2 Alignment Class A set of alignments that map to the same interpretation Definition 5.1 Implementation Sets of circuits that share the same interpretation Definition 5.1 Interpretive Equivalence Measure of distance between sets of implementations Definition 5.2 Interpretive Compression Measure of diameter of a set of implementations Definition 5.3 Congruity Approximation for interpretive equivalence Definition H.1 Representation Similarity Extent to which one set of representations can be linearly transformed into the other and vice versa Definition G.1 Table 2: Notations used throughout our formal treatment. This is partially adapted from both 16 and 55 . Symbol Meaning \Sigma An alphabet \Sigma^{\star} Set of all finite strings constructed from \Sigma S A subset of \Sigma^{\star} A An interpretation K A circuit R A representation {\mathbb{V}} Set of variables in a causal model U Input variable to causal model {\mathbb{F}} Functions in causal model that determine variable values \succ Partial ordering over variables of causal model induced by {\mathbb{F}} {\mathbb{V}}^{\mathbb{F}}(u) Solution of all variables to input u {\bm{v}}_{k}^{{\mathbb{F}}}(u) Solution of {\bm{v}}_{k} to input u \pi An alignment between variables of two causal models \Pi A class of alignments \Pi^{-1}(A) The set of implementations of A induced by alignments \Pi d_{\mathrm{interp}} Interpretive equivalence d_{\mathrm{H}} Hausdorff distance \kappa(A,K,\Pi) Interpretive compression d_{\mathrm{repr}} Representation similarity \Lip(f) Lipschitz constant of f \overset{\mathrm{i.i.d.}}{\sim} Sample i.i.d from some distribution Definition D.1 (Circuit) . An m -circuit of h on S is a causal graph with m components that satisfies four properties: 1. Input variable is S -valued: U\in S 2. Hidden variables are real-valued: for each {\bm{v}}_{k}\in{\mathbb{V}} , {\bm{v}}\in\mathbb{R}^{n_{k}} 3. Existence of a terminal output variable: {\bm{v}}_{\rm{out}}\in{\mathbb{V}} such that {\bm{v}}_{\rm{out}}\in\mathbb{R}^{d} , and there does not exist {\bm{v}}^{\prime}\in{\mathbb{V}} where {\bm{v}}_{\rm{out}}\succ{\bm{v}}^{\prime} 4. Sufficient description of h on S : for each s\in S , {\bm{v}}_{\rm{out}}^{\mathbb{F}}(s)=h(s) Remark . For any h and m\geq 1 , an m -circuit of h must exist by the trivial causal model. To see this, let U\in S and {\bm{v}}_{\rm{out}}\triangleq h(U) . Then, for any 1\leq i\leq m-1 , define {\bm{v}}_{i}\triangleq U\mapsto 1 . In other words, we construct a causal graph with effectively one variable, letting the other hidden variables be constant functions. Remark . m -circuits are also not unique. For instance, a joint change of bases to any {\bm{v}}_{k}\in{\mathbb{V}}\setminus{\bm{v}}_{\rm{out}} and f_{k} results in a new causal model that preserves functional faithfulness to h ( 49 ; 18 ) . Definition D.2 (Intervention) . Let ({\mathbb{V}},U,{\mathbb{F}},\succ) be a causal model. An intervention of {\bm{v}}_{k}\in{\mathbb{V}} , {\rm{do}}({\bm{v}}_{k}\leftarrow\tilde{f}) , redefines {\bm{v}}_{k}\triangleq\tilde{f}(\tilde{P}a({\bm{v}}_{k}),U) , for \tilde{P}a({\bm{v}}_{k})\subset\{{\bm{v}}\in{\mathbb{V}}:{\bm{v}}\succ{\bm{v}}_{k}\} . We denote the set of interventions on {\bm{v}}_{k} as {\mathcal{I}}({\bm{v}}_{k}) . In other words, we replace the process used to derive {\bm{v}}_{k} ’s value, f , with a new function \tilde{f} . Given any circuit, one type of concrete intervention is activation patching . For a set of hidden variables, we patch its solutions using values derived from some other input ( 71 ; 73 ; 25 , inter alia ) . Interventions often serve as a check of consistency between circuits and their interpretations ( 5 ; 17 ) . We define this notion next. Definition D.3 (Abstraction) . Let K_{1}\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ_{1}),K_{2}\triangleq(\tilde{\mathbb{V}},\tilde{U},\tilde{\mathbb{F}},\succ_{2}) be causal models. K_{2} abstracts K_{1} if there exists surjective mappings \pi:\mathrm{SubsetsOf}({\mathbb{V}})\to\tilde{\mathbb{V}},\omega:{\mathcal{I}}({\mathbb{V}})\to{\mathcal{I}}(\tilde{\mathbb{V}}) that satisfies for all {\bm{v}}_{k}\in{\mathbb{V}},\tilde{\bm{v}}_{k}\in\tilde{\mathbb{V}},{\rm{do}}({\bm{v}}_{k}\leftarrow f)\in{\mathcal{I}}({\bm{v}}_{k}) : \displaystyle\tilde{\bm{v}}_{k}=\pi\left(\bigcup_{{\bm{v}}_{k}\in\pi^{-1}(\tilde{\bm{v}}_{k})}f_{k}(Pa({\bm{v}}_{k}),U)\right), (D.1) \displaystyle\pi({\rm{do}}({\bm{v}}_{k}\leftarrow f))={\rm{do}}(\pi({\bm{v}}_{k})\leftarrow\omega(f)). (D.2) We call \pi an alignment . Equation ( D.1 ) describes an observational consistency constraint: if K_{2} abstracts K_{1} , then by only observing the hidden variables of K_{1} we can infer all hidden values of K_{2} . Viewed under this lens, the alignment \pi projects the fine-grained variables {\mathbb{V}} to the coarse-grained ones \tilde{\mathbb{V}} . On the other hand, Equation ( D.2 ) is an intervention consistency constraint—stating that \pi and \omega must commute. In words, this means that if we apply the intervention {\rm{do}}({\bm{v}}_{k}\leftarrow f) to yield a new causal model and then abstract that causal model through alignment \pi , this is equivalent to abstracting the causal model and then applying the transformed intervention on the high-level model. This implies that all of the causal relationships between variables in K_{2} can be constructed by relationships between variables in K_{1} . Under this light, an interpretation of a circuit is simply an abstraction. Definition D.4 (Interpretation) . A causal model A\triangleq({\mathbb{V}}^{\star},U,{\mathbb{F}}^{\star},\succ) that abstracts a circuit K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) is an \eta -faithful interpretation of K if there exists an output variable {\bm{v}}^{\star}_{\rm{out}}\in{\mathbb{V}}^{\star} such that \lVert{{\bm{v}}^{\star,{\mathbb{F}}}_{\rm{out}}-{\bm{v}}^{\mathbb{F}}_{\rm{out}}}\rVert<\eta for the output variable {\bm{v}}_{\rm{out}}\in{\mathbb{V}} . Remark . Interpretations are abstractions of circuits . However, Definition D.4 does not rely on the fact that circuits have real-valued hidden variables. Since interpretations also have an output variable, one could further construct an interpretation of an interpretation (Proposition F.2 ). Remark . For any circuit, K is an interpretation of itself (under the identity abstraction), albeit this is not very useful. Remark . We do not require that A ’s variables be real-valued, nor do we require that there exists a metric over the solutions of A (in contrast to K ). This is compatible with the conventions in mechanistic interpretability where often an interpretation is a symbolic description of K that may not necessarily be performing numeric computation. Appendix E Interpretations Must Be Abstraction Dependent Our notion of an interpretation is dependent on a fixed level of abstraction (the variable set {\mathbb{V}} ). This requirement may seem stringent, especially since we colloquially think of interpretations (or algorithms) as abstraction-independent entities. Further, abstraction- dependent interpretations immediately give rise to non-identifiability , where a single interpretation may correspond many implementations (at different levels of abstraction) and a single circuit may also correspond many interpretations, each at a different level of abstraction. In this section, we argue that non-identifiability is a necessary price we must pay for a meaningful comparison between interpretations. Our main result here is that if defined over an existence statement of {\mathbb{V}} interpretive equivalence must collapse to functional equivalence. Theorem E.1 . For i\in\{1,2\} , let S\subset\Sigma^{\star} and h_{i} be a language model such that h_{1}(S)=h_{2}(S) . Suppose that h_{i} admit a circuit K_{i}\triangleq({\mathbb{V}}_{i},U,{\mathbb{F}}_{i},\succ) . Then, 1. For i\in\{1,2\} , there exists an abstraction of K_{i} , A_{i}\triangleq({\mathbb{V}}_{A},U,{\mathbb{F}}_{i}^{A},\succ) such that there is a bijective translation 12 12 12 In general, a bijective translation is neither stronger nor weaker than an abstraction ( 16 ) . (Definition E.1 ) between A_{1},A_{2} . 2. There is a language model h^{\star} with circuit K^{\star} such that h^{\star}(S)=h_{1}(S)=h_{2}(S) and K_{1},K_{2} are abstractions of K^{\star} . 3. If both K_{1},K_{2} are not minimal (i.e. there exists at least one causal variable such that any intervention does not affect its output), then there exists a bijective translation between K_{1},K_{2} . Since we do not require circuits to be minimal (see Section 4 ), statement (3) implies that we must restrict the set of permissible alignments. Otherwise, any non-minimal circuit is 0-interpretively equivalent to any other non-minimal circuit (see Proposition F.4 ). Recall that for any causal model K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) , given an input U\leftarrow{\bm{u}} , there exists a solution to each of its variables which we denote as {\mathbb{V}}^{\mathbb{F}}({\bm{u}}) . If K is an m -circuit, then {\mathbb{V}}^{{\mathbb{F}}}({\bm{u}})\in\Sigma^{\star}\times\mathbb{R}^{n_{1}}\times\ldots\times\mathbb{R}^{n_{m}}\times\mathbb{R}^{|\Sigma|}. (E.1) For brevity, we slightly abuse the notation and denote {\mathbb{V}}^{{\mathbb{F}}}\triangleq\mathbb{R}^{n_{1}}\times\ldots\times\mathbb{R}^{n_{m}}\times\mathbb{R}^{|\Sigma|} . In words, a bijective translation is a causally-consistent, bijective mapping between the solutions of two causal models. Essentially, if one model is a bijective translation of another, we can think of it simply as a relabeling of the other one. Definition E.1 . For i\in\{1,2\} , let K_{i}\triangleq({\mathbb{V}}_{i},U,{\mathbb{F}}_{i},\succ) be a causal model. For any {\bm{u}}\in S , \tau:{\mathbb{V}}_{1}^{{\mathbb{F}}_{1}}\to{\mathbb{V}}_{2}^{{\mathbb{F}}_{2}} is a bijective translation if \tau satisfies Equation ( D.1 ) and there exists \omega:{\mathcal{I}}({\mathbb{V}}_{1})\to{\mathcal{I}}({\mathbb{V}}_{2}) that satisfies Equation ( D.2 ). Proposition E.2 . For any n,m\in\mathbb{Z}^{+} , there exists a bijection between \mathbb{R}^{n} and \mathbb{R}^{m} . We are now ready for the proof of Theorem E.1 . Proof. We prove all claims constructively. 1. For i\in\{1,2\} , let A_{i} be the trivial causal model associated with h_{i} : it has two causal variables U,{\bm{v}}_{\rm{out}} such that {\bm{v}}_{\rm{out}}\triangleq h_{i}(U) . Consider an an alignment map \pi_{i} such that \pi_{i}({\bm{v}}_{\rm{out}})={\bm{v}}_{\rm{out}} , \pi_{i}(U)=U , and for all other {\bm{v}}\in{\mathbb{V}}_{i},\pi_{i}({\bm{v}})=\perp (i.e. \pi is undefined on other variables in {\mathbb{V}}_{i} ). Define the intervention map \omega to be the identity if and only if {\rm{do}}({\bm{v}}_{k}\leftarrow f)\in{\mathcal{I}}({\mathbb{V}}_{i}) is an intervention on U or {\bm{v}}_{\rm{out}} . Then, it follows that the only valid interventions defined by \omega are those directly on the input and output variables. It is easy to see that both Equation ( D.1 ) and ( D.2 ) hold. Since h_{1}=h_{2} , A_{1},A_{2} constructed in this way are the same causal model. Thus, the identity map forms a trivial bijective translation between them. 2. Let K^{\star}\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) such that {\mathbb{V}}={\mathbb{V}}_{1}\cup{\mathbb{V}}_{2}\cup\{U,{\bm{v}}_{\rm{out}}\} . Construct {\mathbb{F}} such that it runs K_{1},K_{2} in parallel and then averages their output. Concretely, let {\bm{v}}^{1}_{\rm{out}},{\bm{v}}^{2}_{\rm{out}}\in{\mathbb{V}} be the output variables of {\mathbb{V}}_{1},{\mathbb{V}}_{2} , respectively. Also let U^{1},U^{2}\in{\mathbb{V}} be their input variables. Define Pa(U_{1})=Pa(U_{2})=\{U\}\qquad Pa({\bm{v}}_{\rm{out}})=\{{\bm{v}}^{1}_{\rm{out}},{\bm{v}}^{2}_{\rm{out}}\}, and their transition functions f_{U_{1}}(U)=f_{U_{2}}(U)=U\qquad f_{{\bm{v}}_{\rm{out}}}(\{{\bm{v}}^{1}_{\rm{out}},{\bm{v}}^{2}_{\rm{out}}\},U)=\frac{1}{2}\left(f_{{\bm{v}}^{1}_{\rm{out}}}(Pa({\bm{v}}^{1}_{\rm{out}}),U)+f_{{\bm{v}}^{2}_{\rm{out}}}(Pa({\bm{v}}_{\rm{out}}^{2}),U)\right). Clearly, K^{\star} is a valid circuit for some language model h=h_{1}=h_{2} . Let \pi_{1},\pi_{2} be the identity alignments from K^{\star}\to K_{i} . By the same constructive above for \omega , K_{1},K_{2} must be both be an abstraction for K^{\star} . 3. Let \tau be the identity function that maps {\bm{v}}_{\rm{out}}^{1}\in{\mathbb{V}}_{1}\to{\bm{v}}_{\rm{out}}^{2}\in{\mathbb{V}}_{2} . By Definition 4.2 , for i\in\{1,2\} , any variable {\bm{v}}_{k}\in{\mathbb{V}}_{i}\setminus\{U,{\bm{v}}_{\rm{out}}^{i}\} takes on values in \mathbb{R}^{m_{k}} for m_{k}\in\mathbb{Z}^{+} . By assumption, for both models K_{1},K_{2} , there exists at least one causal variable that is “unused.“ Denote this variable as {\bm{v}}_{\mathrm{unused}}^{i} . By Proposition E.2 , there exists a bijection that maps \sigma:{\mathbb{V}}_{1}\setminus\{U,{\bm{v}}_{\rm{out}},{\bm{v}}_{\mathrm{unused}}^{1}\}\to{\bm{v}}_{\mathrm{unused}}^{2} . Composing \sigma\circ\tau yields a bijection from {\mathbb{V}}_{1}^{{\mathbb{F}}_{1}}\to{\mathbb{V}}_{2}^{{\mathbb{F}}_{2}} . It is easy to see that this is a bijective translation which satisfies Equation ( D.1 ) and ( D.2 ). By symmetry, the other direction also holds. ∎ Appendix F Properties of Interpretive Equivalence and Compression As discussed in Section 5 , an interpretation of a model need not be a lossless description of that model. For many applications an interpretable explanation is necessarily lossy with respect to the learned model. In this sense, an abstraction alone is too restrictive as it requires any interpretation to preserve the exact functional behavior of the model. Herein, we explore some properties of interpretive equivalence, compression and illustrate their similarities and differences with causal abstraction. We first show that both implementations and interpretations compose. That is, an implementation of an implementation is also an implementation and an interpretation of an interpretation is also an interpretation . Lemma F.1 . For i\in\{1,2,3\} , let K_{i}\triangleq({\mathbb{V}}_{i},U,{\mathbb{F}}_{i},\succ_{i}) be causal models. Let \pi_{1,2} be an alignment from K_{1}\to K_{2} and \pi_{2,3} an alignment from K_{2}\to K_{3} . Then, K_{3} is an abstraction of K_{1} with an alignment map \pi_{2,3}\circ\pi_{1,2} . Remark F.1 (On Composing Alignments) . We are liberally abusing the notation here. Technically, the composition \pi_{2,3}\circ\pi_{1,2} is not well defined since the alignments have incompatible types: \pi_{1,2}:\mathrm{SubsetsOf}({\mathbb{V}}_{1})\to{\mathbb{V}}_{2} and \pi_{2,3}:\mathrm{SubsetsOf}({\mathbb{V}}_{2})\to{\mathbb{V}}_{3} . We write \pi_{2,3}\circ\pi_{1,2} to mean the process of starting from the causal graph K_{1} applying the alignment \pi_{1,2} to yield a new causal graph K_{2} and then applying the alignment \pi_{2,3} to yield the causal graph K_{3} . This is possible because an alignment alone is sufficient to induce an abstraction (see 16 16 , Remark 32 for a construction). Proof. We first check Equation ( D.1 ). Since \pi_{2,3} is an alignment that enables K_{2} to abstract K_{1} , for any {\bm{v}}_{k}^{3}\in{\mathbb{V}}_{3} , \displaystyle{\bm{v}}_{k}^{3} \displaystyle=\pi_{2,3}\left(\bigcup_{{\bm{v}}_{k}^{2}\in\pi_{2,3}^{-1}({\bm{v}}_{k}^{3})}f_{k}^{2}(Pa({\bm{v}}_{k}^{2}),U)\right). By Definition 4.1 , {\bm{v}}_{k}^{2}=f_{k}^{2}(Pa({\bm{v}}_{k}^{2}),U) . Then, because K_{2} is an abstraction of K_{1} through the alignment map \pi_{1,2} , it follows that \displaystyle{\bm{v}}_{k}^{3} \displaystyle=\pi_{2,3}\left(\bigcup_{{\bm{v}}_{k}^{2}\in\pi_{2,3}^{-1}({\bm{v}}_{k}^{3})}\pi_{1,2}\left(\bigcup_{{\bm{v}}_{k}^{1}\in\pi_{1,2}^{-1}({\bm{v}}_{k}^{2})}f_{k}^{1}(Pa({\bm{v}}_{k}^{1}),U)\right)\right), \displaystyle=(\pi_{2,3}\circ\pi_{1,2})\left(\bigcup_{{\bm{v}}_{k}^{2}\in\pi_{2,3}^{-1}({\bm{v}}_{k}^{3})}\left(\bigcup_{{\bm{v}}_{k}^{1}\in\pi_{1,2}^{-1}({\bm{v}}_{k}^{2})}f_{k}^{1}(Pa({\bm{v}}_{k}^{1}),U)\right)\right), \displaystyle=(\pi_{2,3}\circ\pi_{1,2})\left(\bigcup_{{\bm{v}}_{k}^{1}\in(\pi_{2,3}\circ\pi_{1,2})^{-1}({\bm{v}}_{k}^{3})}f_{k}^{1}(Pa({\bm{v}}_{k}^{1}),U)\right). It remains to show that the mappings between interventions holds. Denote \omega_{1,2}:{\mathcal{I}}({\mathbb{V}}_{1})\to{\mathcal{I}}({\mathbb{V}}_{2}) and \omega_{2,3}:{\mathcal{I}}({\mathbb{V}}_{2})\to{\mathcal{I}}({\mathbb{V}}_{3}) . We claim that Equation ( D.2 ) is satisfied with \omega_{2,3}\circ\omega_{1,2} . Since \omega_{1,2} and \omega_{2,3} satisfy Equation ( D.2 ), \displaystyle\pi_{2,3}({\rm{do}}({\bm{v}}_{k}^{2}\leftarrow f)) \displaystyle={\rm{do}}(\pi_{2,3}({\bm{v}}_{k}^{2})\leftarrow\omega_{2,3}(f)), \displaystyle\pi_{2,3}(\pi_{1,2}({\rm{do}}({\bm{v}}_{k}^{1}\leftarrow f))) \displaystyle={\rm{do}}(\pi_{2,3}(\pi_{1,2}({\bm{v}}_{k}^{1}))\leftarrow\omega_{2,3}(\omega_{1,2}(f))), \displaystyle(\pi_{2,3}\circ\pi_{1,2})({\rm{do}}({\bm{v}}_{k}^{1}\leftarrow f)) \displaystyle={\rm{do}}((\pi_{2,3}\circ\pi_{1,2})({\bm{v}}_{k}^{1})\leftarrow(\omega_{2,3}\circ\omega_{1,2})(f)), where {\bm{v}}_{k}^{2}\in{\mathbb{V}}_{2} , {\bm{v}}_{k}^{1}\in\pi_{1,2}^{-1}({\bm{v}}_{k}^{2}) , and f are chosen arbitrarily. ∎ Proposition F.2 . Let K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) be a circuit. Let A\triangleq({\mathbb{V}}_{A},U,{\mathbb{F}}_{A},\succ) be an \eta -faithful interpretation of K through an alignment \pi . Let A^{\star}\triangleq({\mathbb{V}}_{A}^{\star},U,{\mathbb{F}}_{A}^{\star},\succ) be an \eta -faithful interpretation of A through an alignment \pi^{\star} . Then, A^{\star} is an 2\eta -faithful interpretation of K through an alignment \pi^{\star}\circ\pi . Proof. There exists an alignment \pi from K\to A and an alignment \pi^{\star} A\to A^{\star} . By Lemma F.1 , A^{\star} is an abstraction of K with the alignment map \pi^{\star}\circ\pi . There exists a variable {\bm{v}}_{\rm{out}}^{A^{\star}}\in{\mathbb{V}}_{A}^{\star},{\bm{v}}_{\rm{out}}^{A}\in{\mathbb{V}}_{A} such that \left\|{\bm{v}}_{\rm{out}}^{A^{\star}}-{\bm{v}}_{\rm{out}}^{A}\right\|<\eta and a variable {\bm{v}}_{\rm{out}}\in{\mathbb{V}} such that \left\|{\bm{v}}_{\rm{out}}^{A}-{\bm{v}}_{\rm{out}}\right\|<\eta . By triangle inequality, \left\|{\bm{v}}_{\rm{out}}^{A^{\star}}-{\bm{v}}_{\rm{out}}\right\|<2\eta . ∎ Proposition F.3 . Let K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) be a circuit. Let A be an \eta -interpretation of K and \Pi be an alignment class that maps K ’s variables onto A ’s. Let A^{\star} be an \eta -interpretation of A and \Pi_{\star} be an alignment class that maps A ’s variables onto A^{\star} ’s. Then, \Pi^{-1}(\Pi_{\star}^{-1}(A_{\star})) are implementations of A^{\star} . Proof. The proof is trivial and directly follows from Proposition F.2 . ∎ We now relate interpretive equivalence to causal abstraction and show necessary and sufficient conditions for interpretive equivalence using abstractions. For i\in\{1,2\} , let K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) be circuits with \eta -interpretations A_{i}\triangleq({\mathbb{V}}_{i}^{A},U,{\mathbb{F}}_{i}^{A},\succ) . Let \Pi_{i} be alignment classes that map K_{i} ’s variables onto A_{i} ’s variables. Proposition F.4 . Suppose A_{1} is an abstraction of A_{2} and vice versa through alignment maps \pi_{1,2},\pi_{2,1} . A_{1},A_{2} are 0-interpretively equivalent if \pi_{2,1}\circ\Pi_{1}\subset\Pi_{2} and \pi_{1,2}\circ\Pi_{2}\subset\Pi_{1} . Proof. We show that \Pi_{1}^{-1}(A_{1})\subset\Pi_{2}^{-1}(A_{2}) . Take any F\in\Pi_{1}^{-1}(A_{1}) . By Definition 5.1 , there exists \pi_{1}\in\Pi_{1} that forms an abstraction from ({\mathbb{V}},U,F,\succ)\to({\mathbb{V}}_{1}^{A},U,{\mathbb{F}}_{1}^{A},\succ) . By Lemma F.1 , \pi_{2,1}\circ\pi_{1} is an alignment that forms an abstraction from ({\mathbb{V}},U,F,\succ)\to({\mathbb{V}}_{2}^{A},U,{\mathbb{F}}_{2}^{A},\succ) . Since \pi_{2,1}\circ\pi_{1}\in\Pi_{2}^{-1}(A_{2}) , this implies that F\in\Pi_{2}^{-1}(A_{2}) . Applying the same argument in the other direction, \Pi_{2}^{-1}(A_{2})\subset\Pi_{1}^{-1}(A_{1}) . Therefore, \Pi_{1}^{-1}(A_{1})=\Pi_{2}^{-1}(A_{2}) and by Definition 5.2 , A_{1},A_{2} are 0-interpretively equivalent. ∎ Proposition F.5 . Let \Pi_{i}^{-1}(A_{i}) be closed. Suppose A_{1},A_{2} are 0-interpretively equivalent. Suppose there exists \pi_{i}\in\Pi_{i} that each admits a section , s_{i}:{\mathbb{V}}_{i}^{A}\mapsto\mathrm{SubsetsOf}({\mathbb{V}}_{i}) , such that \pi_{i}(s_{i}({\bm{v}}))={\bm{v}} , for any {\bm{v}}\in{\mathbb{V}}_{i}^{A} . Suppose that there exists some \pi_{1}^{\prime},\pi_{2}^{\prime} such that \pi_{2}^{\prime}\circ s_{1}\circ\pi_{1}\in\Pi_{2} and \pi_{1}^{\prime}\circ s_{2}\circ\pi_{2}\in\Pi_{1} . Then, A_{1} is an abstraction of A_{2} and vice versa with alignment maps \pi_{2}^{\prime}\circ s_{1},\pi_{1}^{\prime}\circ s_{2} . Proof. Since \Pi_{i}^{-1}(A_{i}) is closed, Definition 5.2 implies that \Pi_{1}^{-1}(A_{1})=\Pi_{2}^{-1}(A_{2}) . For brevity, we denote \pi_{2,1}\triangleq\pi_{2}^{\prime}\circ s_{1} . Note that \pi_{2,1}:{\mathbb{V}}_{1}^{A}\to{\mathbb{V}}_{2}^{A} , we can easily extend this to subsets V\subset{\mathbb{V}}_{1}^{A} by \pi_{2,1}(V)\triangleq\pi_{2}^{\prime}\left(\bigcup_{v\in V}s_{1}(v)\right). (F.1) Now, we refer to \pi_{2,1} as Equation ( F.1 ). We first show that \pi_{2,1} satisfies Equation ( D.1 ). Fix {\bm{v}}_{2}\in{\mathbb{V}}_{2}^{A} . By Definition 4.1 , {\bm{v}}_{2}\triangleq f_{{\bm{v}}_{2}}(Pa({\bm{v}}_{2}),U). (F.2) By assumption, \pi_{2,1}\circ\pi_{1}\in\Pi_{2} . Thus, \pi_{2,1}\circ\pi_{1} is an alignment map that enables an abstraction from K\to A_{2} , we can rewrite Equation ( F.2 ) as \displaystyle{\bm{v}}_{2} \displaystyle=(\pi_{2,1}\circ\pi_{1})\left(\bigcup_{{\bm{v}}_{k}\in(\pi_{2,1}\circ\pi_{1})^{-1}({\bm{v}}_{2})}f_{{\bm{v}}_{k}}(Pa({\bm{v}}_{k}),U)\right), (F.3) \displaystyle=(\pi_{2,1}\circ\pi_{1})\left(\bigcup_{{\bm{v}}_{k}\in\pi_{2,1}^{-1}({\bm{v}}_{2})}\bigcup_{{\bm{v}}\in\pi_{1}^{-1}({\bm{v}}_{k})}f_{{\bm{v}}}(Pa({\bm{v}}),U)\right), (F.4) \displaystyle=\pi_{2,1}\left(\pi_{1}\left(\bigcup_{{\bm{v}}_{k}\in\pi_{2,1}^{-1}({\bm{v}}_{2})}\bigcup_{{\bm{v}}\in\pi_{1}^{-1}({\bm{v}}_{k})}f_{{\bm{v}}}(Pa({\bm{v}}),U)\right)\right), (F.5) \displaystyle=\pi_{2,1}\left(\bigcup_{{\bm{v}}_{k}\in\pi_{2,1}^{-1}({\bm{v}}_{2})}\pi_{1}\left(\bigcup_{{\bm{v}}\in\pi_{1}^{-1}({\bm{v}}_{k})}f_{{\bm{v}}}(Pa({\bm{v}}),U)\right)\right), (F.6) \displaystyle=\pi_{2,1}\left(\bigcup_{{\bm{v}}_{k}\in\pi_{2,1}^{-1}({\bm{v}}_{2})}f_{{\bm{v}}_{k}}(Pa({\bm{v}}_{k}),U)\right). (F.7) The last equality follows from the fact that \pi_{1} is an alignment. Therefore, \pi_{2,1} satisfies Equation ( D.1 ). By the same argument, this also holds for \pi_{1,2} . It remains to check Equation ( D.2 ). Since \pi_{2}^{\prime},\pi_{1}^{\prime} are alignments, they admit \omega_{2}^{\prime},\omega_{1}^{\prime} that satisfy Equation ( D.2 ). Let t_{2},t_{1} be sections of \omega_{2}^{\prime},\omega_{1}^{\prime} , respectively. We claim that \omega_{2,1}\triangleq\omega_{2}^{\prime}\circ t_{1} satisfies Equation ( D.2 ). Since \pi_{2,1}\circ\pi_{1} is an alignment, and take {\bm{v}}_{k}\in{\mathbb{V}} , \displaystyle(\pi_{2,1}\circ\pi_{1})({\rm{do}}({\bm{v}}_{k}\leftarrow f)) \displaystyle={\rm{do}}((\pi_{2,1}\circ\pi_{1})({\bm{v}}_{k})\leftarrow(\omega_{2,1}\circ\omega_{1}^{\prime})(f)), \displaystyle\pi_{2,1}(\pi_{1}({\rm{do}}({\bm{v}}_{k}\leftarrow f))) \displaystyle={\rm{do}}(\pi_{2,1}(\pi_{1}({\bm{v}}_{k}))\leftarrow\omega_{2,1}(\omega_{1}^{\prime}(f))), \displaystyle\pi_{2,1}({\rm{do}}(\pi_{1}({\bm{v}}_{k})\leftarrow\omega_{1}^{\prime}(f))) \displaystyle={\rm{do}}(\pi_{2,1}(\pi_{1}({\bm{v}}_{k}))\leftarrow\omega_{2,1}(\omega_{1}^{\prime}(f))). {\rm{do}}(\pi_{1}({\bm{v}}_{k})\leftarrow\omega_{1}^{\prime}(f))\in{\mathcal{I}}({\mathbb{V}}_{1}^{A}) and \pi_{1}({\bm{v}}_{k})\in{\mathbb{V}}_{1}^{A} . Thus, through relabeling, {\bm{v}}_{k}^{1}\triangleq\pi_{1}({\bm{v}}_{k}) and f_{1}\triangleq\omega_{1}^{\prime}(f) , \pi_{2,1}({\rm{do}}({\bm{v}}_{k}^{1}\leftarrow f_{1}))={\rm{do}}(\pi_{2,1}({\bm{v}}_{k}^{1})\leftarrow\omega_{2,1}(f_{1})). And, Equation ( D.2 ) holds. ∎ Appendix G Proofs for Representation Similarity and Interpretive Equivalence Suppose we have two circuits over a shared task S for models h_{1},h_{2} defined over different neural architectures. These circuits we notate as K_{1}\triangleq({\mathbb{V}}_{1},U,{\mathbb{F}}_{1},\succ) and K_{2}\triangleq({\mathbb{V}}_{2},U,{\mathbb{F}}_{2},\succ) , respectively. Let us also assume each circuit admits an (L_{i},\delta_{i}) -representation R_{i}\triangleq({\mathbb{H}}_{i},U,{\mathbb{F}}_{i},\succ) through alignment maps \rho_{i} . First, we define a notion of distance between these two representations. We note that since the representations themselves are a function of the input (recall Definition 4.1 ), we are essentially defining a distance in a function space. Definition G.1 (Representation Similarity) . Denote by H_{i}\triangleq\concat{({\mathbb{H}}_{i}^{{\mathbb{F}}_{i}})} to be the direct sum (concatenation) of all hidden variables in {\mathbb{H}}_{i} . Then, representation similarity 13 13 13 Our formulation is inspired by 11 ’s ( 11 ) notion of representation alignment between language encoders. Although Definition G.1 is not a true similarity, this naming stays consistent with the literature ( 30 ) . between R_{1} and R_{2} is d_{\mathrm{repr}}(R_{1},R_{2})\triangleq\max\left(\inf_{\left\|A\right\|_{\mathrm{op}}\leq 1}\left\|AH_{1}-H_{2}\right\|,\inf_{\left\|B\right\|_{\mathrm{op}}\leq 1}\left\|H_{1}-BH_{2}\right\|\right), (G.1) where A,B are linear operators and \left\|\cdot\right\| is a direct-sum norm. We first show that representation similarity (Definition G.1 ) is a pseudometric. Lemma G.1 . For any representations R_{1},R_{2},R_{3} , d_{\mathrm{repr}} satisfies 1. d_{\mathrm{repr}}(R_{1},R_{1})=0 2. d_{\mathrm{repr}}(R_{1},R_{2})=d_{\mathrm{repr}}(R_{2},R_{2}) 3. d_{\mathrm{repr}}(R_{1},R_{3})\leq d_{\mathrm{repr}}(R_{1},R_{2})+d_{\mathrm{repr}}(R_{2},R_{3}) Proof. The first two properties are trivial. So, it remains to show triangle inequality. Let H_{i} be the concatenation of hidden variables in R_{i} . Let \varphi_{1,2},\varphi_{2,3} be arbitrary operators where \left\|\varphi_{1,2}\right\|_{\mathrm{op}},\left\|\varphi_{2,3}\right\|_{\mathrm{op}}\leq 1 that map H_{2}\to H_{1} and H_{3}\to H_{2} , respectively. Then, \displaystyle\left\|H_{1}-\varphi_{1,2}\circ\varphi_{2,3}H_{3}\right\| \displaystyle=\left\|H_{1}-\varphi_{1,2}H_{2}+\varphi_{1,2}H_{2}-\varphi_{1,2}\circ\varphi_{2,3}H_{3}\right\|, \displaystyle\leq\left\|H_{1}-\varphi_{1,2}H_{2}\right\|+\left\|\varphi_{1,2}H_{2}-\varphi_{1,2}\circ\varphi_{2,3}H_{3}\right\|, \displaystyle\leq\left\|H_{1}-\varphi_{1,2}H_{2}\right\|+\left\|\varphi_{1,2}\right\|_{\mathrm{op}}\left\|H_{2}-\varphi_{2,3}H_{3}\right\| \displaystyle\leq\left\|H_{1}-\varphi_{1,2}H_{2}\right\|+\left\|H_{2}-\varphi_{2,3}H_{3}\right\|. Taking infimum over \varphi_{1,2},\varphi_{2,3} gives the desired result. ∎ Definition G.2 . Let (Z,d) be a (pseudo)metric space. Let S\subset Z . The diameter of S , denoted \diam(S) is defined as \diam(S)\coloneqq\sup_{a,b\in S}d(a,b). (G.2) Lemma G.2 . Let (Z,d) be a (pseudo)metric space and d_{\mathrm{H}} be its Hausdorff distance. Then, for A,B\subset Z and a\in A,b\in B , d_{\mathrm{H}}{(A,B)}\leq\diam(A)+\diam(B)+d(a,b). Proof. Fix any a^{\prime}\in A . Then, \displaystyle\inf_{b^{\prime}\in B}d(a^{\prime},b^{\prime}) \displaystyle\leq d(a^{\prime},b), \displaystyle\leq d(a^{\prime},a)+d(a,b), \displaystyle\leq\diam(A)+d(a,b). By symmetry, for any b^{\prime}\in A , it follows that \inf_{a^{\prime}\in A}d(a^{\prime},b^{\prime})\leq\diam(B)+d(a,b) . Therefore, by the definition of the Hausdorff distance, we yield the desired bound. ∎ Definition G.3 . Let K_{1}\triangleq({\mathbb{V}},U,F_{1},\cdot),K_{2}\triangleq({\mathbb{V}},U,F_{2},\cdot) be two arbitrary circuits defined over variables {\mathbb{V}} . Suppose some pseudometric d:{\mathbb{V}}\times{\mathbb{V}}\to\mathbb{R}^{\geq 0} . Let R\triangleq({\mathbb{H}},U,\cdot,\prec) be representation spaces and \rho:\mathrm{SubsetsOf}({\mathbb{V}})\to{\mathbb{H}} an alignment map. Denote R_{i} the representations of K_{i} constructed through \rho . We say that \rho is Lipschitz continuous with Lipschitz constant \Lip(\rho) if d_{\mathrm{repr}}\left(R_{1},R_{2}\right)\leq\Lip(\rho)d(F_{1},F_{2}). (G.3) Lemma G.3 . For i\in\{1,2\} let R_{i} be (\cdot,\delta_{i}) -representations for models h_{i} . Suppose that \left\|h_{1}-h_{2}\right\|<\Delta . Then, d_{\mathrm{repr}}(R_{1},R_{2})\leq\delta_{1}+\delta_{2}+\Delta . Proof. Let H_{i} be the concatenation of all hidden variables in R_{i} . Since R_{i} is a (L_{i},\delta_{i}) -representation, there must exist maps A_{i},B_{i} such that \left\|A_{i}H_{i}-h_{i}\right\|<\delta_{i}\qquad\text{and}\qquad\left\|H_{i}-B_{i}h_{i}\right\|<\delta_{i}. To see this, we know that for each hidden layer k , there exists A_{i,k} such that \left\|A_{i,k}H_{i,k}-h_{i}\right\|<\delta_{i} . Then, L_{i}\delta_{i}\geq\sum_{k=1}^{L_{i}}\left\|A_{i,k}H_{i,k}-h_{i}\right\|\geq\left\|\sum_{k=1}^{L_{i}}(A_{i,k}H_{i,k}-h_{i})\right\|=\left\|\sum_{k=1}^{L_{i}}A_{i,k}H_{i,k}-L_{i}h_{i}\right\|. (G.4) Define a single (block diagonal) linear map A_{i} that acts on H_{i}=(H_{i,1},\ldots,H_{i,L_{i}}) where \left\|A_{i}H_{i}\right\|\triangleq\frac{1}{L_{i}}\sum_{k=1}^{L_{i}}\left\|A_{i,k}H_{i,k}\right\|. With the \ell_{\infty} direct-sum norm on \bigoplus_{k}\mathbb{R}^{n_{k}} and \left\|A_{i}\right\|_{\mathrm{op}}\leq 1 . Substituting A_{i} into Equation ( G.4 ), we have that \left\|A_{i}H_{i}-h_{i}\right\|<\delta_{i} . Repeating same construction for B_{i} yields the “decoder” inequality. Consider \inf_{A_{0}}\left\|A_{0}H_{1}-H_{2}\right\| , \displaystyle\inf_{A_{0}}\left\|A_{0}H_{1}-H_{2}\right\| \displaystyle\leq\left\|B_{2}A_{1}H_{1}-H_{2}\right\|, \displaystyle\leq\left\|B_{2}A_{1}H_{1}-B_{2}h_{2}\right\|+\left\|B_{2}h_{2}-H_{2}\right\|, \displaystyle\leq\left\|B_{2}\right\|_{\mathrm{op}}\left(\left\|A_{1}H_{1}-h_{1}\right\|+\left\|h_{1}-h_{2}\right\|\right)+\delta_{2}, \displaystyle\leq\delta_{1}+\delta_{2}+\Delta. The same argument also holds in the other direction: \displaystyle\inf_{A_{0}}\left\|H_{1}-A_{0}H_{2}\right\| \displaystyle\leq\left\|H_{1}-B_{1}A_{2}H_{2}\right\|, \displaystyle\leq\left\|H_{1}-B_{1}h_{1}\right\|+\left\|B_{1}h_{1}-B_{1}A_{2}H_{2}\right\|, \displaystyle\leq\delta_{1}+\left\|B_{1}\right\|_{\mathrm{op}}\left(\left\|h_{1}-h_{2}\right\|+\left\|h_{2}-A_{2}H_{2}\right\|\right), \displaystyle\leq\delta_{1}+\delta_{2}+\Delta. By Definition G.1 , d_{\mathrm{repr}}(R_{1},R_{2})\leq\delta_{1}+\delta_{2}+\Delta . ∎ G.1 Proof of Main Result 1 For i\in\{1,2\} , suppose that A_{i} is an \eta_{i} -faithful interpretation of K_{i} . Let \Pi,\Pi_{\star} be alignment classes that map K_{1} ’s variables onto A_{1},A_{2} ’s variables, respectively. Assume that circuits admit L_{i} -layered representations through an alignment map \rho_{i} . Let F,\tilde{F}\in\Pi^{-1}(A_{1}) , then {\mathbb{V}}^{F}:\Sigma^{\star}\to\mathbb{R}^{n} for some n\in\mathbb{Z}^{+} (Definition 4.2 ). Assumption G.4 . Let \Phi:\mathbb{R}^{n}\to\mathbb{R}^{m} and assume the implementation distance d is in the form d(F,\tilde{F})\triangleq\lVert{\Phi{\mathbb{V}}^{F}-\Phi{\mathbb{V}}^{\tilde{F}}}\rVert, for some function norm. \Phi is a distillation map that compiles all of the activations of {\mathbb{V}}^{F} into a tractable representation. In a sense, \Phi exposes the measurable part of {\mathbb{V}}^{F} . Denote R_{F} to be the representations of F constructed through the alignment map \rho_{1} . Since R_{F} is a deterministic causal model (Definition 4.3 ), we denote {\mathbb{H}}^{F}:\Sigma^{\star}\to\mathbb{R}^{p} to be the solution of all variables in R_{F} given some input setting. In other words, {\mathbb{H}}^{F} is the concatenation of all hidden representations of R_{F} . This is the same construction used in Definition G.1 . Assumption G.5 . Assume there exists a 1-Lipschitz map \Psi:\mathbb{R}^{p}\to\mathbb{R}^{m} such that \lVert\Psi{\mathbb{H}}^{F}-\Psi{\mathbb{H}}^{\tilde{F}}\rVert\leq d_{\mathrm{repr}}(R_{F},R_{\tilde{F}})\qquad\text{and}\qquad\left\|\Phi{\mathbb{V}}^{F}-\Psi{\mathbb{H}}^{F}\right\|<\omega. (G.5) In other words, the measurable part of our circuit is approximately determined by its representation. Next, we assume that there exists at least one implementation in \Pi_{\star}^{-1}(A_{2}) that admits a reasonable representation under \rho_{1} . Since we only enforce that \rho_{1} is a valid abstraction map for K_{1} , without this assumption, one could construct an adversarial implementation of A_{2} whose causal variables are completely discarded by \rho_{1} . Assumption G.6 . There exists F^{\star}\in\Pi_{\star}^{-1}(A_{2}) such that its representation under \rho_{1} , R_{F^{\star}} , is an (L_{1},\delta_{1}) -representation. We are now ready to prove the main result. Main Result . Let \Pi and \Pi_{\star} be alignment classes that map K_{1} ’s variables into A_{1},A_{2} ’s variables, respectively. Then, the approximate interpretive equivalence of A_{1},A_{2} under alignments \Pi^{-1},\Pi_{\star}^{-1} is d_{\mathrm{interp}}(A_{1},A_{2})\leq\kappa(A_{1},K_{1},\Pi)+\kappa(A_{2},K_{1},\Pi_{\star})+2\omega+d_{\mathrm{repr}}(R_{1},R_{2})+\delta_{1}+\delta_{2}+\Delta. (G.6) Proof. By Lemma G.2 , for any F\in\Pi^{-1}(A_{1}),\tilde{F}\in\Pi_{\star}^{-1}(A_{2}) , d_{\mathrm{interp}}(A_{1},A_{2})\leq\kappa(A_{1},K_{1},\Pi)+\kappa(A_{2},K_{1},\Pi_{\star})+d(F,\tilde{F}). So it remains to upper bound d(F,\tilde{F}) for a single pair (F,\tilde{F}) . Let us choose F to be the circuit K_{1} which has representations R_{1} . By Assumption G.6 , choose \tilde{F}\in\Pi_{\star}^{-1}(A_{2}) that admits an (L_{1},\delta_{1}) -representation (which we denote as R_{\tilde{F}} ) under the alignment \rho_{1} . By Assumption G.4 , d(F,\tilde{F})\triangleq\lVert\Phi{\mathbb{V}}^{F}-\Phi{\mathbb{V}}^{\tilde{F}}\rVert. We add and subtract \Psi{\mathbb{H}}^{F},\Psi{\mathbb{H}}^{\tilde{F}} , \displaystyle d(F,\tilde{F}) \displaystyle\leq\lVert\Phi{\mathbb{V}}^{F}-\Psi{\mathbb{H}}^{F}\rVert+\lVert\Psi{\mathbb{H}}^{F}-\Psi{\mathbb{H}}^{\tilde{F}}\rVert+\lVert\Psi{\mathbb{H}}^{\tilde{F}}-\Phi{\mathbb{V}}^{\tilde{F}}\rVert, \displaystyle\leq 2\omega+\lVert\Psi{\mathbb{H}}^{F}-\Psi{\mathbb{H}}^{\tilde{F}}\rVert. Since \Psi is 1-Lipschitz \lVert\Psi{\mathbb{H}}^{F}-\Psi{\mathbb{H}}^{\tilde{F}}\rVert\leq d_{\mathrm{repr}}(R_{1},R_{\tilde{F}}) , where R_{\tilde{F}} is the causal model associated with {\mathbb{H}}^{\tilde{F}} . By triangle inequality, \displaystyle d_{\mathrm{repr}}(R_{1},R_{\tilde{F}}) \displaystyle\leq d_{\mathrm{repr}}(R_{1},R_{2})+d_{\mathrm{repr}}(R_{2},R_{\tilde{F}}), \displaystyle\leq d_{\mathrm{repr}}(R_{1},R_{2})+\delta_{1}+\delta_{2}+\Delta. The last inequality follows from Assumption G.6 and Lemma G.3 . Putting it all together, d_{\mathrm{interp}}(A_{1},A_{2})\leq\kappa(A_{1},K_{1},\Pi)+\kappa(A_{2},K_{1},\Pi_{\star})+2\omega+d_{\mathrm{repr}}(R_{1},R_{2})+\delta_{1}+\delta_{2}+\Delta. ∎ G.2 Proof of Main Result 2 For i\in\{1,2\} , let us denote A_{i} as \eta_{i} -faithful interpretations of K_{i} . Main Result . Let \Pi and \Pi_{\star} be an alignment that map K_{1} ’s variables onto A_{1},A_{2} ’s variables. Suppose A_{1},A_{2} are {\epsilon} -approximate interpretively equivalent under the alignment classes \Pi_{\star},\Pi . Then, \displaystyle d_{\mathrm{repr}}(R_{1},R_{2}) \displaystyle\leq{\epsilon}(\Lip(\rho_{1})+1)+\delta_{1}+\delta_{2}+\Delta. (G.7) Proof. Since A_{1},A_{2} are {\epsilon} -interpretively equivalent under the alignment cases \Pi,\Pi_{\star} , there exists F_{2}^{\prime}\in\Pi_{\star}(A_{2}) such that d(F_{1},F_{2}^{\prime})\leq{\epsilon} . Let R_{1},R_{2}^{\prime} be the representations of F_{1},F_{2}^{\prime} under the alignment \rho_{1} , respectively. Then, \rho_{1} is Lipschitz continuous implies that d_{\mathrm{repr}}(R_{1},R_{2}^{\prime})\leq\Lip(\rho_{1})d(F_{1},F_{2}^{\prime})\leq\Lip(\rho_{1}){\epsilon}. (G.8) Since R_{1} is an (L_{1},\delta_{1}) -representation of h_{1} , there exists A_{1},B_{1} with \left\|A_{1}\right\|_{\mathrm{op}},\left\|B_{1}\right\|_{\mathrm{op}}\leq 1 such that \left\|A_{1}H_{1}-h_{1}\right\|\leq\delta_{1}\qquad\text{and}\qquad\left\|H_{1}-B_{1}h_{1}\right\|\leq\delta_{1}, where H_{1} is the concatenation of all variables in R_{1} . d_{\mathrm{repr}}(R_{1},R_{2}^{\prime})\leq\Lip(\rho_{1}){\epsilon} implies there exists A^{\prime},B^{\prime} such that \left\|A^{\prime}H_{1}-H_{2}^{\prime}\right\|\leq\Lip(\rho_{1}){\epsilon}\qquad\text{and}\qquad\left\|H_{1}-B^{\prime}H_{2}^{\prime}\right\|\leq\Lip(\rho_{1}){\epsilon}, where H_{2}^{\prime} is the concatenation of all variables in R_{2}^{\prime} . We now bound the “encoder” and “decoder” error of R_{2}^{\prime} separately: \displaystyle\left\|A_{1}A^{\prime}H_{2}^{\prime}-h_{2}\right\| \displaystyle\leq\left\|A_{1}(A^{\prime}H_{2}^{\prime}-H_{1})\right\|+\left\|A_{1}H_{1}-h_{1}\right\|+\left\|h_{1}-h_{2}\right\|, \displaystyle\leq\left\|A_{1}\right\|_{\mathrm{op}}\left\|A^{\prime}H_{2}^{\prime}-H_{1}\right\|+\delta_{1}+\Delta, \displaystyle\leq\Lip(\rho_{1}){\epsilon}+\delta_{1}+\Delta. And, \displaystyle\left\|H_{2}^{\prime}-B^{\prime}B_{1}h_{2}\right\| \displaystyle\leq\left\|H_{2}^{\prime}-B^{\prime}H_{1}\right\|+\left\|B^{\prime}(H_{1}-B_{1}h_{1})\right\|+\left\|B^{\prime}B_{1}(h_{1}-h_{2})\right\|, \displaystyle\leq\Lip(\rho_{1}){\epsilon}+\delta_{1}+\Delta. Therefore, R_{2}^{\prime} is a (L_{1},\Lip(\rho_{1}){\epsilon}+\delta_{1}+\Delta) -representation of h_{2} . Directly applying Lemma G.3 , d_{\mathrm{repr}}(R_{2}^{\prime},R_{2})\leq\Lip(\rho_{1}){\epsilon}+\delta_{1}+\delta_{2}+\Delta . By Lemma G.1 , d_{\mathrm{repr}}(R_{1},R_{2})\leq d_{\mathrm{repr}}(R_{1},R_{2}^{\prime})+d_{\mathrm{repr}}(R_{2}^{\prime},R_{2})\leq{\epsilon}(\Lip(\rho_{1})+1)+\delta_{1}+\delta_{2}+\Delta. ∎ Where is faithfulness in all of this? Unintuitively, neither Main Result 1 nor 2 explicitly contain the faithfulness of A_{1},A_{2} ( \eta_{1},\eta_{2} , respectively). Our insight is that faithfulness is implicitly encoded into the alignment classes \Pi,\Pi_{\star} . This is because any valid alignment must be consistent with the faithfulness of A_{1},A_{2} (see Definition 5.1 ). In this way, \eta_{1},\eta_{2} determine which alignment classes are non-empty. This is why alignment classes form the crux of our constructions, because they describe both structural and behavioral constraints over abstractions. Appendix H Computing Interpretive Equivalence For i\in\{1,2\} , let A_{i} be an interpretation and K_{i} be a circuit. Let \Pi_{i} be an alignment class that maps K_{i} ’s variables into A_{i} ’s variables. We define probability spaces (\Pi_{i}^{-1}(A_{i}),\cdot,\mathbb{P}_{A_{i}}) . Suppose \rho_{1},\rho_{2} are alignment maps. Definition H.1 (Congruity) . Denote F_{1},F_{1}^{\star}\overset{\mathrm{i.i.d.}}{\sim}\mathbb{P}_{1} , F_{2},F_{2}^{\star}\overset{\mathrm{i.i.d.}}{\sim}\mathbb{P}_{2} . Let R_{1},R_{1}^{\star},R_{2},R_{2}^{\star} be their representations, respectively, under the alignment maps \rho_{1},\rho_{2} . p_{1}\triangleq\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{1}^{\star})0}\left(\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{1}^{\star})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}]+\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{2})d_{\mathrm{repr}}(R_{1},R_{2}) . For arbitrary {\epsilon}>0 , this event occurs with probability 1 when d_{\mathrm{repr}}(R_{1},R_{2})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon} and d_{\mathrm{repr}}(R_{1},R_{3})\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}]-\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{2})0 was arbitrary, we yield the desired result. ∎ Corollary H.1 . Let F_{1},F_{1}^{\star}\overset{\mathrm{i.i.d.}}{\sim}\mathbb{P}_{1},F_{2},F_{2}^{\star}\overset{\mathrm{i.i.d.}}{\sim}\mathbb{P}_{2} . Let \tilde{C} be the congruity between \Pi_{1}^{-1}(A_{1}),\Pi_{2}^{-1}(A_{2}) . Define \displaystyle f(F_{1},F_{1}^{\star},F_{2},F_{2}^{\star}) \displaystyle\triangleq\inf_{{\epsilon}>0}\Big[\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{1}^{\star})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}] \displaystyle\qquad+2\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{2})\kappa(A_{2},K_{2},\Pi_{2})+{\epsilon}])\Big]. Then, 2-f(F_{1},F_{1}^{\star},F_{2},F_{2}^{\star})\leq\tilde{C}\leq f(F_{1},F_{1}^{\star},F_{2}), (H.5) when f(F_{1},F_{1}^{\star},F_{2},F_{2}^{\star})\geq 1 . Proof. By Definition H.1 , \tilde{C}(p_{1},p_{2})\triangleq 1-|p_{1}+p_{2}-1| . We break \tilde{C} into two cases: \tilde{C}(p_{1},p_{2})=\begin{cases}2-(p_{1}+p_{2})&\text{if}~1\leq p_{1}+p_{2}\leq 2,\\ p_{1}+p_{2}&\text{if}~0\leq p_{1}+p_{2}\leq 1.\end{cases} Since \tilde{C}(p_{1}+p_{2})=\tilde{C}(2-p_{1}-p_{2}) , any lower (resp. upper) bound proved on one side transfers to a lower (resp. upper) bound on the reflected side. Thus, we separately bound each case. Consider first 1\leq p_{1}+p_{2}\leq 2 . Then, \displaystyle\tilde{C}(p_{1},p_{2}) \displaystyle=2-(p_{1}+p_{2}), \displaystyle\leq\inf_{{\epsilon}>0}\Big[\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{1}^{\star})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}] \displaystyle\qquad+2\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{2})\kappa(A_{2},K_{2},\Pi_{2})+{\epsilon}])\Big], where the last inequality follows from Equation ( H.2 ). Now the case 0\leq p_{1}+p_{2}\leq 1 , \displaystyle\tilde{C}(p_{1},p_{2}) \displaystyle=p_{1}+p_{2}, \displaystyle\geq 2-\inf_{{\epsilon}>0}\Big[\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{1}^{\star})>\kappa(A_{1},K_{1},\Pi_{1})+{\epsilon}] \displaystyle\qquad+2\mathbb{P}[d_{\mathrm{repr}}(R_{1},R_{2})\kappa(A_{2},K_{2},\Pi_{2})+{\epsilon}])\Big]. ∎ H.2 Intervention-Implementation Duality Our results in the previous sections show that estimating both the representation similarity and interpretive compression is crucial to understanding interpretive equivalence. In general, enumerating the set of implementations for a given interpretation is hard. Herein, we make a connection between implementations and interventions. Specifically, we show that it is possible to view interventions as instantiations of new implementations. With this perspective, we bound the sample complexity (in interventions) of interpretive compression estimation. First, given a set of interventions, we define a notion of closure for alignment classes. Definition H.2 . Let K\triangleq({\mathbb{V}},U,{\mathbb{F}},\succ) be some circuit and A be an \eta -faithful interpretation of K . Also, let \Pi be an alignment class that maps K ’s variables onto A ’s. Consider a set of interventions over all variables in {\mathbb{V}} which we denote as {\mathcal{I}}({\mathbb{V}}) (see Definition D.2 ). We say that \Pi is closed under {\mathcal{I}}({\mathbb{V}}) if for each implementation K^{\star}\in\Pi^{-1}(A) , there exists \tilde{f}\in{\mathcal{I}}({\mathbb{V}}) such that K^{\star}={\rm{do}}({\mathbb{V}}\leftarrow\tilde{f}) . Remark . Any alignment class \Pi is closed under a natural set of interventions . Simply, given circuits K,K^{\star}\in\Pi^{-1}(A) , this is the set of interventions that replace K ’s transition functions, {\mathbb{F}} , with all of K^{\star} ’s transition functions, {\mathbb{F}}^{\star} . Remark . Dually, instead of explicitly enumerating an alignment class, we could instead specify a set of interventions under which K ’s interpretation is preserved. This implicitly induces an alignment class. With this method, one only needs to argue that the interventions are interpretation-preserving. This is the method we use in GetImpl to construct implementation sets. Definition H.3 . Let C be some set and d:C\times C\to\mathbb{R}^{\geq 0} be some distance metric over C . Let \{x_{1},\ldots,x_{m}\}\subset C . Then, the empirical diameter of C is \max_{1\leq i1-\delta where \hat{\kappa} is the empirical diameter of \Pi^{-1}(A) computed from m sampled interventions. Proof. From Definition H.3 , \mathbb{P}[\kappa(A,K,\Pi)\geq\hat{\kappa}_{m}(A,K,\Pi)]=1 . Consider the event \kappa(A,K,\Pi)-\hat{\kappa}(A,K,\Pi)>{\epsilon} . This occurs only if we draw m interventions \{\iota_{1},\ldots,\iota_{m}\} where there exists at least one ball B_{i} where there does not exist some \iota_{i} such that {\rm{do}}({\mathbb{V}}\leftarrow\iota_{i})\in B_{i} . This event occurs with probability at most (1-p_{\min})^{m} . By the Poisson approximation, we have that (1-p_{\min})^{m}\leq e^{-mp_{\min}} . Setting this less than \delta and rearranging, we yield the result. ∎