On the Generalizability of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals"
Asen Dotsinski [email protected], Udit Thakur [email protected], Marko Ivanov [email protected], Mohammad Hafeez Khan [email protected], Maria Heuss [email protected] · 2025
Read On the Generalizability of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals" by Asen Dotsinski [email protected], Udit Thakur [email protected], Marko Ivanov [email protected], Mohammad Hafeez Khan [email protected], Maria Heuss [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22977.txt.
Introduction
Asen Dotsinski and four colleagues wrote a 2025 paper checking whether a well-known study of language models holds up when repeated. The original 2024 study, Competition of Mechanisms, traced how a language model handles a clash between facts it learned in training and a counterfactual pattern repeated in the prompt.
The reproduction team re-ran the experiments on GPT-2 and on the larger Pythia 6.9B model. They confirmed the main findings. Factual and counterfactual information sit in identifiable places in the network. Attention blocks dominate the competition between them, and individual attention heads specialize in one side or the other. Successful reproductions matter because interpretability claims guide real decisions about how these models are used and controlled.
Catalog record
| CREATOR | Asen Dotsinski [email protected], Udit Thakur [email protected], Marko Ivanov [email protected], Mohammad Hafeez Khan [email protected], Maria Heuss [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Generalizability of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals" Asen Dotsinski [email protected], Udit Thakur [email protected], Marko Ivanov [email protected], Mohammad Hafeez Khan [email protected], Maria Heuss [email protected] Abstract We present a reproduction study of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals" ( Ortu et al. 2024 ) , which investigates competition of mechanisms in language models between factual recall and counterfactual in-context repetition. Our study successfully reproduces their primary findings regarding the localization of factual and counterfactual information, the dominance of attention blocks in mechanism competition, and the specialization of attention heads in handling competing information. We reproduce their results on both GPT-2 ( Radford et al. 2019 ) and Pythia 6.9B ( Biderman et al. 2023 ) . We extend their work in three significant directions. First, we explore the generalizability of these findings to even larger models by replicating the experiments on Llama 3.1 8B ( Grattafiori et al. 2024 ) , discovering greatly reduced attention head specialization. Second, we investigate the impact of prompt structure by introducing variations where we avoid repeating the counterfactual statement verbatim or we change the premise word, observing a marked decrease in the logit for the counterfactual token. Finally, we test the validity of the authors’ claims for prompts of specific domains, discovering that certain categories of prompts skew the results by providing the factual prediction token as part of the subject of the sentence. Overall, we find that the attention head ablation proposed in Ortu et al. 2024 is ineffective for domains that are underrepresented in their dataset, and that the effectiveness varies based on model architecture, prompt structure, domain and task. 1 Introduction Large language models (LLMs) have been growing in size and capabilities over the last few years, showing ever-increasing accuracy on a wide range of downstream tasks ( Brown et al. 2020 ; Liu et al. 2024 ) . Despite their remarkable achievements, these models remain largely opaque. Gaining deeper insight into the inner workings of LLMs could bring us closer to solving many currently open problems, such as generating faithful explanations for their outputs ( Turpin et al. 2023 ) or accurately evaluating their true capabilities ( Shevlane et al. 2023 ; van der Weij et al. 2024 ) . Interpretability research has made incremental progress in understanding individual mechanisms within LLMs. Previous studies have investigated specific computational components, such as copy mechanisms ( Elhage et al. 2021 ) , factual knowledge recall ( Geva et al. 2021 ; Meng et al. 2022 ) , and attention heads specializing in certain tasks ( Voita et al. 2019 ) . However, a critical gap remains in understanding how these diverse mechanisms interact and compete during model inference. The original work by Ortu et al. 2024 addresses this challenge by using the TransformerLens ( Nanda & Bloom 2022 ) for analyzing mechanism competition in language models. Their research focuses on the interplay between factual knowledge recall and counterfactual comprehension, discovering a small number of attention heads responsible for the result of the competition. Our reproduction study aims to achieve two primary objectives. First, we seek to independently verify the original study’s findings regarding mechanism competition and information flow patterns. Second, we aim to extend the research by investigating the robustness of the setup in multiple ways. Our extensions include testing the findings on larger model architectures through experiments with Llama 3.1 8B ( Grattafiori et al. 2024 ) , modifying the prompt structure to avoid biasing the model by repeating the entire prompt in the same way, and analysing the impact of specific types of prompts based on their subject matter. 2 Scope of Reproducibility Ortu et al. 2024 employ two primary analytical methods: logit inspection using unembedding matrices ( nostalgebraist 2020 ) and scaling the output of attention heads to measure their importance (similar to attention head patching and knockout ( Wang et al. 2023 ; Geva et al. 2023 ) ). They validate their findings across multiple model scales, focusing predominantly on GPT-2 small ( Radford et al. 2019 ) and Pythia 6.9B ( Biderman et al. 2023 ) . Our contribution lies in the implementation, verification, and extension of the findings of Ortu et al. 2024 . We proceed by verifying the key claims made by the authors, which we summarize and re-state as follows: • Positional Information Encoding : Factual and counterfactual information are encoded differently, with factual attributes appearing in subject positions while counterfactual information is encoded in attribute positions. This pattern becomes monotonically stronger with each layer, and particularly so after about half of the model layers have been applied. • Attention Block Dominance : Attention blocks contribute more significantly to mechanism competition than MLP blocks, being primarily responsible for writing both factual and counterfactual information to the final position. • Attention Head Specialization : A few highly activated attention heads are effectively responsible for the outcome of the mechanism competition. They attend to the attribute position regardless of which mechanism they support, and they largely contribute to the competition by promoting or suppressing the counterfactual (i.e. copy) mechanism, leaving the factual (i.e. recall) mechanism largely unchanged. • Cross-Model Generalization : These competition patterns persist across model scales, from GPT-2 small to Pythia 6.9B. Beyond verifying these core claims, we extend the work through additional investigations: 1. Model Scaling Study We evaluate the above claims for a larger and more modern model through experiments with Llama 3.1 8B ( Grattafiori et al. 2024 ) . 2. Prompt Structure Sensitivity We investigate how prompt structure influences mechanism competition, particularly examining how changing the structure of the prediction sentence from the counterfactual one impacts the model’s propensity to repeat the counterfactual claim. 3. Premise Word Sensitivity We investigate the effect of different "premise words" (i.e. changing the "Redefine:" at the start of the prompt with other words) on the propensity of the model to predict the factual versus the counterfactual token. We also confirm whether this holds up after ablating the attention heads that are evaluated by Ortu et al. 2024 to play the largest role in this ratio. 4. Domain Robustness We analyze the effect of the prompt domain on the findings of the original study. A robust experiment setup should be invariant to the specific topic of the prompts. 3 Methodology While the code from the original study is available, it is distributed across multiple branches for different experiments, with many scripts requiring significant modifications to function properly. We consolidate and improve the codebase, converting plotting scripts from R to Python and rewriting visualization code to ensure reproducibility. 1 1 1 The code for our reproduction study and extensions is available at https://github.com/asendotsinski/comp-mech-generalizability . 3.1 Model Implementation Our reproduction study focuses on two primary models used in the original paper: GPT-2 small and Pythia 6.9B. To test the generalizability of the results, we also run the same experiments on Llama 3.1 8B. Following Ortu et al. 2024 , we use these models as provided by the TransformerLens ( Nanda & Bloom 2022 ) library for mechanistic interpretation. The library provides support for analyzing the behaviour of specific logit outputs, preserving the original pretrained model weights