Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers

Andrew J. Nam, Henry C. Conklin, Yukang Yang, Thomas L. Griffiths, Jonathan D. Cohen, Sarah-Jane Leslie · 2025

Read Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers by Andrew J. Nam, Henry C. Conklin, Yukang Yang, Thomas L. Griffiths, Jonathan D. Cohen, Sarah-Jane Leslie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13737.txt.

Introduction

Andrew Nam and co-authors sort the attention heads inside transformer models by what they truly cause. Their paper from 2025 presents causal head gating, which learns soft gates over heads and labels each as helping, hindering, or idle for a task. Past study styles needed set prompts or target labels; this one needs neither.

Tests ran across the Llama 3 model family on syntax, common sense, and math tasks. Gate scores held up under removal tests and cause chain checks, which marks them as causal, not mere links. A contrast form isolates sub circuits for task parts. Findings show many sparse task fit sub circuits, head roles that shift with partners, and split paths for instruction and example learning.

Catalog record

CREATORAndrew J. Nam, Henry C. Conklin, Yukang Yang, Thomas L. Griffiths, Jonathan D. Cohen, Sarah-Jane Leslie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers Andrew J. Nam, Henry C. Conklin, Yukang Yang, Thomas L. Griffiths, Jonathan D. Cohen, Sarah-Jane Leslie Abstract We present causal head gating (CHG), a scalable method for interpreting the functional roles of attention heads in transformer models. CHG learns soft gates over heads and assigns them a causal taxonomy—facilitating, interfering, or irrelevant—based on their impact on task performance. Unlike prior approaches in mechanistic interpretability, which are hypothesis-driven and require prompt templates or target labels, CHG applies directly to any dataset using standard next-token prediction. We evaluate CHG across multiple large language models (LLMs) in the Llama 3 model family and diverse tasks, including syntax, commonsense, and mathematical reasoning, and show that CHG scores yield causal—not merely correlational—insight — validated via ablation and causal mediation analyses. We also introduce contrastive CHG, a variant that isolates sub-circuits for specific task components. Our findings reveal that LLMs contain multiple sparse, sufficient sub-circuits, that individual head roles depend on interactions with others (low modularity), and that instruction following and in-context learning rely on separable mechanisms. 1 Introduction Large language models (LLMs) achiam2023gpt ; liu2024deepseek ; grattafiori2024llama represent the state-of-the art across a wide array of domains, exhibiting remarkable generalization and problem-solving capabilities. Yet, as these models grow in scale and complexity, they become increasingly opaque, making it more difficult to understand, predict, or control their behavior, which raises concerns about safety and misuse bommasani2021opportunities ; wei2023jailbroken ; weidinger2024holistic . This has motivated a growing body of work on interpretability , which seeks to better understand how LLMs learn and represent information, and how their responses can be shaped ( tenney2019bert ; bricken2023towards ) . Interest has focused in particular on transformer-based architectures vaswani2017attention such as GPT achiam2023gpt , LLaMA grattafiori2024llama , Gemma team2025gemma , and DeepSeek liu2024deepseek , in which the central processing blocks consist of multi-head attention followed by multi-layer perceptrons. Here, there has been considerable research on the roles of individual attention heads, which have been found to exhibit some level of human-interpretability elhage2021mathematical ; todd2023function ; yang2025emergent . Two broad categories of approaches dominate research on mechanistic interpretability in LLMs. The first uses a trained mapping from latent representations to human-interpretable concepts, such as syntactic features tenney2019bert ; tenney2019you ; hewitt2019structural or identifiable items (e.g., the Golden Gate Bridge templeton2024scaling ). The second uses causal interventions to identify portions of a single weight matrix or individual attention heads responsible for a specific behavior ( lee2024mechanistic ; voita2019analyzing ) . These approaches often focus on small portions of a model — ‘zooming in’ ( olah2020zoom ) in an effort to interpret the role of a single computational subgraph. However, in deep-learning models, computation is often distributed hinton1986learning and the role of one component is dependent on another elhage2022superposition ; fakhar2022systematic ; giallanza2024integrated , making the behavior of such complex distributed systems difficult to predict from an understanding of their parts alone ( mitchell2009complexity ) . To apply a distributed perspective to mechanistic interpretability, we introduce causal head gating (CHG) which seeks to identify a parametrically weighted set of heads that contribute to a model’s execution of a given task. Given a dataset that defines a task, we fit a set of gating values for each attention head that applies a soft ablation to its output using next-token prediction, so that task-facilitating heads remain unaltered while any task-interfering heads are suppressed. Using a simple regularization procedure that further separates irrelevant heads from those that facilitate or interfere with task performance, CHG assigns meaningful scores to each attention head across an entire model according to its task contribution. We use these scores to define a taxonomy of task relevance according to how individual attention heads contribute to a model’s distributed computation of a given task, describing each head as facilitating, interfering or irrelevant . In this respect, CHG offers an exploratory complement to standard hypothesis-driven approaches to mechanistic interpretablity, assigning causal roles without relying on predefined hypotheses about what each head might be doing. Beyond its conceptual contribution, CHG also offers several practical methodological advantages over existing mechanistic interpretability tools. First, because CHG operates directly on next-token prediction, it avoids the need for externally-provided labels tenney2019bert ; tenney2019you ; hewitt2019structural ; templeton2024scaling , controlled input-output pairs tenney2019bert ; tenney2019you ; hewitt2019structural , or rigid prompt templates wang2022interpretability ; todd2023function ; yang2025emergent , which are often required for decoding and interventional approaches. Second, CHG naturally accommodates complex target outputs, including chain-of-thought reasoning wei2022chain , where the solution spans multiple intermediate steps. Finally, CHG is highly scalable: it introduces only one learnable parameter per attention head and requires no updates to the underlying model weights, so that the CHG parameters can be fitted in minutes using gradient-based optimization, even for LLMs with billions of parameters. To test its efficacy, we apply CHG across a diverse set of tasks—mathematical, commonsense, and syntactic reasoning—and across LLMs ranging from 1 to 8 billion parameters with varying training paradigms. We use CHG to analyze not only where specific computations take place, but also how distributed they are across attention heads, and how these patterns vary across different tasks and models. We also validate the causal scores produced by CHG by comparing them against targeted ablations as well as causal mediation analysis todd2023function ; wang2022interpretability , showing strong agreement between predicted and observed effects. Finally, we extend CHG to a contrastive setting to identify distinct sub-circuits that support instruction following versus in-context learning, suggesting that even semantically similar tasks can be underpinned by separable mechanisms. Our main contributions are fourfold: 1. We introduce causal head gating (CHG), a parametric, scalable method for identifying potentially distributed, task-relevant sub-circuits in transformer models without requiring prompt templates or labeled outputs, and extend it with contrastive CHG to isolate heads supporting specific sub-tasks. 2. We propose a simple causal taxonomy of heads—facilitating, interfering, and irrelevant—that quantifies the effect of each on task performance using CHG-derived scores. 3. We use CHG to show that models contain multiple low-overlap, task-sufficient sub-circuits, suggesting head roles are not fully modular but depend on interactions with other heads. 4. We use CHG to show that instruction following and in-context learning rely on separable circuits at the head level, and that one can be selectively suppressed through CHG-guided head gating without disrupting the other. 2 Related Work Representational decoders Representational decoders are models trained to map hidden activations to externally labeled properties tenney2019bert ; tenney2019you ; hewitt2019structural , estimating the mutual information between representations and those properties belinkov2022probing ; pimentel2020information . However, such probing results are difficult to interpret: simpler decoders may underfit and miss relevant features (false negatives), while complex decoders may overfit and learn spurious correlations (false positives) hewitt2019designing ; voita2020information , requiring complexity-accuracy tradeoffs to contextualize results voita2020information . Moreover, although decodability indicates that a property is encoded in the representation, it does not imply that the model uses that information for its task—highlighting a correlational finding rather than a causal one ravichander2020probing . Finally, representational decoders require labeled datasets, constraining their use to curated, predefined properties. For a comprehensive review of the probing framework and its limitations, see belinkov2022probing . Sparse autoencoders can be

Read the complete text