Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation

Joakim Edin, Casper L. Christensen, Róbert Csordás, Tuukka Ruotsalo, Zhengxuan Wu, Maria Maistro, Jing Huang, Lars Maaløe · 2025

Read Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation by Joakim Edin, Casper L. Christensen, Róbert Csordás, Tuukka Ruotsalo, Zhengxuan Wu, Maria Maistro, Jing Huang, Lars Maaløe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17630.txt.

Introduction

Joakim Edin and co-authors fix a flaw in gradient-based methods that locate the parts of a language model behind a given behaviour. Most such methods test each part alone, yet model parts interact. One bad case is attention self-repair, where softmax shifts hide the effect of strong attention scores as other positions make up the loss.

Their method, Gradient Interaction Modifications, accounts for feature interactions during backpropagation itself. It reached top results on the circuit localization track of the Mechanistic Interpretability Benchmark. It also beat past gradient methods on feature attribution across varied tasks. The authors explain why older methods underrate some parts, and they share the method as a Python package.

Catalog record

CREATORJoakim Edin, Casper L. Christensen, Róbert Csordás, Tuukka Ruotsalo, Zhengxuan Wu, Maria Maistro, Jing Huang, Lars Maaløe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17630

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation Joakim Edin, Casper L. Christensen, Róbert Csordás, Tuukka Ruotsalo, Zhengxuan Wu, Maria Maistro, Jing Huang, Lars Maaløe Abstract Circuit localization methods aim to identify the subset of model components responsible for specific behaviors in large language models, enabling detailed mechanistic analysis. Most existing methods assume components act independently and estimate importance by perturbing each component in isolation. However, components in neural networks interact, and ignoring these interactions leads to systematic misestimation of component importance. We find that one particularly problematic interaction is attention self-repair, in which softmax redistribution causes gradients for influential attention scores to vanish as other positions with similar values compensate. We introduce Gradient Interaction Modifications (GIM), a technique that explicitly accounts for feature interactions during backpropagation. GIM achieves state-of-the-art performance on the circuit localization track of the Mechanistic Interpretability Benchmark and outperforms existing gradient-based methods on feature attribution across diverse tasks. By accounting for interaction effects and explaining why prior methods underestimate component importance, GIM enables more faithful mechanistic analysis of large language models. GIM is available as a Python package at https://github.com/corticph/gim . Figure 1 : Attention self-repair. When multiple values associated with large attention weights contain similar information, ablating one attention score has little effect on the output because the softmax activation function compensates by increasing the weight of other positions. This results in perturbation-based and gradient-based explanation methods underestimating the importance of components that contribute to the attention scores through the keys and queries. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities, yet our understanding of their inner mechanisms remains limited. This opacity hinders our ability to explain model outputs, catch hallucinations, and ensure reliable behavior. The field of mechanistic interpretability aims to address this by deciphering how models process information internally. One key technique is circuit localization , which isolates the subset of model components (e.g., attention heads, neurons, connections) responsible for a specific behavior, enabling researchers to study them in detail [ 35 ] . Most circuit localization methods assume components are independent, measuring each component’s importance by perturbing it individually and observing the effect on the output [ 5 , 20 ] . However, components in neural networks interact; perturbing multiple components simultaneously can yield different effects than perturbing them individually. When interactions are ignored, methods may systematically misestimate component importance. We propose Gradient Interaction Modifications (GIM) , a gradient-based attribution method that estimates the impact of perturbing multiple interacting components simultaneously. We develop GIM by analyzing how different operations in transformers introduce feature interactions, then design targeted backpropagation modifications to account for them. Through this process, we discover that the attention mechanism introduces an OR-gate-like interaction: when multiple positions have large attention weights and similar value vectors, perturbing any single attention score has minimal effect because the softmax redistributes weight to redundant positions ( Figure 1 ). We denote this phenomenon attention self-repair and prove that it causes gradients to vanish even for influential attention scores. Our experiments show that this occurs across all inputs and LLMs. GIM addresses attention self-repair through temperature-adjusted softmax gradients (TSG) , which approximate the effect of perturbing multiple attention scores simultaneously. GIM also incorporates layernorm freeze [ 25 ] and gradient normalization [ 1 ] to handle interactions in normalization layers and multiplicative operations. Our main contributions are: 1. We introduce GIM , a gradient-based attribution method that accounts for component interactions. GIM achieves state-of-the-art performance on the circuit localization track of the Mechanistic Interpretability Benchmark [ 22 ] . 2. We identify attention self-repair , an OR-gate-like interaction where softmax redistribution causes attribution methods to underestimate component importance, and prove it leads to vanishing gradients. 3. We show that GIM can also be used for feature attribution, outperforming existing methods across several LLMs and tasks. 2 Related Work 2.1 Circuit Localization A neural network can be viewed as a computational graph, where nodes are components (e.g., attention heads, MLPs) and edges represent information flowing between them. Circuit localization methods estimate the importance of either nodes or edges for a specific behavior. By isolating the most important nodes or edges, we obtain a subgraph called a circuit , which can then be studied in detail. We focus on edge-level methods, which provide finer granularity than node-level by revealing how information flows between components. Edge Activation Patching (EActP) [ 5 ] estimates edge importance by measuring the change in model output when replacing an edge’s activation with a counterfactual activation. However, this requires one forward pass per edge, which is computationally expensive for large models that comprise billions of edges. Edge Attribution Patching (EAP) [ 29 ] is a faster approximation that leverages gradients to estimate all edge importances in a single backward pass. EAP first runs a forward pass on a counterfactual example to cache activations \tilde{a}_{i} , then runs a forward and backward pass on the original example. The importance of edge (i,j) from upstream node i to downstream node j is: I_{ij}=(a_{i}-\tilde{a}_{i})\cdot\frac{\partial z}{\partial h_{j}} (1) where a_{i} is the activation of node i on the original example, h_{j} is the input to node j , and z is the model output. EAP-IG [ 11 ] extends EAP with integrated gradients [ 26 ] . Information Flow Routes (IFR) [ 9 ] tracks information flow by analyzing attention patterns across the layers. Uniform Gradient Sampling (UGS) [ 14 ] uses gradient descent to learn a pruning mask over edges, optimizing for sparsity while maintaining model behavior. Hybrid-ens is an ensemble method that combines EAP-IG with a edge-masking method similar to UGS [ 21 ] . All these approaches estimate the effect of perturbing each edge individually, while we estimate the effect of perturbing multiple edges simultaneously. 2.2 Feature Interactions Feature interactions occur when the joint effect of perturbing multiple components differs from the sum of their individual effects [ 32 ] . A well-studied interaction in language models is self-repair : when a component is ablated, other components compensate, making the ablated component appear unimportant [ 35 , 19 ] . This is a form of logical OR-gate behaviour. Prior work has identified backup attention heads [ 18 ] , MLP erasure neurons [ 19 ] , and layer normalization rescaling [ 25 ] as sources of self-repair. Layer normalization rescaling occurs because ablating a component reduces the magnitude of activations, causing layer normalization to amplify the remaining components’ contributions. We identify a new source: softmax redistribution within the attention mechanism. 2.3 Gradient Modifications for Feature Interactions First-order gradients measure the effect of infinitesimal perturbations to each edge independently, failing to capture feature interactions. Previous work has attempted to leverage higher-order gradients to estimate interactions, but these methods require a prohibitive amount of memory and computation [ 12 ] . Instead, several studies modify standard backpropagation to better account for feature interactions. TransformerLRP [ 2 ] treats layer normalization and attention weights as constants during backpropagation, thereby avoiding the interactions introduced by these operations. However, not backpropagating through attention weights discards potentially crucial information. Achtibat et al. [1] propose normalizing the gradients of multiplicative interactions by dividing by the number of inputs. We refer to this method as Gradient Norm and call treating layer

Read the complete text