Weight-sparse transformers have interpretable circuits
Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing · 2025
Read Weight-sparse transformers have interpretable circuits by Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13653.txt.
Introduction
Leo Gao and 5 co-authors train weight-sparse transformers so their internal circuits stay readable to humans. Mechanistic interpretability seeks circuits that people can follow, and forcing most weights to zero leaves each neuron with few links. Pruning then isolates the small subnetwork that carries a chosen hand-crafted task.
The recovered circuits often use neurons and channels that match natural concepts, joined by a handful of clear connections. Making weights sparser trades raw capability for clarity, while larger models push that frontier outward, though scaling past tens of millions of active weights stays hard. Early results also suggest the method can help explain existing dense models, not just new sparse ones.
Catalog record
| CREATOR | Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weight-sparse transformers have interpretable circuits Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, Dan Mossing Abstract Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections. To recover fine-grained circuits underlying each of several hand-crafted tasks, we prune the models to isolate the part responsible for the task. These circuits often contain neurons and residual channels that correspond to natural concepts, with a small number of straightforwardly interpretable connections between them. We study how these models scale and find that making weights sparser trades off capability for interpretability, and scaling model size improves the capability-interpretability frontier. However, scaling sparse models beyond tens of millions of nonzero parameters while preserving interpretability remains a challenge. In addition to training weight-sparse models de novo , we show preliminary results suggesting our method can also be adapted to explain existing dense models. Our work produces circuits that achieve an unprecedented level of human understandability and validates them with considerable rigor. 1 Introduction While neural networks, such as large language models, have rapidly increased in capability in recent years, we still understand very little about how they work. Mechanistic interpretability seeks to reverse engineer neural networks and fully understand the algorithms they implement internally. Figure 1: An illustration of our overall setup. We first train weight-sparse models. Then, for each of a curated suite of simple behaviors, we prune the model down to the subset of nodes required to perform the task. We ablate nodes by pruning to their mean activation value over the pretraining distribution. A major difficulty for interpreting transformers is that the activations and weights are not directly comprehensible; for example, neurons activate in unpredictable patterns that don’t correspond to human-understandable concepts. One hypothesized cause is superposition ( Elhage et al. 2022b ) , the idea that dense models are an approximation to the computations of a much larger untangled sparse network. Existing approaches have made progress on tackling superposition by first learning a basis in which activations appear sparse, and then attempting to understand the computations of the model within that basis ( Marks et al. 2024 ; Ameisen et al. 2025 ) . However, these approaches obtain human-understandable circuits by abstracting away complex computations that are only partially understood. Thus, the resulting circuits may reflect the chosen abstractions in addition to the model’s true mechanisms. Here, we introduce a new paradigm which leads to substantially simpler and more general circuits that we can fully understand even at the lowest levels of abstraction. To do this, we train transformers where the vast majority of weights are zeros; i.e., the L_{0} norm of the weights is small. This constraint drastically simplifies model computations. As each neuron can only read from or write to a few residual channels, the model is discouraged from distributing concept representations across multiple residual channels or using more neurons than strictly needed to represent a single concept. We show that the model learns disentangled circuits for different tasks by isolating the minimal circuit which can perform each task and showing that it is compact. Within these circuits, we find that neuron activations often correspond to simple concepts, such as “tokens following a single quote” or “depth of list nesting”, and the weights encode connections between concepts that are often intuitive. As a relatively rigorous validation, we further demonstrate that our disentangled circuits are necessary and sufficient for the model’s behavior on these tasks; mean-ablating every neuron except the few that are part of the circuit preserves task loss, whereas deleting the few nodes in the circuit severely harms task loss. Figure 2: Our weight-sparse models learn simpler task-specific circuits than dense models. We examine a sparse model and a dense model with the same pretraining loss. We sweep target loss, and find the size of the minimal circuit in each model that can achieve that loss, averaged across tasks. Sparse model circuits are roughly 16-fold smaller at any given loss. Figure 3: Scaling the total parameter count of weight-sparse models improves the capability-interpretability Pareto frontier. Making models sparser ( i.e. decreasing the L_{0} norm of weights) while holding total parameter count fixed trades off the two, harming capability but improving interpretability. We define capability as pretraining loss; see Section 2.2 for our definition of interpretability. Down and to the left is better. Although weight-sparse training has substantial benefits for interpretability, it has the critical disadvantage that it requires training new models de novo ; these models are extremely inefficient to train and deploy, and are unlikely to ever reach frontier capabilities. 1 1 1 See Appendix B for more discussion on the inefficiencies of sparse training. While we believe that training larger interpretable models would be scientifically valuable, we’d also like to use weight-sparse training to better understand existing dense models. We show preliminary results using bridges at each layer to couple a weight-sparse model’s representations to those of a target dense model, so that our model can then serve as an interpretable replacement for the original dense model. To make it easier to replicate our results, we release the weights and pruned circuits for all sparse models used in the experiments in this paper, as well as code for visualizing circuits, at https://github.com/openai/circuit_sparsity/ . Figure 4: The string closing circuit. We omit no detail, showing all 12 nodes and 9 edges needed to complete the task near perfectly. First, 0.mlp converts the token embeddings into “quote detector” and “quote type classifier” residual channels, which are read by key and value channels respectively in 10.attn . Subsequent tokens attend to the key and copy the value to predict the corresponding closing quote. In the diagram, the vertical bundle of lines under each input token is its residual stream. Activations of important nodes on cherry-picked task examples are shown on the left. Dashed horizontal lines mark layer boundaries. \otimes denotes scalar multiplication; directly merged lines denote scalar addition. Black numbers indicate channel or neuron indices. Red and blue numbers mark positive and negative weights (or biases). This diagram only shows the relevant attention path. Inactive parts of the circuit are greyed out, and irrelevant layers are omitted. 2 Methods We first train weight-sparse models ---Transformer models with most of their parameters set to zero. 2 2 2 Not to be confused with sparsity via mixture-of-experts ( Shazeer et al. 2017 ) , which is weight-dense in our terminology, because its weights are almost all nonzero. All of our models are pretrained on a dataset of Python code. We then examine our models’ behavior on a curated suite of simple, unambiguous tasks where they are forced to choose between one of two completions. To assess the interpretability of our models, we isolate the small sparse circuits that our models use to perform each task using a novel pruning method. Since interpretable models should be easy to untangle, individual behaviors should be implemented by compact standalone circuits. Sparse circuits are defined as a set of nodes connected by edges. Our definition of nodes is maximally granular