Partially Rewriting a Transformer in Natural Language
Gonçalo Paulo, Nora Belrose · 2025
Read Partially Rewriting a Transformer in Natural Language by Gonçalo Paulo, Nora Belrose (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18838.txt.
Introduction
Gonçalo Paulo and Nora Belrose wrote this paper in two thousand twenty-five. It tries to rewrite part of a large language model in plain words, swapping a feedforward block for a wider net whose sparse neurons each carry a written gloss.
First a transcoder stands in for the block, and an automated pipeline drafts a gloss for each neuron. Then a second model plays simulator, guessing each neuron's firing from its gloss and the text around it. The swap's harm is then gauged at the model's output. The added loss is no worse than zeroing the block out whole. A sparse autoencoder run on the residual stream fares the same. Plain glosses, the team finds, are not yet rich enough to beat that floor.
Catalog record
| CREATOR | Gonçalo Paulo, Nora Belrose |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Partially Rewriting a Transformer in Natural Language Gonçalo Paulo, Nora Belrose Abstract The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model’s final output. With our pipeline, the model’s increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline. 1 Introduction While large language models (LLMs) have reached human level performance in many areas ( Guo et al. 2025 ) , we understand little about their internal representations. Early mechanistic interpretability research attempted to explain the activation patterns of individual neurons ( Olah et al. 2020 ; Gurnee et al. 2023 ; Gurnee et al. 2024 ) , but research has found that most neurons are “polysemantic”, activating in semantically diverse contexts ( Arora et al. 2018 ; Elhage et al. 2022 ) . Sparse autoencoders (SAEs) were proposed to address polysemanticity ( Cunningham et al. 2023 ) . SAEs consist of two parts: an encoder that transforms activation vectors into a sparse, higher-dimensional latent space, and a decoder that projects the latents back into the original space. Both parts are trained jointly to minimize reconstruction error. Recently, a significant effort was made to scale SAE training to larger models, like GPT-4 ( Gao et al. 2024 ) and Claude 3 Sonnet ( Templeton et al. 2024 ) , and they have become an important interpretability tool for LLMs. Paulo et al. 2024 took inspiration on Bills et al. 2023 and built an automated pipeline for generating natural language explanations of SAE features and evaluating how good these explanations are, although rigorously measuring how interpretable an explanation is still a complicated and methodologically fraught task. Recently Dunefsky et al. 2024 proposed sparse transcoders as an alternative method for extracting interpretable features from LLMs. The architecture of the transcoder is identical to that of an SAE, but it is trained to predict the output of a feedforward network given its input. We can then entirely replace the original FFN with its transcoder approximation, thereby partially rewriting the model in terms of more interpretable primitives. Figure 1: Distribution of predicted activations for all latents. On the left we compare the distribution of predicted activations before normalization, and on the right we show what the distribution looks like after quantile normalization. Before normalization, the predictor model systematically over-predicts high activation values by multiple orders of magnitude. Quantile normalization primarily has the effect of enforcing a prior in favor of features not being active. The idea of rewriting a neural net in a more interpretable form is not new. The “microscope AI” framework ( Hubinger 2019 ) aims to analyze a neural network’s learned representations to gain actionable insights for humans, rather than using the network directly. These insights would likely take the form of natural language explanations of the network’s features and circuits. Microscope AI aims to reduce risks associated with model deployment while still benefiting from the model’s knowledge. Imitative generalization is a proposal to extend this idea by jointly optimizing the network and its human-interpretable annotations to maximize their prior likelihood ( Barnes 2021 ) . In this work, we pursue the following idea: if the latents of a transcoder are interpretable enough, we can simulate their activations using natural language explanations. Specifically, we replace the encoder of the transcoder with an LLM prompted to predict the activation of each latent given its explanation and the textual context. We then patch this modified transcoder back into the model, hopefully yielding behavior nearly identical to the unpatched model. In the limit, we could use this to “rewrite” every feedforward layer in the model in terms of interpretable features and operations on those features. With our pipeline, the model’s increase in loss is only slightly smaller than when entirely replacing the sparse MLP output with the zero vector. Our results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline. There are many potential ways to improve the quality and specificity of explanations, and we hope this work inspires the community to invest more resources in this direction. 2 Methods We begin by training a sparse transcoder on the MLP of the sixth layer of Pythia 160M ( Biderman et al. 2023 ) . Our loss function is the mean squared error between the transcoder’s output and the MLP output, with no auxiliary loss terms. Sparsity is continously enforced on the transcoder latents using the TopK activation function proposed by Gao et al. 2024 with k=32 . We train over the first 8B tokens of Pythia’s training corpus, the Pile ( Gao et al. 2020 ) , using the Adam optimizer ( Kingma 2014 ) , a sequence length of 2049, and a batch size of 64 sequences. We have also added a linear “skip connection” to the transcoder, which we find improves its ability to approximate the original MLP at no cost to interpretability scores. That is, the transcoder takes the functional form f(x)=\mathbf{W}_{2}\mathrm{TopK}(\mathbf{W}_{1}x+\mathbf{b}_{1})+\mathbf{W}_{\mathrm{skip}}x+\mathbf{b}_{2} (1) Both \mathbf{W}_{2} and \mathbf{W}_{\mathrm{skip}} are zero-initialized, and \mathbf{b}_{2} is initialized to the empirical mean of the MLP outputs, so that the transcoder is a constant function at the beginning of training. We leave a deeper analysis of the skip connection for future work. We also train a sparse autoencoder on the residual stream, with the same training conditions as the transcoder, but without a skip connection. Figure 2: Partially rewriting an LLM. After training a Transcoder, or any type of SAE, we generate explanations for all the latents using the contexts where that latent is active. An LLM is tasked to summarize or otherwise find patterns in the activations and output a simple, single sentence explanation for that latent. These explanations are used by another instance of an LLM to predict wether the latent should be active in a given token. After some post-processing of those predictions, a reconstruction vector is calculated using the decoder directions of the latents that are considered to be active for that token. We use the automated interpretability pipeline released by Paulo et al. 2024 to generate explanations and scores for transcoder and SAE latents. Then we modify the pipeline to do “single” token simulation, tasking an LLM to determine if a latent is active on the last token of a sequence, and by how much (Figure