Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders
Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi · 2025
Read Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders by Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22332.txt.
Introduction
Mengyu Ye, Jun Suzuki, Tatsuro Inaba, and Tatsuki Kuribayashi ask whether the costly machinery of sparse autoencoders is needed to read a language model's mind. Interpretability research has lately meant training proxy modules, especially sparse autoencoders, to surface clean features, then grading the features. Surprisingly few studies check whether the model's own stored features are already just as readable.
Viewing the feed-forward layers as key-value memories, the team grades the vectors stored there against the autoencoder's features on modern interpretability benchmarks. The verdict is close: the two sit in a similar range, with the autoencoder ahead only minimally in some respects and, in others, the plain feed-forward features surprisingly ahead. The comparison reframes a research fashion: before adding a proxy, measure what the original parameters already explain.
Catalog record
| CREATOR | Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi Abstract Recent interpretability work on large language models (LLMs) has been increasingly dominated by a feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated. This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately, only a few studies have made such comparisons systematically so far. In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAEs and FFs diverged. These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors, and FF key-value parameters serve as a strong baseline in modern interpretability research 1 1 1 Project page: https://muyo8692.com/projects/ff-kv-sae . 1 Introduction Transformer-based language models (LMs) have exhibited outstanding performance on a wide variety of tasks [ 10 , 35 , 1 , 44 , 45 ] , whereas their underlying mechanisms remain opaque [ 47 , 37 , 50 , 17 , 38 , 18 , 34 , 28 , 31 ] . This issue has been tackled in the interpretability field, and in earlier days, the field has typically adopted a top-down approach, where, given candidate features or algorithms, e.g., syntactic structure, it has been inspected where in the original model those are encoded. Nowadays, as a variety of capabilities emerge in larger LMs, the question tends to be more on the bottom-up , feature-discovery side: what kind of features are encoded in the model?; and how can we discover and control them? This feature-discovery age has brought two trends to the interpretability community simultaneously: (i) training an external proxy module dedicated to this purpose, namely, sparse auto-encoder (SAEs), to decompose neuron activations into simpler basic features [ 53 , 8 , 24 , 30 , 21 , 16 ] ( proxy-based analysis ), and (ii) developing new comprehensive interpretability benchmarks [ 36 , 27 ] to test the quality of discovered features. This paper explores one overlooked question in the field, to what extent a proxy-based, artificial decomposition of neuron activations empirically benefits the model interpretation. In other words, feed-forward (FF) layers naturally implement the decomposition of neural activation into a set of feature vectors, through the lens of FF as key-value memories [ 17 ] ( FF-KV analysis ), why not first evaluate such organic features in FFs with the newly developed interpretability benchmarks? Proxy-based and FF-KV analysis have complementary advantages, and thus, there is no immediate reason to dismiss the latter. For example, while some proxy-based methods have a theoretical motivation to handle superposition, they also have limitations that FF-KV analysis can automatically bypass: proxy modules can additionally expose biases to the interpretation, e.g., specific features are repeatedly found [ 8 , 49 , 11 ] ; the external proxy hallucinates features [ 22 ] ; and additional computation costs are needed to interpret the model. Furthermore, the FF activations are reported to be naturally sparse even without any regularization [ 29 ] . Thus, if FF-KV and SAE analyses yield comparable results, there are several advantages (more simply put, from Occam’s Razor principle) to adopting the former FF-KV analyses. To gauge the (dis)similarities between FF-KVs’ and SAEs’ interpretability level, we perform both automatic evaluation and manual feature analyses. Automatic evaluation with SaeBench demonstrates surprising similarities between the two approaches. The evaluation scores fell into a similar range in all eight metrics in SAEBench , and the inter-metrics tendencies are also paralleled, e.g., causal intervention scores are poorer than feature disentanglement scores in both methods. One can even observe some advantages of FF-KVs; for example, features in the original FFs tend to avoid feature overlapping, resulting in better absorption scores [ 11 ] (i.e., less redundancy) than those in SAEs. These comparable quality is further supported by human manual evaluation of feature qualities. Conceptual features can be found with almost equal ease from both FF-KV features and SAEs. These tentatively conclude that features from FF-KVs and SAEs serve a quite similar level of interpretability from both quantitative and qualitative perspectives. In our analysis, we further investigate the faithfulness of proxy-discovered features, considering FF-KV features as gold, how large is the overlap between the feature sets of the original FF-KV module and that of the proxy module? We analyzed such an overlap with Transcoder (TC), the closest counterpart to FF-KV, as a proxy model, and revealed that the majority of TC features do not have similar counterparts in the original FF module. This aligns with the existing report that SAEs can interpret even random Transformers [ 22 ] , and perhaps the proxy module hallucinates new features rather than translating the workings of the original FF module, encouraging further research on the faithfulness of the learned features, with FF-KV features as grounding points. To sum up, our study reveals that proxy-based methods such as SAEs empirically offer very limited advantage over the direct analysis of FFs (i.e., key-value memories). That is, the theoretical advantage of SAEs is not observed empirically, at least through the lens of the current evaluation scheme, and encourages the inclusion of FF-KV features as a strong baseline when assessing feature-discovery methods in the interpretability field. 2 Background 2.1 Related Work Dictionary Learning and LLMs Interpretation. Dictionary learning has been proposed to address polysemanticity of the representation [ 3 , 52 , 42 , 40 , 14 , 5 , 20 ] , and this has been applied to interpret the internal activations of LLMs, represented by sparse autoencoders (SAEs) [ 53 , 8 , 24 , 30 , 21 , 16 , 25 ] . Specifically, these introduce a proxy module to decompose and reconstruct a model’s activation, and seek interpretable features in it. Apparently, promising results were observed in earlier days: the learned features are highly interpretable and can be directly used to steer the model’s behavior [ 46 ] : modification on a feature will either eliminate the corresponding behavior, or enhance it. Mixed Reports on SAE Features. Although the SAEs get increasing attention, concurrent works have brought skeptical views on their success. For example, SAE-based feature steering quality is inferior to simple baselines utilizing activations [ 51 ] ; SAEs can learn meaningful features even from a randomly initialized Transformer [ 22 ] ; and they exhibit no clear advantage in downstream tasks and sometimes underperform linear probes that use the model’s raw activations [ 9 , 26 ] . This study, at a high level, provides additional support for such criticisms of the general advantage of SAEs. Interpretability of Feed-Forward (FF) Layers. There have been a fair number of studies to interpret the feed-forward (FF) layer in Transformers directly [ 13