Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features

Yiting Liu, Zhi-Hong Deng · 2026

Read Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features by Yiting Liu, Zhi-Hong Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22447.txt.

Introduction

Yiting Liu and Zhi-Hong Deng read sparse autoencoder features by their weights, not just their firing patterns. These tools split a model's inner states into parts a human can grasp. Their 2026 preprint notes that most readings watch when a feature fires, and skip what the feature does next.

A feature exists to rebuild a state that the model then uses. Reading its outgoing weights shows that role. The shift can give truer, more useful labels for features. Better labels mean better maps of how models think. The work refines a core safety and science tool.

Catalog record

CREATORYiting Liu, Zhi-Hong Deng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22447

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features Yiting Liu, Zhi-Hong Deng Abstract Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation methods infer feature semantics from activation patterns, but overlook that features are trained to reconstruct activations that serve computational roles in the forward pass. We introduce a novel weight-based interpretation framework that measures functional effects through direct weight interactions, requiring no activation data. Through three experiments on Gemma-2 and Llama-3.1 models, we demonstrate that (1) 1/4 of features directly predict output tokens, (2) features actively participate in attention mechanisms with depth-dependent structure, and (3) semantic and non-semantic feature populations exhibit distinct distribution profiles in attention circuits. Our analysis provides the missing out-of-context half of SAE feature interpretability. 1 Introduction A language model represents more features than it has directions, forcing it to encode features in superposition, which likely explains why many neurons are polysemantic and difficult to explain ( Elhage et al. 2022 ; Bills et al. 2023 ) . Sparse autoencoders (SAEs) have emerged as a promising approach for decomposing such representations into more interpretable features ( Huben et al. 2023 ; Bricken et al. 2023 ; Gao et al. 2024 ; Templeton et al. 2024 ) . By training higher-dimensional reconstructions of model activations, SAEs aim to disentangle the superposed representations into sparse and semantically meaningful components. Feature: Gemma-2-9B , Layer 35, Index 11433 Activation on Tokens: 121.24 . In contrast , we cannot reject the null hypothesis of no enrichment of seasonal SNPs among cl inal SNPs as defined by cl inal * q *- value ([ Fig 111.87 medication was commonly practised ( 3 5 . 9 %) but this was even more significant among Turkish health workers ( 5 4 . 6 %) \{}[ [@ CR 1 9 107.16 of male cons cripts were current smokers ; however , smoking was more frequent among those who were rejected for MS ( 6 7 . 7 %) than among Activation-based Explanation: scientific terminology and statistical data related to genetic research and health outcomes Weight-based: predict next token “among” Figure 1 : An SAE feature with activated tokens highlighted and the highest activation values boxed. The activation-based generated by GPT-4o-mini is superficial and fails to capture the causal effect. Formally, given an activation vector {\bm{x}}\in{\mathbb{R}}^{d_{\text{model}}} from one layer of the model, an SAE is trained to find a sparse feature activation vector {\bm{z}}\in{\mathbb{R}}^{d_{\text{sae}}} (where the feature dimension d_{\text{sae}}\gg d_{\text{model}} ) that can reconstruct the original activation: \displaystyle{\bm{z}} \displaystyle=\sigma\left({\bm{x}}{\bm{W}}_{\text{enc}}+{\bm{b}}_{\text{enc}}\right) (1) \displaystyle\hat{{\bm{x}}} \displaystyle={\bm{z}}{\bm{W}}_{\text{dec}}+{\bm{b}}_{\text{dec}} (2) where \sigma is a nonlinear activation function, typically ReLU. The model is trained to minimize reconstruction loss, and the columns of the decoder matrix {\bm{W}}_{\text{dec}} are interpreted as “features” in the model’s activation space ( Galichin et al. 2025 ) . Sparsity is enforced either by applying ReLU in combination with an L_{1} penalty on {\bm{z}} , or by substituting \sigma with more sophisticated operators ( Gao et al. 2024 ; Bussmann et al. 2025 , e.g.,) , which encourages most features to remain inactive for any given {\bm{x}} . This decomposition aims to make language models more interpretable, helping explain how they process information and make decisions. Because these features emerge as unlabeled latent variables from the training process, their meanings must be inferred post-hoc. Existing interpretation methods primarily address this by asking “When does this feature activate?” and analyzing input contexts where the feature exhibits high activation. This top-down, contextual approach has been scaled through automated explanation generation, where large language models are prompted with activating text examples ( Bricken et al. 2023 ; Gao et al. 2024 ; Paulo et al. 2025 ) to assign plausible meanings to SAE features. However, this perspective only addresses half of the interpretability challenge. LLM-generated descriptions are often high-level summaries of activation contexts, and they are noted for being overly broad and oversimplified ( Ma et al. 2025 ) . Even if such explanations are descriptively accurate, they cannot clearly specify the impact on the model when a feature is activated or deactivated, which is essential for interventional applicability. Critically, SAEs are optimized for reconstruction rather than for explicitly isolating human-interpretable concepts. A feature’s primary role is to accurately rebuild activations in the computation graph, and it should therefore inherit the computational function of those activations. We argue that mainstream interpretation methods are more correlational than causal. By focusing on activation patterns, they overlook the mechanistic effects inherent to the SAE training process. This creates a gap between semantic descriptions and functional understanding, leaving the computational roles of most features unexplored. To bridge this gap, we introduce a complementary, bottom-up perspective that asks “What does this feature do?” We propose out-of-context interpretation , a purely weight-based methodology that analyzes features through their direct interactions with downstream model components, requiring no activation data. By computing the products of feature vectors with the model’s weight matrices, we can directly assess a feature’s potential to influence output predictions and participate in computational circuits. Our contributions are threefold. First, we introduce a novel paradigm for SAE interpretability by developing the first principled framework for analyzing features from a purely weight-based perspective. Second, we identify and quantify computational roles that remain hidden to activation-based methods: semantic features exhibit a U-shaped distribution, concentrating in the early and late layers, while attention-specialized features display an inverted-U distribution, peaking in the mid-layers. Third, we validate and visualize these findings across 100 SAEs from three models, demonstrating that weight-based analysis uncovers both architectural invariants and architecture-specific computational strategies. To facilitate future research, we will publicly release the code required to reproduce all experiments. 2 Related Work As discussed, the predominant approach to SAE interpretability focuses on activation-based analysis, in which textual examples that activate a feature are presented to LLMs to generate semantic explanations ( Bricken et al. 2023 ; Gao et al. 2024 ; Paulo et al. 2025 ) . This paradigm has spurred a rich ecosystem, including online tools for interactive inspection ( Lin 2023 ) , and has been extended to downstream applications like classification ( Gallifant et al. 2025 ) and specialized training for rare concepts ( Muhamed et al. 2025 ) . However, the quality of these automated explanations remains a known challenge: benchmarking efforts have revealed difficulties in effective steering ( Wu et al. 2025 ) , and providing more context can adversely lead to vaguer explanations ( Juang et al. 2024 ) . Meanwhile, previous studies have only preliminarily considered the computational roles. As a proof of concept, Templeton et al. 2024 explored the idea of features as computational intermediates, though their analysis remained in-context by examining feature attributions and ablations on specific inputs. More related to our methodology, Gur-Arieh et al. 2025 have conducted “output-centric” analyses to enhance feature explanation within the framework of LLM-based generation. However, they focused only on the interaction between the feature decoder and the unembedding matrix ( {\bm{W}}_{\text{dec}}{\bm{W}}_{\text{U}} ), as they reported lower performance scores for the embedding matrix even in models with tied weights. Similarly, Paulo et al. 2025 proposed intervention-based scoring, which touched upon a computational direction but still served to refine correlational explanations. The

Read the complete text