BatchTopK Sparse Autoencoders
Bart Bussmann, Patrick Leask, Neel Nanda · 2024
Read BatchTopK Sparse Autoencoders by Bart Bussmann, Patrick Leask, Neel Nanda (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06410.txt.
Introduction
Bart Bussmann, Patrick Leask, and Neel Nanda published this machine learning paper in 2024. Sparse autoencoders are tools for opening up language models: they split the model's inner activity into a set of sparse, human-readable features. A popular design, the TopK autoencoder, forces exactly the same number of active features for every input. That rigidity wastes capacity on simple inputs and starves complex ones.
The authors relax the rule to the level of a whole batch of inputs. Their BatchTopK method lets each sample use as many or as few features as it needs, while the average across the batch stays fixed. On two models, GPT-2 Small and Gemma 2 2B, it rebuilt inner activity more faithfully than TopK at the same average sparsity, and it matched a leading rival method. A practical bonus: the average feature count is set directly, with no costly tuning search.
Catalog record
| CREATOR | Bart Bussmann, Patrick Leask, Neel Nanda |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06410
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BatchTopK Sparse Autoencoders Bart Bussmann, Patrick Leask, Neel Nanda Abstract Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting language model activations by decomposing them into sparse, interpretable features. A popular approach is the TopK SAE, that uses a fixed number of the most active latents per sample to reconstruct the model activations. We introduce BatchTopK SAEs, a training method that improves upon TopK SAEs by relaxing the top-k constraint to the batch-level, allowing for a variable number of latents to be active per sample. As a result, BatchTopK adaptively allocates more or fewer latents depending on the sample, improving reconstruction without sacrificing average sparsity. We show that BatchTopK SAEs consistently outperform TopK SAEs in reconstructing activations from GPT-2 Small and Gemma 2 2B, and achieve comparable performance to state-of-the-art JumpReLU SAEs. However, an advantage of BatchTopK is that the average number of latents can be directly specified, rather than approximately tuned through a costly hyperparameter sweep. We provide code for training and evaluating BatchTopK SAEs at https://github.com/bartbussmann/BatchTopK . 1 Introduction Sparse autoencoders (SAEs) have been proven effective for finding interpretable directions in the activation space of language models [ 1 , 2 , 9 , 6 ] . SAEs find approximate, sparse, linear decompositions of language model activations by learning a dictionary of interpretable latents from which the activations are reconstructed. The objective used in training SAEs [ 1 ] has both a sparsity and a reconstruction term. These are naturally in tension as, for an optimal dictionary of a given size, improving the reconstruction performance requires decreasing sparsity and vice versa. Recently, new architectures have been proposed to address this issue, and achieve better reconstruction performance at a given sparsity level, such as Gated SAEs [ 6 ] , JumpReLU SAEs [ 7 ] , and TopK SAEs [ 3 ] . In this paper, we introduce BatchTopK SAEs, a novel variant that extends TopK SAEs by relaxing the top- k constraint to a batch-level constraint. This modification allows the SAE to represent each sample with a variable number of latents, rather than assuming that all model activations consist of the same number of units of analysis. By selecting the top activations across the entire batch, BatchTopK SAEs enable more flexible and efficient use of the latent dictionary, leading to improved reconstruction without sacrificing average sparsity. During inference we remove the batch dependency by estimating a single global threshold parameter. Through experiments on the residual streams of GPT-2 Small [ 5 ] and Gemma 2 2B [ 8 ] , we show that BatchTopK SAEs consistently outperform both TopK and JumpReLU SAEs in terms of reconstruction performance across various dictionary sizes and sparsity levels, although JumpReLU SAEs have less downstream CE degradation in large models with a high number of active latents. Moreover, unlike JumpReLU SAEs, BatchTopK SAEs allow direct specification of the sparsity level without the need for tuning additional hyperparameters. 2 Background: Sparse Autoencoder Architectures Figure 1: On GPT-2 Small activations, BatchTopK largely achieves better NMSE and CE than standard TopK across different dictionary sizes, for a fixed number of active latents of 32 (Left). JumpReLU SAEs are omitted from this comparison as their L0 cannot be fixed to a value. For fixed dictionary size (12288) and varying levels of k, BatchTopK outperforms TopK and JumpReLU SAES, both in terms of NMSE and CE (Right). Sparse autoencoders aim to learn efficient representation of data by reconstruction inputs while enforcing sparsity in the latent space. In the context of large language models, SAEs decompose model activations \mathbf{x}\in\mathbb{R}^{n} into sparse linear combinations of learned directions, which are often interpretable and monosemantic. An SAE consists of an encoder and a decoder: \displaystyle\mathbf{f}(\mathbf{x}) \displaystyle:=\sigma(\mathbf{W}_{\text{enc}}\mathbf{x}+\mathbf{b}_{\text{enc}}), (1) \displaystyle\hat{\mathbf{x}}(\mathbf{f(x)}) \displaystyle:=\mathbf{f(x)}\mathbf{W}_{\text{dec}}+\mathbf{b}_{\text{dec}}. (2) where \mathbf{f}(\mathbf{x})\in\mathbb{R}^{m} is the sparse latent representation and \hat{\mathbf{x}}(\mathbf{f})\in\mathbb{R}^{n} is the reconstructed input. \mathbf{W}^{\text{enc}} is the encoder matrix with dimension n\times m and \mathbf{b}^{\text{enc}} is a vector of dimension m ; conversely \mathbf{W}^{\text{dec}} is the decoder matrix with dimension m\times n and \mathbf{b}^{\text{dec}} is of dimension n . The activation function \sigma enforces non-negativity and sparsity in \mathbf{f}(\mathbf{x}) , and a latent i is active on a sample \mathbf{x} if f_{i}({\mathbf{x}})>0 . SAEs are trained on the activations of a language model at a particular site, such as the residual stream, on a large text corpus, using a loss function of the form \mathcal{L}(\mathbf{x}):=\underbrace{\|\mathbf{x}-\hat{\mathbf{x}}(\mathbf{f}(\mathbf{x}))\|_{2}^{2}}_{\mathcal{L}_{\text{reconstruct}}}+\underbrace{\lambda\mathcal{S}(\mathbf{f}(\mathbf{x}))}_{\mathcal{L}_{\text{sparsity}}}+\alpha\mathcal{L}_{\text{aux}} (3) where \mathcal{S} is a function of the latent coefficients that penalizes non-sparse decompositions, and \lambda is a sparsity coefficient, where higher values of \lambda encourage sparsity at the cost of higher reconstruction error. Some architectures also require the use of an auxiliary loss \mathcal{L}_{\text{aux}} , for example to recycle inactive latents in TopK SAEs. ReLU SAEs [ 1 ] use the L1-norm S(\boldsymbol{f}):=||\boldsymbol{f}||_{1} as an approximation to the L0-norm for the sparsity penalty. This provides a gradient for training unlike the L0-norm, but suppresses latent activations harming reconstruction performance [ 6 ] . Furthermore, the L1 penalty can be arbitrarily reduced through reparameterization by scaling the decoder parameters, which is resolved in [ 1 ] by constraining the decoder directions to the unit norm. Resolving this tension between activation sparsity and value is the motivation behind the newer architecture variants. TopK SAEs [ 3 , 4 ] enforce sparsity by retaining only the top k activations per sample. The encoder is defined as: \boldsymbol{f}(\boldsymbol{x}):=\text{TopK}(\mathbf{W}_{\text{enc}}\mathbf{x}+\mathbf{b}_{\text{enc}}) (4) where TopK zeroes out all but the k largest activations in each sample. This approach eliminates the need for an explicit sparsity penalty but imposes a rigid constraint on the number of active latents per sample. An auxiliary loss \mathcal{L}_{aux}=||e-\hat{e}||^{2} is used to avoid dead latents, where \hat{e}=W_{dec}z is the reconstruction using only the top- k_{aux} dead latents (usually 512), this loss is scaled by a small coefficient \alpha (usually 1/32). JumpReLU SAEs [ 7 ] replace the standard ReLU activation function with the JumpReLU activation, defined as \text{JumpReLU}_{\theta}(z):=zH(z-\theta) (5) where H is the Heaviside step function, and \theta is a learned parameter for each SAE latent, below which the activation is set to zero. JumpReLU SAEs are trained using a loss function that combines L2 reconstruction error with an L0 sparsity penalty, using straight-through estimators to train despite the discontinuous activation function. A major drawback of the sparsity penalty used in JumpReLU SAEs compared to (Batch)TopK SAEs is that it is not possible to set an explicit sparsity and targeting a specific sparsity involves costly hyperparameter tuning. While evaluating JumpReLU SAEs, [ 7 ] chose the SAEs from their sweep that were closest to the desired sparsity level, however this resulted in SAEs with significantly different sparsity levels being directly compared. JumpReLU SAEs use no auxiliary loss function. 3 BatchTopK Sparse Autoencoders We introduce BatchTopK SAEs as an improvement over standard TopK SAEs. In BatchTopK, we replace the sample-level TopK operation with a batch-level BatchTopK function. Instead of selecting the top k activations for each individual sample, we select the top n\times k activations across the entire batch of n samples, setting all other activations to zero. This allows for a more flexible allocation of active latents, where some samples may use more than k latents while others use fewer, potentially leading to better reconstructions of the activations that