Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models

Aashiq Muhamed, Mona Diab, Virginia Smith · 2024

Read Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models by Aashiq Muhamed, Mona Diab, Virginia Smith (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00743.txt.

Description

Abstract Understanding and mitigating the potential risks associated with foundation models (FMs) hinges on developing effective interpretability methods. Sparse Autoencoders (SAEs) have emerged as a promising tool for disentangling FM representations, but they struggle to capture rare, yet crucial concepts in the data.

Catalog record

CREATORAashiq Muhamed, Mona Diab, Virginia Smith
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models Aashiq Muhamed, Mona Diab, Virginia Smith Abstract Understanding and mitigating the potential risks associated with foundation models (FMs) hinges on developing effective interpretability methods. Sparse Autoencoders (SAEs) have emerged as a promising tool for disentangling FM representations, but they struggle to capture rare, yet crucial concepts in the data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to illuminate these elusive dark matter features by focusing on specific subdomains. We present a practical recipe for training SSAEs, demonstrating the efficacy of dense retrieval for data selection and the benefits of Tilted Empirical Risk Minimization as a training objective to improve concept recall. Our evaluation of SSAEs on standard metrics, such as downstream perplexity and L 0 L_{0} sparsity, show that they effectively capture subdomain tail concepts, exceeding the capabilities of general-purpose SAEs. We showcase the practical utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs achieve a 12.5% increase in worst-group classification accuracy when applied to remove spurious gender information. SSAEs provide a powerful new lens for peering into the inner workings of FMs in subdomains. 1 Introduction Interpretability is crucial for ensuring the safety and reliability of foundation models (FMs) ( Bommasani et al., 2021 ) . A key challenge in interpretability research is to scalably explain the myriad unanticipated behaviors in FMs. Sparse Autoencoders (SAEs) have recently emerged as a promising tool for disentangling the complex, high-dimensional representations within FMs into meaningful, human-interpretable features without supervision ( Cunningham et al., 2023 ; Gao et al., 2024 ; Braun et al., 2024 ; Bricken et al., 2023 ) . However, even massively wide SAEs, trained on vast amounts of data, may only capture a fraction of the concepts embedded within these models ( Templeton et al., 2024 ) . A significant portion of rare or highly specific concepts remain essentially invisible due to their infrequent activation. These elusive features, akin to dark matter in the universe of interpretability, pose a significant challenge for understanding and mitigating potential risks associated with FMs. While larger SAEs did exhibit some features for rarer concepts, Templeton et al. (2024) found compelling evidence suggesting a vast amount of dark matter features were still being missed. For example, they found features for some of San Francisco’s neighborhoods, but their model still lacked features for smaller entities like coffee shops or street intersections. They observed that if a concept is present only once every billion tokens, we may need a billion-feature SAE to capture it reliably. This raises a critical question: can we develop more efficient methods than simply scaling SAE width to capture the tail concepts we are interested in? This paper introduces Specialized Sparse Autoencoders (SSAEs), a novel approach designed to address this challenge. Instead of aiming to capture all concepts, as in current SAE practices, we propose SSAEs as an unsupervised targeted method for efficiently extracting rare features related to specific subdomains. By focusing on a particular subdomain, we can train SSAEs to learn features representing tail concepts without needing to scale to billions of features. Furthermore, instead of relying solely on scaling, we investigate whether Tilted Empirical Risk Minimization (TERM), which approximates minimax risk at large tilt parameters, can further improve the representation of tail concepts within SSAEs. Our key contributions are: 1. Specialized Sparse Autoencoders: An unsupervised method for efficiently extracting rare, subdomain-specific features. We demonstrate empirically that SSAEs capture a greater proportion of tail concepts than standard SAEs trained on general-purpose data, achieving a 12.5% increase in worst-group classification accuracy on the Bias in Bios dataset when used to remove spurious gender information. 2. Subdomain Data Selection Strategies: A practical recipe for training SSAEs, starting with a small seed dataset and leveraging various data selection strategies to identify relevant training data from the FM’s pretraining corpus. We find that Dense retrieval is particularly effective while TracIn reranking can offer further improvements. 3. Tilted Empirical Risk Minimization for SAEs: A novel training objective for SAEs designed to improve concept recall. At large tilt values, TERM encourages more balanced learning of head and tail concepts. We show that TERM-trained SSAEs are more interpretable, exhibit improved concept detection, while maintaining comparable downstream perplexity. We envision SSAEs as versatile tools for concept detection and control across domains where identifying rare features is crucial, such as AI safety (detecting deception), healthcare (identifying outliers), and fairness (recognizing underrepresented groups). See Appendix M for additional examples. Related Work Much interpretability research focuses on analyzing coarse-grained model components like induction heads and MLP modules ( Olsson et al., 2022 ; Elhage et al., 2022b ; Geva et al., 2023 ; Meng et al., 2022 ; Nanda et al., 2023b ) , or fine-grained units like linear probes ( Kim et al., 2018 ; Belinkov, 2022 ; Geiger et al., 2023 ; Zou et al., 2023 ) . Both have limitations. The inherent polysemanticity of coarse-grained components complicates interpretation. Fine-grained analysis, while potentially more precise, is constrained by reliance on curated datasets that isolate behavior, limiting generalizability to unknown mechanisms. Feature disentanglement methods, such as SAEs ( Bricken et al., 2023 ; Cunningham et al., 2023 ) , offer a promising unsupervised alternative, aiming to identify human-interpretable directions in an FM’s latent space. For additional work see Appendix A . 2 Methodology 2.1 Sparse Autoencoders (SAE) The superposition hypothesis in FMs suggests that a limited number of neurons encode a much larger number of concepts, leading to complex and overlapping representations ( Elhage et al., 2022b ) . Superposition, while efficient, makes it challenging to interpret individual neuron representations or directions in representation space. Sparse autoencoders (SAEs) offer a potential solution by learning to reconstruct FM representations at a layer using a sparse set of features in a higher-dimensional space, potentially disentangling superposed features and revealing more interpretable representations ( Elhage et al., 2022a ; Olshausen and Field, 1997 ) . In a well-trained SAE, individual features in the hidden dimension align with underlying sparse, semantically meaningful features ( Donoho, 2006 ) . SAEs decompose a model’s activation x\in\mathbb{R}^{n} into a sparse, linear combination of feature directions: x\approx x_{0}+\sum_{i=1}^{M}f_{i}(x)d_{i}, where d_{i} are M\gg n latent unit-norm feature directions, and the sparse coefficients f_{i}(x)\geq 0 are the corresponding feature activations for x . The right-hand side of this equation has the structure of an autoencoder: an input activation x is encoded into a (sparse) feature activations vector f(x)\in\mathbb{R}^{M} , which is then linearly decoded to reconstruct x . We parameterize a single-layer autoencoder (f,\hat{x}) as follows: f(x):=\text{ReLU}(W_{enc}(x)+b_{enc}) and \hat{x}(f):=W_{dec}f+b_{dec} where W_{enc}\in\mathbb{R}^{M\times n} and W_{dec}\in\mathbb{R}^{n\times M} are the encoding and decoding weight matrices, and b_{enc}\in\mathbb{R}^{M} and b_{dec}\in\mathbb{R}^{n} are the bias vectors. The training objective combines a reconstruction loss and a sparsity penalty: L(x)=\|x-\hat{x}(f(x))\|_{2}^{2}+\lambda\|f(x)\|_{1} (1) where \lambda>0 is a hyperparameter controlling the trade-off between reconstruction fidelity and sparsity. We constrain the columns of W_{dec} to have unit norm during training ( Bricken et al., 2023 ) . In existing work, SAEs for FMs are trained on the same large, general-purpose dataset used to train the underlying FM ( Bricken et al., 2023 ; Cunningham et al., 2023 ; Rajamanoharan et al., 2024 ; Gao

Read the complete text