Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders
Luke Marks, Alasdair Paren, David Krueger, Fazl Barez · 2024
Read Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders by Luke Marks, Alasdair Paren, David Krueger, Fazl Barez (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01220.txt.
Description
Abstract Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting their effectiveness. We propose Mutual Feature Regularization (MFR) , a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features.
Catalog record
| CREATOR | Luke Marks, Alasdair Paren, David Krueger, Fazl Barez |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders Luke Marks, Alasdair Paren, David Krueger, Fazl Barez Abstract Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting their effectiveness. We propose Mutual Feature Regularization (MFR) , a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features. We motivate MFR by showing that features learned by multiple SAEs are more likely to correlate with features of the input. By training on synthetic data with known features of the input, we show that MFR can help SAEs learn those features, as we can directly compare the features learned by the SAE with the input features for the synthetic data. We then scale MFR to SAEs that are trained to denoise electroencephalography (EEG) data and SAEs that are trained to reconstruct GPT-2 Small activations. We show that MFR can improve the reconstruction loss of SAEs by up to 21.21% on GPT-2 Small, and 6.67% on EEG data. Our results suggest that the similarity between features learned by different SAEs can be leveraged to improve SAE training, thereby enhancing performance and the usefulness of SAEs for model interpretability. 1 Introduction Interpretability aims to explain the relationship between neural network internals and neural network outputs. Many interpretability techniques examine raw activations, equating proposed fundamental units of neural networks such as neurons or polytopes to human understandable concepts ( Erhan et al., 2009 ; Nguyen et al., 2016 ; Bau et al., 2017 ; Olah et al., 2018 ; Black et al., 2022 ) . These techniques often benefit from a clean correspondence between those fundamental units and concepts, and may fail if concepts are distributed over many units, or many concepts focused in a single unit, such as in the case of feature superposition ( Elhage et al., 2022 ) . We describe features of the input as the atomic, human-understandable concepts represented by input data. To derive a representation of activations with a stronger one-to-one correspondence of features and neurons, sparse autoencoders (SAEs) have been trained on neural network activations. The decoders of SAEs trained on neural network activations have been shown to form dictionaries of features more easily explained than the neurons themselves, making SAEs potentially useful for understanding the internals of neural networks ( Bricken et al., 2023 ; Cunningham et al., 2024 ; Gao et al., 2024 ) . Despite the recent popularity of SAEs, early results suggest they may learn features that are not features of the input, reducing their usefulness for interpretability ( Till, 2024 ; Huben, 2024 ; Anders et al., 2024 ) . One failure mode considers transformations on the space of inputs: features of the input that are ‘split’ over multiple decoder weights, or multiple features ‘composed’ in one decoder weight. Conceivably, the representation learned by the SAE could be so varied from the input as to contain features entirely incompatible with the input space. These failures are alarming, as studying an SAE would not be guaranteed to reveal information about the neural network that SAE was trained on activations from. In the worst case, if features were commonly split and composed, it is not obvious why studying SAEs would be more useful than studying the raw activations directly, although prior work has given evidence against this. We hypothesize that if a feature is learned by multiple SAEs, that feature is more likely to be a feature of the input, and show that this is true for SAEs trained on synthetic data comprised of known features. Based on this result, we encourage multiple SAEs trained on the same data to learn common features through conditionally reinitializing SAE weights, and an auxiliary penalty calculated using the similarity of the SAE weights. We name this reinitialization technique and auxiliary penalty Mutual Feature Regularization (MFR) . Using SAEs trained with MFR , we learn more features of the input than baseline SAEs when training on synthetic data (Section 3 ). We then train SAEs with MFR on activations from GPT-2 Small ( Radford et al., 2019 ) and on electronencephalography (EEG) data, showing that MFR improves SAEs at scale, and on real-world data (Section 4 ). Our findings indicate that MFR helps avoid features not in the input space and improves performance on key SAE evaluations, potentially increasing their usefulness for interpretability. Figure 1: Our experimental pipeline for training SAEs with MFR . In step one, we extract activations from a neural network, represented by the interconnected nodes on the left. These activations are the inputs for our SAEs. In step two, we train multiple SAEs on the extracted activations. Each SAE learns to reconstruct the input activations through a sparsity constraint on the hidden layer. MFR involves several steps: We first check for inactive features in the SAE hidden state after applying the TopK activation function. If too many inactive features are detected, we reinitialize the weights of the affected SAE. We also include an auxiliary penalty to encourage the SAEs to learn similar features, shown by the final text box. 2 Background 2.1 Sparse autoencoders Olshausen and Field (1996) introduced unsupervised learning of sparse representations, capturing structure in data more efficiently than dense representations. Sparse autoencoders (SAEs) have since found wide application in domains such as representation learning Coates et al. (2011) ; Henaff et al. (2011) , denoising Vincent et al. (2010) ; Duan et al. (2014) , and anomaly detection Sakurada and Yairi (2014) ; Xu et al. (2015) . SAEs reconstruct an input \mathbf{x}\in\mathbb{R}^{d} through a hidden representation \mathbf{h}\in\mathbb{R}^{h} , minimizing the reconstruction loss \left\|\mathbf{x}-\hat{\mathbf{x}}\right\|_{2}^{2} while maintaining sparsity in \mathbf{h} , written as \mathbf{\hat{x}}=\mathbf{W}^{\prime}\sigma(\mathbf{W}\mathbf{x}+\mathbf{b}) , where \mathbf{W}\in\mathbb{R}^{h\times d} is the encoder weight matrix, \mathbf{W}^{\prime}\in\mathbb{R}^{d\times h} is the decoder weight matrix, \mathbf{b}\in\mathbb{R}^{h} is the encoder bias, and \sigma is an activation function on \mathbf{h} . Ideally columns of \mathbf{W}^{\prime} correspond to features comprising \mathbf{x} . Recent work has shown that the TopK activation function ( Makhzani and Frey, 2013 ) better approximates the L0 norm in training than alternative techniques such as L1 regularization ( Gao et al., 2024 ) , allowing more precise control over the sparsity of \mathbf{h} . The TopK activation function on \mathbf{h} is defined as: \sigma_{k}(\mathbf{h})_{i}=\begin{cases}h_{i}&\text{if }h_{i}\geq\tau_{k}(\mathbf{h})\\ 0&\text{otherwise}\end{cases} where \tau_{k}(\mathbf{h}) is the k th largest activation in \mathbf{h} . We focus exclusively on SAEs with a TopK activation function, and use the transpose of the encoder weights as the decoder, halving the number of trainable parameters with minimal performance impact as shown by Cunningham et al. (2024) . 2.2 Evaluating Sparse autoencoders Evaluating SAEs is challenging due to the lack of a ground truth for the input features represented by large neural networks. Thus, SAE evaluations act as proxies the extent to which interpretable representation of these input features have been learned in a way that does not require access to them. The reconstruction loss, measured as the Euclidean distance between the SAE input and output, is a widely used metric for the faithfulness of an SAE’s learned representations to the input features represented by a neural network. Although reconstruction loss does not account for the interpretability