Sparse Autoencoders, Again?

Yin Lu, Xuening Zhu, Tong He, David Wipf · 2025

Read Sparse Autoencoders, Again? by Yin Lu, Xuening Zhu, Tong He, David Wipf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04859.txt.

Introduction

Yin Lu, Xuening Zhu, Tong He, and David Wipf take a fresh look at sparse autoencoders. These nets learn a small set of hidden traits in data. The traits can show shared patterns in large language models. They can also trace the bent shapes that images form. Yet the base plan has hardly changed in decades. It pairs a plain coder and decoder with a fixed rule that keeps the code sparse.

The team pins down weak points that past work missed. The flaws show up in the base nets and in a close kin that uses chance in its coder. Their mixed model steps past those flaws. Proofs show its best fits can rebuild data spread on a union of shapes. Tests on made up and real data gauge each shape more truly. The model also makes sparser codes at the same rebuild error, and beats both older kinds.

Catalog record

CREATORYin Lu, Xuening Zhu, Tong He, David Wipf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04859

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse Autoencoders, Again? Yin Lu, Xuening Zhu, Tong He, David Wipf Abstract Is there really much more to say about sparse autoencoders (SAEs)? Autoencoders in general, and SAEs in particular, represent deep architectures that are capable of modeling low-dimensional latent structure in data. Such structure could reflect, among other things, correlation patterns in large language model activations, or complex natural image manifolds. And yet despite the wide-ranging applicability, there have been relatively few changes to SAEs beyond the original recipe from decades ago, namely, standard deep encoder/decoder layers trained with a classical/deterministic sparse regularizer applied within the latent space. One possible exception is the variational autoencoder (VAE), which adopts a stochastic encoder module capable of producing sparse representations when applied to manifold data. In this work we formalize underappreciated weaknesses with both canonical SAEs, as well as analogous VAEs applied to similar tasks, and propose a hybrid alternative model that circumvents these prior limitations. In terms of theoretical support, we prove that global minima of our proposed model recover certain forms of structured data spread across a union of manifolds. Meanwhile, empirical evaluations on synthetic and real-world datasets substantiate the efficacy of our approach in accurately estimating underlying manifold dimensions and producing sparser latent representations without compromising reconstruction error. In general, we are able to exceed the performance of equivalent-capacity SAEs and VAEs, as well as recent diffusion models where applicable, within domains such as images and language model activation patterns. A link to the code is here . 1 Introduction Autoencoders represent a widely-applicable neural network model for obtaining useful low-dimensional representations of data, particularly when training labels are not available ( Berahmand et al. 2024 ; Goodfellow et al. 2016 ) . The basic architecture is simple: An encoder network first projects each observable data sample to a latent representation (usually low-dimensional), after which a decoder network attempts sample reconstruction based on this latent representation. Sparse autoencoders (SAE) ( Ng 2011 ; Ranzato et al. 2006 ; Ranzato et al. 2007 ) adopt an additional degree of design flexibility by accommodating a higher-dimensional latent space, but with the inclusion of an attendant regularization factor applied to the encoder output that encourages sparse representations (meaning most dimensions are pushed to zero). This capability facilitates adaptive latent representations reflecting complex underlying data structure, whereby the pattern of nonzeros can vary on a sample-by-sample basis unlike with a regular autoencoder. Informally speaking, autoencoders (including SAEs) can be viewed as something like the horseshoe crabs of the deep learning kingdom, having existed for decades (or “eons”) with more or less the same hardy conceptual design ( Bourlard & Kamp 1988 ; Hinton & Zemel 1993 ; LeCun 1987 ; Rumelhart et al. 1986 ) . For example, the SAE models used for learning interpretable representations of present-day large language model (LLM) activation patterns ( Bricken et al. 2023 ; Cunningham et al. 2024 ; Chaudhary & Geiger 2024 ; Gao et al. 2024 ) are nearly identical in form to original architectures from long ago; likewise for a variety of other modern use cases ( Lan et al. 2024 ; Movva et al. 2025 ; Pach et al. 2025 ; Stevens et al. 2025 ) . One notable exception is the variational autoencoder (VAE) ( Kingma & Welling 2014 ; Rezende et al. 2014 ) , which substitutes the deterministic latent space of traditional autoencoders, with an alternative stochastic representation. Although seemingly distinct from the SAE, it has recently been shown that VAEs can also address sparse autoencoding tasks in certain settings ( Dai et al. 2021 ) . Still, VAE models notwithstanding, given the long-lasting and durable usage of SAEs with minimal changes, it is natural to question the extent to which further innovation is possible on top of the original SAE design space, or if any such innovation is even needed at all. We answer such questions in the affirmative based on the observation that both SAE and VAE designs have notable unresolved, yet distinct, shortcomings. While details will be deferred to Section 2 , we summarize here that SAEs generally rely on multiple hyperparameters and potentially-nonconvex regularization factors that complicate both training and the interpretation of the latent representations produced by the encoder. In contrast, VAEs benefit from a hyperparameter-free and smoother loss surface, but are presently ill-equipped to produce adaptive sparse representations like the SAE. With this backdrop in mind, our contributions herein are encapsulated as follows: • Algorithmic: Although both SAE and VAE models can reconstruct data using sparse encoder representations, we carefully detail their disjoint strengths and weaknesses in Section 2 ; subsequently we propose a hybrid VAE-like approach in Section 3 that delivers the best of both worlds. We refer to our model as VAEase , for an easy -to-apply VAE with a daptive s parse e stimation . • Theoretical: In Section 4 we rigorously prove that for data residing on a union of low-dimensional manifolds, globally-optimal VAEase solutions can provably recover the underlying manifold structure, adapting to multiple distinct manifolds, unlike existing VAEs. Likewise, we demonstrate simple illustrative settings whereby VAEase maintains fewer local minima relative to an analogous deterministic SAE model. • Empirical: Finally, we present complementary experiments in Section 5 that show VAEase outperforms existing SAE and VAE architectures, as well as related diffusion models, across tasks such as estimating manifold dimensions or maximally sparse representations of large language model activations. We conclude this section by emphasizing that, although our proposed approach explicitly relies on a variational chassis, the work itself as a whole is not directly related to generative modeling per se, the original purpose for which VAEs were designed. Instead, we are repurposing specific VAE attributes for new sparse autoencoding tasks. 2 Basics of Sparse Autoencoding In this section we will first present a broad SAE formulation covering typical use cases. Later, we will demonstrate how VAEs can be leveraged for similar purposes, before concluding with a brief contextualization w.r.t. related diffusion models, as have recently been applied to learning manifold dimensions. 2.1 Canonical SAE Formulation Suppose we collect d -dimensional data points \mathbf{x}\in{\mathcal{X}}\subseteq\mathbb{R}^{d} drawn from some probability measure \omega such that \int_{\mathcal{X}}\omega(d\mathbf{x})=1 . Like traditional autoencoders ( Bourlard & Kamp 1988 ; LeCun 1987 ; Rumelhart et al. 1986 ) , an SAE model is composed of an encoder that maps samples \mathbf{x} to a latent representation \mathbf{z}\in\mathbb{R}^{\kappa} , and a decoder that subsequently attempts to reconstruct \mathbf{x} from \mathbf{z} . These modules are instantiated as functions given by \bm{\mu}_{z}(~\cdot~;\phi):{\mathcal{X}}\rightarrow\mathbb{R}^{\kappa} and \bm{\mu}_{x}(~\cdot~;\theta):\mathbb{R}^{\kappa}\rightarrow\mathbb{R}^{d} respectively, with the goal of learning parameters \{\theta,\phi\} such that \mathbf{x}\approx\bm{\mu}_{x}\big[\bm{\mu}_{z}(\mathbf{x};\phi);\theta\big] for all \mathbf{x}\in{\mathcal{X}} . 1 1 1 We remark that the range of \bm{\mu}_{x} need not be {\mathcal{X}} exactly, and perfect reconstructions will not necessarily be feasible at all times. What differentiates SAEs from regular auto-encoders (AEs) is that, in addition to seeking an encoder-decoder pairing capable of accurately reconstructing data samples, we also favor latent representations \mathbf{z}=\bm{\mu}_{z}(\mathbf{x};\phi) that are sparse ( Goodfellow et al. 2016 ; Ng 2011 ; Ranzato et al. 2006 ; Ranzato et al. 2007 ) . More formally, we ideally seek \|\mathbf{z}\|_{0}\ll\kappa (or at least approximately so), where \|\cdot\|_{0} denotes the \ell_{0} -norm that counts the number of nonzero elements

Read the complete text