Are Sparse Autoencoder Benchmarks Reliable?
David Chanin · 2026
Read Are Sparse Autoencoder Benchmarks Reliable? by David Chanin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18229.txt.
Introduction
David Chanin's 2026 paper audits SAEBench metrics, the standard suite for sparse autoencoders that interpret large language models. Autoencoder progress depends on benchmarks that truly separate better models from worse ones, so metrics face reseed-noise tests. He also checks ground-truth correlation on synthetic models and discriminability across training runs. Two metrics, targeted probe perturbation and spurious correlation removal, fail multiple lenses and should not be used. The rest prove noisier and less discriminating than assumed, with sae-probes k-sparse probing most reliable.
Catalog record
| CREATOR | David Chanin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95442 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are Sparse Autoencoder Benchmarks Reliable? David Chanin Abstract Sparse autoencoders (SAEs) are a core interpretability tool for large language models, and progress on SAE architectures depends on benchmarks that reliably distinguish better SAEs from worse ones. We audit the SAE quality metrics in SAEBench, the de-facto standard SAE evaluation suite, through three complementary lenses: reseed noise on a fixed SAE, ground-truth correlation on synthetic SAEs, and discriminability across training trajectories. We find that two of these metrics, Targeted Probe Perturbation (TPP) and Spurious Correlation Removal (SCR), fail multiple lenses at their canonical settings and should not be used to evaluate SAEs. The other metrics show higher reseed noise and lower discriminability than the field assumes. The sae-probes variant of k k -sparse probing is the most reliable metric we tested, but even sae-probes struggles to separate variants of the same SAE architecture. Our results show the field needs better SAE benchmarks. 1 Introduction Sparse autoencoders (SAEs) decompose a neural network’s activation vector into a sparse sum of feature directions and are a core mechanistic interpretability technique for large language models (LLMs) [ 9 , 3 ] . However, recent work has called into question the quality of SAEs compared to supervised baselines [ 16 , 32 ] , highlighting the need for better SAE architectures. Benchmarking SAEs is harder than most ML evaluation: the “true features” of an LLM are unknown, so we cannot directly tell how well an SAE has learned these “true features”. Instead, the field relies on a number of proxy metrics collated in the SAEBench library [ 17 ] . Architectural progress comes from small comparisons (a different sparsity penalty, a change to the activation function, an improved auxiliary loss), each staking its case on these proxy metrics, with subsequent work building on whichever variant won. When the proxies rank SAEs reliably, this loop compounds into real progress. When they do not, effort flows into changes that look better on a noisy proxy but are not, and real improvements go undetected. Auditing the proxies is therefore a prerequisite to trusting the architectures they rank. While directly auditing proxy metrics is difficult due to not knowing the ground truth “true features” of an LLM, we can still define properties that a good proxy metric should satisfy. These include: 1. Tracks ground-truth. The score should be monotone in the SAE’s ground-truth quality if ground truth is available (e.g. on synthetic models). 2. Increase throughout training. A metric whose score systematically declines through training is responding to something other than quality. 3. Low reseed noise. The same SAE evaluated on the same data should produce the same score regardless of any random seed used. 4. Discriminative. The metric should distinguish runs by an amount substantially larger than its own per-checkpoint jitter. 5. Internally consistent. The metric should rank SAEs the same way regardless of nominally-interchangeable hyperparameters. In this work, we audit six SAEBench evaluations that target overall SAE quality: sparse probing [ 13 ] , the sae-probes variant of sparse probing [ 16 ] , targeted probe perturbation (TPP) [ 23 ] , spurious correlation removal (SCR) [ 23 ] , automated interpretation [ 1 ] , and RAVEL [ 15 ] . The five desiderata above cannot all be tested on a single SAE panel: low reseed noise needs the same SAE evaluated many times, ground-truth tracking needs SAEs whose ground-truth quality is known, and discriminability needs SAEs that vary in quality. We therefore approach the audit through three complementary lenses, each on a different SAE panel and each isolating a different failure mode: Reseed noise on a real LLM SAE (§ 3 , desideratum 3). We run each SAEBench evaluation five times with different random seeds on a fixed canonical Gemma Scope SAE [ 21 ] (replicated on three other canonical SAEs across model families in Appendix C ). Because the SAE is identical across runs, any score spread is pure metric noise; the resulting per-metric CV spans nearly two orders of magnitude and yields a minimum-reliable-difference threshold for single-seed comparisons. Validity on synthetic SAEs (§ 5 , desideratum 1). Using SynthSAEBench-16k [ 7 ] , a synthetic model whose activations are sparse linear combinations of a known 16k-feature dictionary, we train a panel of SAEs with computable ground-truth quality and check whether each proxy metric’s score correlates with that ground truth. This is the only setting in which we can adjudicate desideratum 1 directly; autointerp and RAVEL cannot be tested in this lens because both require natural-language concepts the synthetic dictionary lacks. Discriminability across training (§ 4 , desiderata 2, 4, and 5). We train two panels of SAEs and snapshot them throughout training: a four-SAE cross-architecture panel with deliberately large differences (BatchTopK vs Matryoshka, k\!\in\!\{50,100\} ), and a four-SAE Matryoshka panel that varies a single hyperparameter n across three independent training seeds per variant. The cross-architecture panel asks whether the metric can tell apart very different SAEs; the Matryoshka panel asks the harder, more practically relevant question of whether it can tell apart the small hyperparameter variants practitioners typically compare. The training-trajectory axis lets us check desideratum 2 directly, and the three Matryoshka seeds let us decompose total score variance into between-SAE signal versus training-seed and snapshot noise. Across the three lenses, TPP and SCR fail multiple desiderata at canonical settings: TPP scores worse the more an SAE is trained, and SCR becomes negatively correlated with ground-truth at large top- N . Every other metric is noisier and less discriminative than the field assumes. The sae-probes variant of sparse probing is the most reliable metric we audit; its two main design choices, a much larger dataset suite (113 vs 5) and cross-validated probe regularization, suggest a concrete path forward for the rest of the suite. Our code is available via Github 1 1 1 Code: https://github.com/decoderesearch/saebench-reliability-audit , and SAE panels are available on Huggingface 2 2 2 SAE panels: https://huggingface.co/decoderesearch/sae-snapshot-panels . 2 Background Sparse autoencoder A sparse autoencoder maps an activation vector x\in\mathbb{R}^{d} to a sparse latent vector z=\sigma(W_{\text{enc}}x+b_{\text{enc}})\in\mathbb{R}^{d_{\text{sae}}} with d_{\text{sae}}\geq d , and reconstructs \hat{x}=W_{\text{dec}}z+b_{\text{dec}} . SAEs are trained to minimize reconstruction error subject to a sparsity penalty on z , with the goal of recovering interpretable feature directions in the underlying model’s activation space [ 3 ] . By “SAE quality” we mean the degree to which the trained latent directions align with semantically meaningful features. 2.1 SAEBench metrics SAEBench [ 17 ] measures quality through six proxy metrics 3 3 3 We do not audit the unlearning metric as it requires chat models while the other metrics use base models. We do not audit absorption and meta structure metrics as these are meant to diagnose a specific failure and are not general quality metrics. . Unless otherwise specified, we leave these at SAEBench default settings. Detailed descriptions of each metric are in Appendix B . Sparse probing Trains a k -sparse logistic regression probe [ 13 ] on the SAE latents to classify natural-language attributes such as language or profession; higher accuracy at small k means concepts align with small numbers of latents. This benchmark uses 5 probing datasets. Sae-probes sparse-probing A new metric added to SAEBench after publication, this is a