Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir · 2026
Read Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations by Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24716.txt.
Introduction
Jonas Klotz and four colleagues test whether sparse autoencoders really extract concepts people can recognise. Sparse autoencoders are used to pull interpretable features out of vision and vision language models. Most checks of their quality rely on proxy metrics or eyeballing examples. The team instead builds a human grounded framework. It measures how well the learned features, called latents, line up with human annotated concepts, with no user studies needed.
Two synthetic benchmarks, synCUB and synCOCO, supply image pairs differing in exactly one attribute. A new matching procedure, Fully Binary Matching Pursuit, links latents to concepts and allows many to one matches. It beats one to one baselines throughout. A second score checks that matched concepts react in the expected direction when an attribute is perturbed. Under sanity checks, only this pairing separates trained autoencoders from untrained ones. Much larger dictionaries, Klotz and colleagues find, can actually reduce interpretability.
Catalog record
| CREATOR | Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24716
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir Abstract Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence. We present a human-grounded evaluation framework that quantifies alignment between SAE latents and human-annotated concepts, without requiring user studies, and validate this matching through targeted attribute perturbations. To enable this intervention-style evaluation in vision, we construct synCUB and synCOCO, synthetic benchmarks of paired images that differ in exactly one attribute. We introduce Fully-Binary Matching Pursuit (FBMP), a coalition-based matching procedure that supports many-to-one mappings between SAE latents and annotated concepts, and consistently outperforms one-to-one baselines. For functional validation, we propose a Targeted Attribute Perturbation Alignment Score (TAPAScore), which tests whether matched concepts respond selectively and in the expected direction under targeted image-level attribute perturbations. Under sanity checks, our matching and TAPAScore are the only evaluated metrics that reliably distinguish trained SAEs from untrained ones. Across SAEs trained on CLIP and DINOv2 embeddings, we find that increased overcompleteness can reduce perturbation alignment, indicating a reduction in interpretability. Our evaluation framework suggests that moderate dictionary sizes provide the best trade-off, yielding the most interpretable SAEs. Code and datasets are available at https://github.com/JonasKlotz/sae-concept-eval . 1 Introduction Figure 1 : Human-grounded evaluation of SAE interpretability. The framework comprises three components: (1) latent-concept matching, where binarized SAE latents are aligned to ground truth attribute vectors; (2) synthetic datasets, where paired images are constructed to differ in exactly one concept (e.g., belly pattern changed from solid to striped); and (3) targeted attribute perturbation alignment, where the induced latent change is measured via the signed responses \delta_{\text{add}} and \delta_{\text{rem}} . TAPAScore combines these components to quantify the perturbation alignment. Large vision and vision-language models such as DINOv2 [ 35 ] and CLIP [ 39 ] rely on high-dimensional internal representations whose structure is difficult to interpret. Understanding what individual latent dimensions represent is therefore critical for trust, controllability, and scientific analysis of these models [ 44 ] . One promising approach is the use of sparse autoencoders (SAEs) to inspect their internal representations, a technique first developed for large language models [ 5 , 21 ] . By enforcing sparsity, SAEs decompose high-dimensional activations into more localized and disentangled components that can be aligned with human-interpretable concepts. Following their success in language-model interpretability, SAEs are being applied to vision encoders to identify sparse latents associated with object parts, textures, and attributes [ 13 , 3 , 28 , 41 ] . However, SAEs are known to exhibit systematic failure modes, such as feature splitting, where a single concept is fragmented across multiple latents [ 5 ] ; feature absorption, where a general feature develops blind spots covered by specialized latents [ 10 ] ; and feature composition, where co-occurring concepts merge into a single latent [ 53 ] . SAE features are furthermore often organized hierarchically, with both fine-grained and higher-level abstractions distributed across latents [ 8 ] . Unless the SAE captures inductive biases that reflect the semantic structure present in the data, there is no guarantee that learned features will align with the concepts that human observers identify as meaningful [ 14 ] . Consequently, we claim that the alignment between SAE features and human-understandable concepts cannot be assumed from architectural design or reconstruction quality alone, but must be evaluated explicitly. Such explicit evaluation requires metrics that capture semantic correspondence. Existing SAE metrics, developed primarily in the language domain, fall into structural and functional families [ 46 ] . Whereas structural metrics, which assess representational properties such as sparsity or reconstruction fidelity, transfer naturally from language to vision, functional evaluation requires controlled interventions that isolate a single semantic change. This condition is particularly difficult to satisfy in vision, where attribute changes in images rarely occur in isolation. As a result, SAE evaluation for vision models remains dominated by qualitative examples and structural proxies, which measure representational organization rather than whether features correspond to semantic concepts. To address this limitation, we operationalize interpretability as the degree to which learned SAE features align with human-understandable concepts present in the data, as in prior work on representation interpretability [ 2 , 23 ] . Human evaluation is considered the gold standard of interpretability assessment [ 11 ] , and human-annotated concepts/attributes provide a practical measure of this standard, as they precisely reflect the factors that human observers consistently identify as meaningful. Figure 1 summarizes our proposed evaluation framework. We first quantify semantic correspondence by matching binarized SAE activations to binary ground truth attribute vectors. We then construct controlled image pairs that differ in exactly one attribute, enabling intervention-style tests where the target change is precisely known. Finally, we introduce TAPAScore, which measures whether the latents previously matched to the perturbed attribute respond in the expected direction. In summary, our contributions are : • We propose matching metrics between SAE latents and human-annotated attributes, including a coalition-based formulation, fully-binary matching pursuit (FBMP), that supports many-to-one mappings. We find that existing metrics fail basic sanity checks that our proposed metrics pass (Fig. 4 ) and that FBMP consistently outperforms one-to-one matchings (Figs. 5 , 6 , top ). • We construct two synthetic datasets, synCUB and synCOCO, consisting of paired images that differ in exactly one attribute or object label, enabling intervention-style evaluation of SAEs in vision. • We propose a targeted attribute perturbation alignment score (TAPAScore) that tests whether matched latent sets respond selectively and in the right direction under targeted attribute perturbations. We show that increased overcompleteness can degrade perturbation alignment (Figs. 5 , 6 , bottom ), and that matching score and TAPAScore are positively correlated (Fig. 7 ). Beyond the framework itself, our evaluation yields consistent empirical guidance for practitioners: increasing overcompleteness improves statistical matching but tends to degrade perturbation alignment; moderate dictionary sizes offer the best trade-off. Based on these findings, we recommend FBMP F_{0.5} matching paired with TAPAScore as the default evaluation protocol. 2 Background and Related Work Sparse Autoencoders (SAEs) [ 5 , 21 ] are a method to solve sparse dictionary learning, where signals are represented as sparse linear combinations of basis elements (atoms) from an overcomplete dictionary [ 43 , 48 ] . More recently, SAEs have been used to identify interpretable concepts in neural network activations. This is motivated by evidence that individual neurons are frequently polysemantic and encode multiple unrelated concepts, suggesting that deep networks store information in superposed representations [ 12 ] . SAEs disentangle these representations by learning a sparse and overcomplete representation in which the number of learned features exceeds the dimensionality of the activation space. SAEs implement dictionary learning through a neural network consisting of an encoder \mathbf{W}_{\text{enc}}\in\mathbb{R}^{D\times L} , decoder \mathbf{W}_{\text{dec}}\in\mathbb{R}^{L\times D} , and shared bias \mathbf{b}\in\mathbb{R}^{D} . Given an embedding \mathbf{x}\in\mathbb{R}^{D} from a pretrained model, the SAE computes sparse activations \mathbf{z}\in\mathbb{R}^{L} and the corresponding reconstruction \hat{\mathbf{x}} as: \mathbf{z}=\sigma\!\left(\mathbf{W}_{\text{enc}}^{\top}(\mathbf{x}-\mathbf{b})\right),\qquad\hat{\mathbf{x}}=\mathbf{W}_{\text{dec}}^{\top}\mathbf{z}+\mathbf{b}. (1) The model parameters are learned by minimizing an objective \mathcal{L}(\mathbf{x})=\mathcal{R}(\mathbf{x})+\lambda\mathcal{S}(\mathbf{x}) with a reconstruction and a sparsity-inducing term. SAE architectures primarily differ in how sparsity is enforced. Vanilla SAEs [ 5