C 2 R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian · 2026
Read C 2 R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders by Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30609.txt.
Introduction
Haoran Jin and four colleagues address two failure modes of sparse autoencoders, tools that unpack language-model activations into sparse, readable features. As dictionaries grow, coherent concepts split across several latents, a flaw called feature splitting. Other features get absorbed, gaining odd exceptions that make them unreliable. Both flaws trace back to inconsistent assignment: the same concept lands on different latents in different samples.
Their remedy, C2R, adds cross-sample consistency regularization during training. It penalises the joint firing of latents that point in similar directions. This pushes each concept toward one shared latent across the whole batch. Wide evaluation shows C2R reduces both splitting and absorption. Crucially, rebuild quality stays intact: the autoencoder still reproduces the original activations faithfully. The authors present this as a fix that improves interpretability without hurting model performance.
Catalog record
| CREATOR | Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
C 2 R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian Abstract Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges. Systematic studies reveal pervasive feature splitting that fragments coherent concepts into non-atomic latents and widespread feature absorption that creates arbitrary exceptions in general features, severely compromising latent reliability. These issues stem from inconsistent latent assignment across samples: without cross-sample constraints, per-sample optimization often allows a single underlying concept to be inconsistently distributed across multiple redundant or interfering latents. To address this, we introduce C 2 R ( C ross-sample C onsistency R egularization). C 2 R explicitly encourages that each semantic feature is consistently represented by a unified latent across the batch by penalizing the co-activation of directionally similar latents. Comprehensive evaluation demonstrates that C 2 R effectively mitigates both splitting and absorption while, crucially, preserving reconstruction fidelity, providing a principled solution that enhances latent interpretability without degrading model performance. Source code is available * * * https://github.com/hr-jin/Cross-sample-Consistency-Regularization . 1 Introduction SAE Constraints Theoretical Guarantee Intuitively Solved Reconstruction Preservation Splitting Absorption Splitting Absorption \ell_{1} \times \times \times \times \checkmark TopK \times \times \times \times \checkmark Batch TopK \times \times \times \times \checkmark Matryoshka \times \times \checkmark \checkmark \times Ort \times \times \times \checkmark \checkmark Ours \checkmark^{*} \checkmark^{*} \checkmark \checkmark \checkmark Table 1 : Comparison of different SAE constraints in terms of theoretical guarantees, intuitive solutions, and reconstruction fidelity. Our cross-sample consistency regularization uniquely offers a theoretical guarantee against splitting and absorption and preserves reconstruction fidelity. ∗ Theoretical guarantee holds under the condition in Eq. 13 , which is empirically satisfied in 88.1% of absorption pairs (see Appendix K ). Sparse Autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models (LLMs), breaking down complex internal representations into sparse, interpretable features ( 10 ; 1 ) . These features provide valuable insights into reasoning, alignment ( 25 ; 24 ; 23 ) , knowledge awareness, hallucinations ( 4 ) , and cross-model feature spaces ( 14 ) of LLMs. This approach is grounded in the hypothesis that the semantic features in a model’s representation are more effectively captured by an overcomplete sparse basis ( 16 ) than by dense neuron activations, which tend to be polysemantic. Ideally, each latent in an SAE corresponds to a single, human-interpretable concept. While SAEs are effective, they face significant challenges, specifically feature splitting ( 1 ; 15 ) and feature absorption ( 3 ) , which undermine the reliability of learned latents. Feature splitting fragments coherent, high-level concepts into overly specific pieces. For example, a single “Mathematics” feature might break down into separate latents for “Algebra,” “Geometry,” and others ( 3 ) . Although these granular features are interpretable, this fragmentation is problematic because it obscures the true high-level concept the model functionally uses. Moreover, this is inefficient: the dictionary wastes capacity on redundant variations of known concepts instead of finding new ones. Feature absorption, on the other hand, creates “holes” in general features when specific latents capture their activations. A “starts with S” latent, for instance, might fail to activate on “short” or “small” because token-specific latents absorb the signal. This effectively changes the latent to “starts with S (except for short/small),” distorting the intended pattern. 15 show that splitting scales with model size, while 3 find that absorption affects hundreds of LLM SAEs. These systematic failures undermine the utility of SAEs for critical tasks like causal analysis and circuit discovery. We argue that these failures arise from a mismatch between the hierarchical nature of language model features and the local scope of standard sparsity constraints. Real-world concepts are inherently hierarchical ( 2 ) , yet common objectives like \ell_{1} ( 1 ) or TopK ( 6 ) enforce sparsity on a per-sample basis, which potentially penalizes the hierarchical structure. activating both uses more of the sparsity budget than activating the child feature alone. Consequently, the optimizer suppresses the parent latent and forces the child latent to take over its role to keep the active count low. Similarly, regarding feature splitting, the objective does not distinguish between activating a general latent or a specific one. The SAE allows disjoint latents to handle different contexts of a single concept, as nothing ensures consistent latent assignment across samples. Solving these issues, therefore, requires looking beyond per-sample optimization to enforce cross-sample consistency in how latents are selected. To address this issue, we propose C 2 R ( C ross-sample C onsistency R egularization). This objective builds on the geometry of the Minkowski inequality ( 8 ) and the strict convexity of the \ell_{2} norm. It exploits the fact that the sum of the norms of separate vectors strictly exceeds the norm of their sum: \|u\|_{2}+\|v\|_{2}>\|u+v\|_{2} for non-aligned vectors. By applying this constraint across the batch dimension, C 2 R makes it expensive to split a concept into multiple disjoint latents. This formulation penalizes spreading semantic information across redundant latents, driving the SAEs to consolidate activations into a single, consistent latent without supervision. Our contributions are threefold: • Theoretical diagnosis: We identify the lack of cross-sample consistency in per-sample sparsity objectives as the root cause of feature splitting and absorption, providing a unified formal analysis of these phenomena. • Principled objective: We propose C 2 R, a novel regularization objective that utilizes decoder geometry and batch-level statistics to enforce consistent latent selection, effectively distinguishing between true polysemanticity and harmful redundancy. • Empirical validation: We demonstrate that C 2 R significantly mitigates splitting and absorption, achieving better feature hierarchy without compromising reconstruction fidelity compared to state-of-the-art baselines. 2 Related Work 2.1 Sparse Autoencoders Sparse Autoencoders (SAEs) are grounded in the linear representation hypothesis, which posits that the dense activation space of a language model is constructed from the superposition of sparse, discernible concepts, referred to as features . The goal of an SAE is to recover these ground-truth features by learning a dictionary of latents . Ideally, there exists a one-to-one mapping where each learned latent corresponds precisely to a single meaningful feature. Formally, given an input activation vector x\in\mathbb{R}^{d_{model}} (e.g., from a Transformer’s residual stream), an SAE projects x into a higher-dimensional sparse latent code f\in\mathbb{R}^{d_{dict}} , where d_{dict}\gg d_{model} . The encoding process is parameterized by an encoder weight matrix W_{e}\in\mathbb{R}^{d_{dict}\times d_{model}} and a bias b_{e} : f=\phi(W_{e}x+b_{e}), (1) where \phi is a non-linear activation function, typically ReLU, TopK, or JumpReLU, designed to induce sparsity. The input is then reconstructed via a linear decoder W_{d}\in\mathbb{R}^{d_{model}\times d_{dict}} : \hat{x}=W_{d}f+b_{d}. (2) The training objective minimizes a combination of reconstruction error and a sparsity penalty: \mathcal{L}_{\text{SAE}}(x)=\|x-\hat{x}\|_{2}^{2}+\lambda\mathcal{R}(f). (3) Common choices for the regularizer \mathcal{R}(f) include the \ell_{1} norm ( 1 ) or the auxiliary loss associated with TopK constraints ( 6 ) . Various architectural improvements have been proposed to enhance SAE quality. Gated SAEs ( 20 ) and JumpReLU SAEs ( 21 ) introduce learnable thresholds to improve the fidelity-sparsity frontier. However, these methods focus on the per-sample activation sparsity rather than enforcing the hierarchical structure of the SAE. 2.2 Structural SAEs More recently, approaches attempting