Structural Instability of Feature Composition
Yunpeng Zhou · 2026
Read Structural Instability of Feature Composition by Yunpeng Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05223.txt.
Introduction
Yunpeng Zhou studies why switching on several learned features at once in a transformer can become unstable, a question at the heart of steering sparse autoencoder units. Such autoencoders pull apart stacked features, yet the theory of composing distinct meaning units together is thin. The usual linear picture of inner codes ignores nonlinear cross-talk inside too-large codebooks.
Zhou models the space as a large sparse cone and derives an asymptotic threshold for collapse when parts combine, under a round codebook model. In a high-bias regime, he shows the ReLU step turns small linked wobbles into a steady drift that grows under composition, a ratchet effect. Tests on structured units from the CLEVR dataset match the forecast scaling, with layered correlations speeding the shift.
Catalog record
| CREATOR | Yunpeng Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structural Instability of Feature Composition Yunpeng Zhou Abstract Sparse Autoencoders (SAEs) have emerged as a powerful paradigm for disentangling feature superposition in transformer-based architectures, enabling precise control via activation steering. However, the theoretical foundations of compositional steering—the simultaneous activation of distinct semantic latents—remain under-explored. The prevailing Linear Representation Hypothesis often abstracts away non-linear interference effects that arise in overcomplete dictionaries. We present a geometric framework for analyzing the instability of feature unions. Modeling the activation space as a high-dimensional sparse cone manifold, we derive an asymptotic compositional-collapse threshold under a spherical dictionary model, characterized by the Gaussian mean width (statistical dimension) of the signal cone. We further show that, in the high-bias regime, ReLU rectification converts microscopic correlation-induced variance fluctuations into a systematic drift that accumulates under composition, yielding interference growth consistent with a ratchet effect. We validate the predicted scaling trends on structured semantic features extracted from CLEVR, where hierarchical correlations accelerate the transition relative to random baselines. Together, our results highlight geometric constraints on the scalability of union-based steering and motivate composition mechanisms that explicitly manage interference beyond naive linear superposition. \coltauthor Whiteknights House, Whiteknights, Reading, RG6 6UR keywords Sparse Representation, Mechanistic Interpretability, Phase Transitions, Compositional Generalization, Feature Steering 1 Introduction The field of interpretability has shifted toward manipulating dynamic activations via Sparse Autoencoders (SAEs), which decompose dense residual streams into interpretable features. Supported by recent scaling laws and resources ( 18 ; 7 ; 12 ) , this decomposition enables activation steering to elicit specific model behaviors ( 14 ; 20 ) . These interventions typically rely on the Linear Representation Hypothesis ( 13 ) , assuming semantic composition equals linear vector addition. However, the geometric stability of this hypothesis is fragile. Evidence suggests features are often multi-dimensional ( 6 ) and lack causal unity ( 10 ; 11 ) . Crucially, simultaneous manipulation of multiple features frequently triggers compositional collapse —a state of semantic incoherence ( 16 ) . While prior work noted interference in structured manifolds ( 5 ; 3 ) , a rigorous geometric characterization of this instability remains elusive. We argue that compositional breakdown is a fundamental consequence of high-dimensional, overcomplete geometry ( m\gg n ). In this regime, inevitable non-orthogonality causes interference noise ( 15 ) . Modeling the activation space as a sparse cone manifold , we show that ReLU non-linearities act as a Ratchet : unlike linear systems where noise cancels out, rectification bias exponentially amplifies geometric interference, rendering the system strictly super-additive to noise. Our contributions are threefold: (1) We establish a Stability Threshold via the Gaussian Mean Width of the signal cone, proving spurious activations are inevitable beyond a critical compositional density; (2) We characterize the Ratchet Mechanism, providing a micro-foundation for how marginal correlations amplify into macroscopic collapse; and (3) We provide Empirical Validation on CLEVR, demonstrating that structured semantic correlations accelerate this phase transition compared to random baselines. These findings establish a hard geometric barrier to scaling naive linear composition in static embedding spaces. 2 Preliminaries We adopt standard notation from high-dimensional probability. Let [m] denote \{1,\dots,m\} . For a vector v\in\mathbb{R}^{n} , \|v\|_{p} denotes its \ell_{p} norm. For a matrix A , \|A\|_{op} denotes its spectral norm, and A_{S} denotes the submatrix restricted to columns indexed by S\subset[m] . 2.1 The Overcomplete Dictionary Model We analyze the activation space \mathbb{R}^{n} where semantic features are encoded by an overcomplete dictionary D\in\mathbb{R}^{n\times m} with m=\delta n ( \delta>1 ). Assumption 1 (Dictionary Regularity) . The dictionary D=[d_{1},\dots,d_{m}] satisfies: 1. Unit Norm: \|d_{i}\|_{2}=1 for all i\in[m] . 2. \mu -Incoherence: The mutual coherence \mu(D)\triangleq\max_{i\neq j}|\langle d_{i},d_{j}\rangle| scales as O(1/\sqrt{n}) . Random Baseline vs. Structural Reality. We primarily utilize the random spherical model (where d_{i}\sim\text{Unif}(\mathbb{S}^{n-1}) ) to derive asymptotically tight phase-transition bounds. While real-world semantics (e.g., CLEVR) exhibit hierarchical correlations that deviate from this i.i.d. assumption, we address the robustness of our bounds under such structural perturbations empirically in Section 4.5 . 2.2 The Sparse Cone Manifold Unlike classical Compressed Sensing focused on linear subspaces, neural activations are constrained by the ReLU function \sigma(x)=\max(0,x) , restricting signals to a union of positive cones. Definition 1 (Positive Feature Cone) . For a support set S\subset[m] , the feature cone \mathcal{C}(S) is the set of non-negative linear combinations of atoms in S : \mathcal{C}(S)\triangleq\left\{\sum_{i\in S}\alpha_{i}d_{i}\mid\alpha_{i}>0\right\}\subset\text{span}(D_{S}). Definition 2 ( k -Sparse Manifold) . The set of valid k -sparse representations forms a non-convex manifold: \mathcal{M}_{k}\triangleq\bigcup_{S\subset[m],|S|\leq k}\mathcal{C}(S). Our stability analysis (Section 4 ) centers on whether the composition z=z_{A}+z_{B} remains close to \mathcal{M}_{2k} or falls into the invalid ambient void, triggering undefined features. 2.3 Composition and Interference We formalize “steering“ as the algebraic addition of latent vectors. Let \alpha^{*}\in\mathbb{R}^{m} be a sparse coefficient vector supported on S . The pre-activation is x=D\alpha^{*} . When composing distinct concepts supported on disjoint sets S_{A} and S_{B} , the idealized state is x_{union}=D(\alpha_{A}+\alpha_{B}) . Physical realization is subject to interference from Ghost Features . Let J=[m]\setminus(S_{A}\cup S_{B}) be the set of inactive atoms. The projection of the composite signal onto a ghost atom d_{j} ( j\in J ) is: \mathcal{I}_{j}(S_{A},S_{B})\triangleq\langle x_{union},d_{j}\rangle=\underbrace{\langle D_{S_{A}}\alpha_{A},d_{j}\rangle}_{\text{Interference from A}}+\underbrace{\langle D_{S_{B}}\alpha_{B},d_{j}\rangle}_{\text{Interference from B}}. A spurious activation occurs if \mathcal{I}_{j}>\beta (the activation bias). The core mathematical challenge is to characterize the distribution of the maximum interference \sup_{j\in J}\mathcal{I}_{j} as a function of the sparsity k=|S_{A}|+|S_{B}| . For a summary of notation, see Table 1 in Appendix A.1 . 3 Micro-Analysis: The Geometry of Pairwise Entanglement In this section, we analyze the local interaction between two distinct semantic factors. Before establishing the macroscopic phase transition (Section 4 ), we must first quantify the geometric mechanism by which the union of two sparse cones generates interference. Consider two disjoint index sets S_{A},S_{B}\subset[m] with cardinalities k_{A},k_{B}\ll n . Let \mathcal{U}_{A}=\text{span}(D_{S_{A}}) and \mathcal{U}_{B}=\text{span}(D_{S_{B}}) be the subspaces spanned by the active atoms. The fundamental challenge of compositional steering lies in the fact that while S_{A}\cap S_{B}=\emptyset , their embedded subspaces are not orthogonal: \mathcal{U}_{A}\perp\mathcal{U}_{B} does not hold in general for overcomplete dictionaries. 3.1 Principal Angles and Subspace Alignment To rigorously measure the geometric antagonism between two concepts, we invoke the notion of principal angles . The alignment between \mathcal{U}_{A} and \mathcal{U}_{B} dictates the worst-case interference. Definition 3 (Interaction Singular Values) . Let Q_{A} and Q_{B} be orthonormal bases for \mathcal{U}_{A} and \mathcal{U}_{B} , respectively. The interaction singular values \sigma_{1}\geq\sigma_{2}\geq\dots\geq\sigma_{\min(k_{A},k_{B})} are the singular values of the cross-projection matrix M_{AB}=Q_{A}^{T}Q_{B} . The smallest principal angle \theta_{\min} satisfies \cos(\theta_{\min})=\sigma_{1} . If \sigma_{1}\approx 1 , the subspaces are nearly parallel, making disentanglement ill-posed. However, in high-dimensional random dictionaries, \sigma_{1} is typically bounded. The danger arises not from the subspaces collapsing onto each other, but from their joint projection onto the complementary dictionary atoms. 3.2 The Leakage Operator and Variance Decomposition We define the Leakage Operator \mathcal{L}_{AB}:\mathcal{U}_{A}\times\mathcal{U}_{B}\to\mathbb{R}^{m-(k_{A}+k_{B})} to quantify the signal bleeding into the ghost features J=[m]\setminus(S_{A}\cup S_{B}) . For any composite steering vector z=D\alpha_{A}+D\alpha_{B} , the pre-activation on a ghost atom d_{j} ( j\in J ) is defined by the inner product \langle z,d_{j}\rangle . To characterize the structural instability, we analyze the second-order moments of this interference. Let G=D^{T}D be the Gram matrix of the dictionary. The interference energy at a ghost feature j depends not only on individual atom coherence