From superposition to sparse codes: interpretable representations in neural networks
David Klindt, Charles O’Neill, Patrik Reizinger, Harald Maurer, Nina Miolane · 2025
Read From superposition to sparse codes: interpretable representations in neural networks by David Klindt, Charles O’Neill, Patrik Reizinger, Harald Maurer, Nina Miolane (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01824.txt.
Introduction
David Klindt and four co-authors argue that the tangled internal code of neural networks can be untangled into features people can name, in this 2025 paper. Networks seem to store concepts in superposition, many features sharing the same neurons, overlaid like transparencies. That overlap makes their reasoning hard to inspect, a problem for both artificial intelligence and neuroscience.
The perspective rests on three steps. First, theory shows that networks trained to classify recover the true hidden features of their data, up to a simple linear remixing. Second, sparse coding techniques, borrowed from signal compression, can undo that remix and pull the features apart. Third, the authors propose quantitative scores for how interpretable the recovered features really are, checked against human judgment. The result is the outline of a science of reading networks: measure first, name second, trust last.
Catalog record
| CREATOR | David Klindt, Charles O’Neill, Patrik Reizinger, Harald Maurer, Nina Miolane |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From superposition to sparse codes: interpretable representations in neural networks David Klindt, Charles O’Neill, Patrik Reizinger, Harald Maurer, Nina Miolane Abstract 1 Abstract Understanding how information is represented in neural networks is a fundamental challenge in both neuroscience and artificial intelligence. Despite their nonlinear architectures, recent evidence suggests that neural networks encode features in superposition , meaning that input concepts are linearly overlaid within the network’s representations. We present a perspective that explains this phenomenon and provides a foundation for extracting interpretable representations from neural activations. Our theoretical framework consists of three steps: (1) Identifiability theory shows that neural networks trained for classification recover latent features up to a linear transformation. (2) Sparse coding methods can extract disentangled features from these representations by leveraging principles from compressed sensing. (3) Quantitative interpretability metrics provide a means to assess the success of these methods, ensuring that extracted features align with human-interpretable concepts. By bridging insights from theoretical neuroscience, representation learning, and interpretability research, we propose an emerging perspective on understanding neural representations in both artificial and biological systems. Our arguments have implications for neural coding theories, AI transparency, and the broader goal of making deep learning models more interpretable. Figure 1: Superposition of neural representations. A) Images generated from GPT-4 prompts: “A”: an elephant , “B”: a pink ball , and “A and B”: an elephant and a pink ball ; B) Test for additivity: Adding the neural representations (ViT-B/16 ( Dosovitskiy et al., 2021 ) ) of the first two images ( f\circ g(\text{``A''}) and f\circ g(\text{``B''}) ) and the neural representation for the combined image ( f\circ g(\text{``A and B''}) ) (c.s.: cosine similarity; each dot corresponds to one dimension of the neural representation); C) Real (not AI generated) images of a dog (“Hazel”), a cat (“Nemo”) and both together. D) Same analysis as in B) , but with the images in C) , showing that additivity even holds on natural images. Further examples ( N=10 , not shown) support that this finding is statistically significant (over a calibrated baseline, i.e., the c.s. of the representations of just “A” and “B”) in neural representation space ( f\circ g , p<10^{-5} ) but not in pixel space ( g , p\approx 0.88 ). 2 Main Understanding how information is represented in neural networks is a fundamental challenge in both neuroscience and artificial intelligence (AI) ( von Helmholtz, 1878 ; McCulloch and Pitts, 1943 ; Kleene, 1951 ; Rosenblatt, 1958 ; Smolensky, 1990 ) . As AI systems—particularly large language models (LLMs)—become integral to critical domains like healthcare, finance, and security, the need for transparency and interpretability is more urgent than ever ( Linardatos et al., 2020 ; Fan et al., 2021 ; Gilpin et al., 2018 ; Leavitt and Morcos, 2020 ) . Neural networks, both biological and artificial, encode information in distributed representations , where patterns of activity across many units encode complex concepts ( Rumelhart et al., 1986 ) . In human cognition, symbolic knowledge emerges from this distributed processing ( Fodor, 1975 ; Chomsky, 2014 ; Tenenbaum et al., 2011 ) . Similarly, deep learning models generate high-dimensional representations of input data, whether in vision, language, or multimodal systems. But even prior, from classical methods such as Latent Semantic Analysis (LSA) to more recent word2vec ( Mikolov et al., 2013 ) and GloVe ( Pennington et al., 2014 ) , all produce distributed word embeddings by capturing semantic relationships through vector operations. This raises a fundamental question: How can we relate distributed neural representations back to human-interpretable concepts? Figure 2: Theory and analysis pipeline. Data ( x ) arise from interpretable features ( z\in\mathbb{R}^{N} ) through a nonlinear function ( g ) and neural representations ( y\in\mathbb{R}^{N} ) arise from data through another nonlinear function ( f ). However, because neural representations have lower dimensionality ( M<N ), they overlay interpretable features in superposition Elhage et al. (2022) . 1) Identifiability theory establishes that the overall mapping from interpretable features to neural representations must be linear. 2) Compressed sensing theory shows that sparse coding can lift sparse features out of superposition, recovering the original interpretable features up to permutations. 3) Since the true interpretable features are unknown, we evaluate the success of sparse coding using a permutation-invariant measure of interpretability as a proxy. The superposition principle. A key hypothesis is that meaningful concepts are encoded in superposition —where multiple features are linearly combined in a shared representational space ( Van Gelder, 1990b ; Arora et al., 2018 ; Elhage et al., 2022 ) . This implies that the representation of a combined concept is well approximated by the sum of the representations of its parts. For instance, as shown in Fig. 1 , the sum of neural representations for an elephant and a pink ball approximates the representation of an image containing both. At first glance, this is surprising: a highly nonlinear neural network (generative model) transforms inputs into images and then another highly nonlinear neural network (image embedding) transforms inputs into neural representations, yet together, these transformations result in an additive representation . Historically, research on neural representations has distinguished between local coding (a single neuron encodes a concept) and distributed coding (a concept is represented by patterns across multiple neurons) ( Rumelhart et al., 1986 ; Thorpe, 1989 ; Quian et al., 2005 ) . Smolensky (1990) formally defined the superposition principle : Definition 1 (Smolensky superposition) . A ‘connectionist’ [distributed] representation function \Psi employs the superpositional representation of conjunction ( \wedge ) if and only if: \Psi\left(\underset{i}{\bigwedge}\,p_{i}\right)=\sum_{i}\Psi(p_{i}). (1) where \bigwedge_{i}p_{i} denotes the conjunction of propositions p_{i} (e.g., \text{``A''}\wedge\text{``B''}=\text{``A and B''} in Fig. 1 ). This implies that the representation of a composite concept is the sum of its individual representations (e.g., \Psi:=f\circ g in Fig. 1 ). This additivity property ( 5 ) suggests that neural networks encode features linearly in high-dimensional spaces . The problem is that such superposed representations are difficult to interpret directly. Sparse coding offers a potential solution: by transforming neural activations into sparse codes, we may uncover representations that better align with human-interpretable concepts ( Olshausen and Field, 1996 ) . In this perspective, we outline a three-step approach for extracting interpretable codes from distributed neural representations: Theory and analysis pipeline (see also Fig. 2 ). 1. Identifiability theory: Neural networks trained on structured data (provably) learn to represent latent variables up to a linear transformation . 2. Sparse coding: Compressed sensing theory guarantees under what conditions we can extract (interpretable) sparse codes from superposition . 3. Quantitative interpretability: Since true latent variables are unknown, we need interpretability metrics as proxies to evaluate extracted sparse codes . 3 Interpretable sparse codes from distributed representations “It is a capital mistake to theorize before one has data.” Sherlock Holmes We now turn to prior research, all of which can be understood from the perspective of our three-step approach: i) assume linear representations, ii) apply sparse coding, and iii) assess interpretability (qualitative and/or quantitative). Table 1 summarizes key (non-exhaustive) contributions in this area, spanning early matrix factorization techniques to modern deep learning methods. Paper Model Methods Finding ( Ritter and Kohonen, 1989 ) SVD SOM Qualitative interpretability ( Honkela et al., 2010 )