Sparsity and Superposition in Mixture of Experts

Marmik Chaudhari, Jeremi Nuer, Rome Thorstenson · 2025

Read Sparsity and Superposition in Mixture of Experts by Marmik Chaudhari, Jeremi Nuer, Rome Thorstenson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23671.txt.

Introduction

Marmik Chaudhari and two co-authors ask how Mixture of Experts models organize knowledge inside, and whether the usual story from dense networks still applies. In dense models, superposition — storing more features than there are dimensions — is driven by how sparse and how important features are, with sharp phase changes. Mixture of Experts models, which route each input to a few specialist subnetworks, might work differently.

The authors find that neither feature sparsity nor importance triggers phase changes here. What matters instead is network sparsity, the share of experts active at once: sparser models show more monosemanticity, meaning single experts carry cleaner, single-meaning features. They propose defining expert specialization by that clean feature representation rather than by load balancing. With the right starting setup, experts settle around coherent feature combinations on their own, hinting that readable models need not cost performance.

Catalog record

CREATORMarmik Chaudhari, Jeremi Nuer, Rome Thorstenson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparsity and Superposition in Mixture of Experts Marmik Chaudhari, Jeremi Nuer, Rome Thorstenson Abstract Mixture of Experts (MoE) models have become central to scaling large language models, yet their mechanistic differences from dense networks remain poorly understood. Previous work has explored how dense models use superposition to represent more features than dimensions, and how superposition is a function of feature sparsity and feature importance. MoE models cannot be explained mechanistically through the same lens. We find that neither feature sparsity nor feature importance cause discontinuous phase changes, and that network sparsity (the ratio of active to total experts) better characterizes MoEs. We develop new metrics for measuring superposition across experts. Our findings demonstrate that models with greater network sparsity exhibit greater monosemanticity . We propose a new definition of expert specialization based on monosemantic feature representation rather than load balancing, showing that experts naturally organize around coherent feature combinations when initialized appropriately. These results suggest that network sparsity in MoEs may enable more interpretable models without sacrificing performance, challenging the common assumption that interpretability and capability are fundamentally at odds. 0 0 footnotetext: Code available at https://github.com/MarmikChaudhari/superposition-moe 1 Introduction Mixture of Experts (MoEs) have become prevalent in state-of-the-art language models, such as Qwen3, Mixtral, and Gemini ( Yang et al. 2025a ; Jiang et al. 2024 ; Google DeepMind 2025 ) , primarily for their computational efficiency and performance gains ( Shazeer et al. 2017 ; Fedus et al. 2022 ) . Subsequent work improve routing (e.g., Expert-Choice routing) and training stability/transfer (e.g., ST-MoE) ( Zhou et al. 2022 ; Zoph et al. 2022 ) . Theoretical and empirical results further show that learnable routers can discover latent cluster structure in data, providing insight for why experts specialize ( Dikkala et al. 2023 ) . However, despite their widespread adoption, MoEs remain poorly understood from a mechanistic interpretability perspective. Interpretability-oriented approaches have sought to make expert behavior more transparent. Yang et al. 2025b proposes MoE-X , which encourages sparsity-aware routing and uses wide, ReLU-based experts to reduce polysemanticity. Park et al. 2025 introduce Monet , scaling the number of experts to enable capability editing via expert activation. Yet these works largely focus on architectural changes; we still lack a mechanistic understanding of how MoEs represent features, how experts affect superposition, and whether experts naturally specialize without extra regularization. Mu & Lin 2025 survey MoE research and identify mechanistic interpretability as a key open challenge. A fundamental challenge in interpreting neural networks is the phenomenon of superposition: when models represent more features than they have dimensions. This allows networks to pack many sparse features into fewer neurons at the cost of making individual neurons polysemantic and difficult to interpret. MoE architectures introduce a new dimension to this problem: network sparsity. Unlike dense models that activate all neurons regardless of input, MoEs activate a fraction of their total parameters ( Shazeer et al. 2017 ) . While dense models exploit feature sparsity by packing many sparse features into shared neurons, MoEs can afford to be more selective, potentially dedicating entire experts to specific feature combinations. We investigate whether (1) MoEs exhibit less superposition than their dense counterparts, (2) there is a discrete phase change in MoE experts as seen in dense models, and (3) we can understand expert specialization through the lens of feature representation rather than just load balancing. We explore these questions using simple models that extend Elhage et al. 2022 ’s framework to MoEs. Our key contributions are as follows: (1) unlike dense models, MoEs do not exhibit sharp phase changes, instead showing more continuous transitions as network sparsity increases; and (2) MoEs consistently exhibit greater monosemanticity (less superposition) than dense models with equivalent active and total parameters, with individual experts representing features more cleanly; (3) we propose an interpretability-focused definition of expert specialization based on monosemantic feature representation, showing that experts naturally organize around coherent feature combinations rather than arbitrary load balancing. 2 Background Learned representations of meaningful ‘features’—loosely, those ideas that have human-interpretable meaning—are often assumed to be linear in terms of activations ( Gorton & Lewis 2025 ) . Thus, measuring activations directly corresponds to the intensity of meaningful features. The superposition hypothesis contends that models are capable of representing far more features than dimensions ( Elhage et al. 2022 ) . In order to have many more features than dimensions in a latent space, features vectors must be packed such that they are not all orthogonal. The Johnson–Lindenstrauss lemma proves that this can be done efficiently by allowing for small amounts of interference. This is an acceptable compromise because most features are extremely sparse (i.e., they are active on only a tiny fraction of inputs). For example, in the complete corpus of all language text, few sentences have to do with the feature of ‘Martin Luther King, Jr.’ Nonlinearities and bias terms allow models to account for moderate interference. Monosemantic features are defined as those that are well-aligned with individual neurons: a single neuron’s activation cleanly reflects the presence or absence of the feature. Superposition, by contrast, refers to representations where multiple features are represented within the same set of neurons, such that no single neuron corresponds to a single meaningful feature. In this setting, neurons and features are polysemantic: a neuron’s activation reflects a linear combination of many features, and a feature is distributed across many neurons. While superposition is efficient for capacity and generalization, it makes interpretation challenging, since observations and interventions on single neurons no longer correspond cleanly to changes in single features. 3 Demonstrating Superposition An intuitive way to think about MoEs is as composition of dense (feed forward) models, where each expert behaves like a dense model in isolation. But it is unclear how to quantify or justify this claim in terms of the mechanistic or representational similarity. We investigated this question by exploring how models vary their representations in terms of superposition. To test whether MoEs exhibit more or less superposition than the dense models, we extended the standard toy models of Elhage et al. 2022 to include MoE variants, which allows us to exactly measure superposition. It also allows us to characterize superposition as a function of feature density (1 - sparsity) and the ratio of active experts ( k ) to the total number of experts in a MoE (E) , i.e. network sparsity . 3.1 Toy Models Our MoE model consists of E experts, each with m hidden dimensions. The input features, n , are routed to the top- k experts as per the learned router g(x)=\mathrm{softmax}(W_{r}x) , where W_{r}\in\mathbb{R}^{n\times E} . Each expert e processes the input through h^{e}=W^{e}x , followed by the reconstruction x^{\prime e}=\mathrm{ReLU}\!\left((W^{e})^{\top}h^{e}+b^{e}\right) . The final output is a weighted combination of the active experts given as x^{\prime}=\sum_{e}w_{e}\,x^{\prime e} where w^{e} are the renormalized gating weights for the top- k selected experts. No load balancing loss is used for this section so that the overall loss simplifies to a reconstruction loss. (a) Norm of each feature’s weight vector \lVert W_{i}\rVert , with colors indicating superposition status (green for features in superposition, purple for monosemantic features). (b) W^{\top}W matrix where each cell represents (\hat{W}_{i}\cdot W_{j}) , revealing interference patterns between features. Figure 1:

Read the complete text