The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise.

Xi Wang, Soufiane Hayou, Eric Nalisnick · 2026

Read The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise. by Xi Wang, Soufiane Hayou, Eric Nalisnick (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09780.txt.

Introduction

Wang and two co-authors argue that expert specialization in mixture-of-experts emerges from representation space, not routing. Since routers are linear maps, hidden-state similarity explains expert-usage similarity. The team confirms this at token and sequence level across five models. They show a load-balancing loss damps directions shared across hidden states, keeping routing varied, which explains specialization collapse on narrow data. Despite this clean account, specialization patterns resist human interpretation: expert overlap for the same question runs no higher than for different questions, at about 60 percent.

Catalog record

CREATORXi Wang, Soufiane Hayou, Eric Nalisnick
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56643
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09780

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise. Xi Wang, Soufiane Hayou, Eric Nalisnick Abstract Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ( ∼ \sim 60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature. 1 Introduction Mixture of Experts ( 14 ; 8 , MoEs,) have become a standard architecture for large language models (LLMs). Compared with decoder-only LLMs ( 27 ) , MoEs replace the dense feed-forward network (FFN) at each transformer block with a sparse mixture of expert blocks, where each token activates only a sparse subset of FFNs, keeping per-token compute manageable while scaling total model capacity. An expert specialization behavior has been widely observed in pre-trained MoEs: sequences from different domains tend to activate markedly different expert subsets (e.g., Sec. 5 in 21 , Fig. 9 in 18 ). Yet despite its importance as both a signal of meaningful feature learning and a prerequisite for parameter efficiency, its mechanistic origins remain largely unexplained. This paper asks: (1) what mechanism produces expert specialization? and (2) to what extent are the resulting patterns humanly interpretable? The key observation is simple: the MoE router is a linear projection from token hidden states to expert logits, therefore tokens with similar hidden states must activate similar experts. We formalize this as Prop. 1 , a tight data-geometry-aware upper bound on logit distance, verified empirically across five pre-trained MoEs (Fig. 1 ). Specialization is therefore a consequence of how the model organizes its representation space, not a property of the routing mechanism itself. We further show (Prop. 2 ) that when using a load-balancing loss to train MoEs, the router is forced to “suppress” shared directions in correlated hidden states, which might provide a theoretical explanation to the specialization collapse observed by 26 under small-batch training. Despite this clean mechanistic picture, specialization patterns frequently defy human intuition. We demonstrate three cases: (1) different models solving the same math question activate expert patterns as dissimilar as those for entirely different questions ( \sim 60% overlap, Fig. 6 ); (2) semantically different prompts activate nearly identical experts during prefilling, but diverge during generation (Fig. 7 ); (3) in deeper layers, sequences from unrelated domains can collapse to identical expert usage during prefilling (Figs. 9 , 10 ), yet the suppressed experts remain consequential once generation begins. Together, these findings reframe expert specialization as a consequence of the geometry of LLM hidden states, rather than as a standalone interpretable phenomenon. Understanding it is therefore at least as difficult as understanding geometry, which remains an open question even in dense models ( 16 ; 30 ) . It may even be harder, since tokens with different hidden states can still activate similar experts. 2 Related work Identification of domain-specialized experts Several works study expert specialization from an interpretability perspective, aiming to verify its existence and locate domain-specific experts in pre-trained MoEs. 5 introduces a LogitLens-based framework for analyzing MoE specialization patterns, 12 proposes a causal-effect method to identify which experts are domain-specific while 34 propose a CommitteeAudit framework. Overall, these works share similar observations: There exists a compact, domain-invariant expert that captures the majority of routing mass across all domains, with some peripheral experts showing domain sensitivity. Our findings offer a mechanistic explanation for this pattern: shared committee experts likely reflect shared information in the hidden state space, potentially arising from shared syntax structure in the datasets they studied or simply from LLM internal computation. Our work differs from these in two key ways. First, rather than cataloguing which experts are domain-specific, we identify why expert usage may or may not correlate with domain / semantic at all, grounding the phenomenon in hidden state geometry. Second, these works rely on short benchmark inputs (MMLU: < 100 tokens; GSM8K: < 200 tokens), whereas we study much longer sequences, including multi-thousand-token reasoning chains, where the gap between prefilling and generation behavior becomes critical. Applications of expert specialization Beyond interpretability, expert specialization has found practical applications in several domains. In multilingual settings, 6 and 4 leverage routing patterns to identify language-specific experts and improve MoE performance on low-resource languages. From a different angle, recent work in the security community ( 7 ; 23 ) independently arrives at a perspective closely related to ours: because expert usage patterns are a deterministic function of input hidden states, they constitute a side channel that can leak information about user queries, raising significant privacy concerns. Expert specialization insights from pre-trained MoE reports Many open-sourced MoEs’ reports have discussed expert specialization, typically by comparing expert activation patterns across corpora drawn from different domains, e.g., Appendix C in DeepSeek V3 ( 18 ) , Sec 6 in dots.llm1 ( 13 ) , and Figure 22 in OLMoE ( 21 ) . A notable exception is Mistral ( 15 ) , whose experts show little specialization, likely a consequence of sparse upcycling during training. It is worth noting that the goal of monitoring specialization during pre-training is primarily practical rather than interpretive: the goal is to prevent expert homogenization, i.e. the failure mode in which all experts converge to redundant representations, wasting model capacity. DeepSeek V3 ( 18 ) operationalizes this by studying the sensitivity of removing top experts, arguing that MoE with better specialization shows higher sensitivity to expert removal, indicating less redundancy; Layer-wise behavior of hidden states in dense models A key implication of our work is that expert specialization can vary across depth because hidden state geometry evolves across layers. A growing body of work studies this evolution in dense transformers. 16 identifies four functional stages of inference — detokenization, feature engineering, prediction ensembling, and residual sharpening — to explain non-uniform performance degradation under layer removal. 30 shows that intermediate layers consistently produce richer representations than final layers for downstream tasks. 1 further finds that syntactic and semantic information are encoded differently across layers. These results translate directly to expert specialization: what a layer’s hidden states encode determines which experts it activates, explaining why specialization patterns differ between early and late layers. 3 Background: MoE Decoder only LLM For a decoder only LLM with L layers, let \bm{X}^{\ell}=\{\bm{x}_{t}^{\ell}\}_{t=1}^{T}\in\mathbb{R}^{T\times D}

Read the complete text