Probing Semantic Routing in Large Mixture-of-Expert Models

Matthew Lyle Olson Neale Ratzlaff Musashi Hinck, Man Luo Sungduk Yu Chendi Xue Vasudev Lal · 2025

Read Probing Semantic Routing in Large Mixture-of-Expert Models by Matthew Lyle Olson Neale Ratzlaff Musashi Hinck, Man Luo Sungduk Yu Chendi Xue Vasudev Lal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10928.txt.

Introduction

Matthew Lyle Olson and co-authors test whether the inner routing of very large language models follows the meaning of the text, in a paper from 2025. These models split work among many expert parts, and a router sends each input to some of them. Gains are often framed as speed, but routing may also sort by topic or sense.

Two controlled tests were run. One held a target word fixed while its sense changed or stayed the same, and the other swapped the word for a close or distant match. Expert overlap differed clearly across these cases, with strong statistical support. The authors conclude that routing in large models of this kind is shaped by meaning, not just by surface form.

Catalog record

CREATORMatthew Lyle Olson Neale Ratzlaff Musashi Hinck, Man Luo Sungduk Yu Chendi Xue Vasudev Lal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing Semantic Routing in Large Mixture-of-Expert Models Matthew Lyle Olson Neale Ratzlaff Musashi Hinck, Man Luo Sungduk Yu Chendi Xue Vasudev Lal Abstract In the past year, large ( > 100 >100 B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of efficiency, prior work has also explored functional differentiation through routing behavior. We investigate whether expert routing in large MoE models is influenced by the semantics of the inputs. To test this, we design two controlled experiments. First, we compare activations on sentence pairs with a shared target word used in the same or different senses. Second, we fix context and substitute the target word with semantically similar or dissimilar alternatives. Comparing expert overlap across these conditions reveals clear, statistically significant evidence of semantic routing in large MoE models. Figure 1: Summary of Experimental Design . We compare expert routing patterns in two controlled experiments. Top: we hold the target word constant, and change the context to either change the meaning of the target word or keep it the same. Bottom: we hold context constant, and substitute the target word for a similar-meaning or different-meaning word. 1 Introduction Since their popularization in Fedus et al. 2022 , the Mixture-of-Experts (MoE) architecture ( Jacobs et al. 1991 ) has been integrated into many frontier large language models (LLMs) ( Lieber et al. 2024 ; Jiang et al. 2024 ; Liu et al. 2024 ; Guo et al. 2025 ; AI 2025 ) . The MoE architecture offers the ability to train far larger models than would normally be possible with dense architectures. Designers can then modulate performance by varying the number of active experts to access a greater portion of the greater model, with the trend being to increase the total and active expert counts. Several prior studies have explored expert activation patterns in MoE models, hypothesizing that each expert may specialize in specific domains, tasks, or topics Zoph et al. 2022 ; Jiang et al. 2024 ; Xue et al. 2024 . While it is intuitive to expect some degree of semantic specialization, previous research has not found clear evidence of routing on the basis of semantics, concluding instead that expert activation is primarily token-dependent rather than being driven by deeper semantic relationships. Given that recent large-scale MoE models have achieved state-of-the-art performance while increasing total expert counts, we investigate whether these models’ expert routing behavior exhibits semantic specialization. We design two controlled experiments. First, we use a word sense disambiguation (WSD) task from the WiC benchmark Pilehvar and Camacho-Collados 2018 , where the same target word appears in two different sentences, either preserving or changing its meaning. This allows us to measure whether expert activation remains stable when the word’s sense is preserved. Second, we study a complementary setting using the lexical substitution benchmark SWORDS Lee et al. 2021 , where we fix the surrounding context but vary the target word, comparing expert overlap between semantically similar and dissimilar word substitutions. We compare the rate of overlap, between models with differing numbers of active and total experts, via a normalized metric based on Cohen’s \kappa that controls for the baseline probability of overlap. We apply these experiments to six MoE models from three model families: DeepSeek-R1 Guo et al. 2025 , DeepSeek-V2-Lite Liu et al. 2024 , Mixtral-8x7B, Mixtral 8x22B Jiang et al. 2024 , Llama-4-Scout and Llama-4-Maverick AI 2025 . For all models, we find that the rate of expert overlap is significantly higher when the meaning of the target word is equal in two sentences than when the meaning of the target word is different. We also find that model scale influences the strength of this specialization: larger models generally exhibit stronger semantic routing signals– with Llama-4 Scout AI 2025 standing out as an exception, showing a pronounced effect despite its smaller total parameter count. Finally, semantic differentiation in expert routing is most prominent in the middle layers, where DeepSeek-R1 exhibits the clearest and most consistent specialization pattern. In summary, our contributions are threefold: (1) We design two complementary semantic probing setups, based on word sense disambiguation and semantic substitution, to systematically assess expert specialization in recent MoE models. (2) We introduce an expert overlap metric to quantify routing similarity and demonstrate its alignment with lexical relationships. (3) We conduct extensive experiments across three MoE model families (DeepSeek, Mixtral, and Llama-4) at various scales, uncovering clear empirical evidence of semantic routing and highlighting its dependence on model size and layer depth. 2 Related Work Current research on expert specialization in MoE models is sparse, yet available studies reveal little evidence of semantic-level differentiation. For example, Xue et al. 2024 tracked token routing patterns across datasets segmented by different topics, languages, and tasks, but failed to find any coherent pattern at such high-level semantics. Rather, they found indications of token-level specialization, mainly concerning low-level semantic features like special characters or auxiliary verbs. Similar findings have been reported in studies using independently developed MoE models ( Zoph et al. 2022 ; Jiang et al. 2024 ; Fan et al. 2024 , e.g.,) . While some neuroscience research has provided evidence that the brain functions like a Mixture of Experts ( Stocco et al. 2010 ; O’Doherty et al. 2021 ) —suggesting the possibility of semantic-level specialization—other studies have shown that MoE models with random routing can perform comparably to those using the more common top-k routing approach ( Roller et al. 2021 ; Zuo et al. 2021 ; Ren et al. 2023 ) . One potential explanation for these mixed results is that prior models (using 8 to 32 experts) might not have been sufficiently expressive to capture fine-grained specialization patterns. The recently-released DeepSeek V3 and Llama 4 Maverick, featuring an extensive network of experts (256 and 128 routed specialists, respectively), provide us with a unique opportunity. Hence, in this study, we test whether a more capable MoE architecture exhibits semantic-level expert specialization. 3 Experiment Settings 3.1 Evaluation Datasets Words-in-Context We leverage polysemy to test for semantic specialization in expert activation patterns. If words that are written the same but have different meanings are routed differently, then this is evidence that routing occurs based on meaning. To test this hypothesis, we use the WiC dataset Pilehvar and Camacho-Collados 2018 (CC BY-NC 4.0), which consists of two types of paired sentences: 1) pairs where a target word has the same sense and 2) pairs where the target word has different senses across sentences. SWORDS We construct a complementary scenario to the WiC experiment, where we test the degree of expert overlap on semantically similar, lexically different input phrases. To do so, we leverage SWORDS Lee et al. 2021 (CC-BY-3.0-US) a lexical substitution benchmark where the corresponding dataset provides semantically annotated sentence pairs with single- and multi-token phrase replacements. We use the SWORDS dataset to construct triples of sentences where a target word is substituted either for a semantically equivalent word or a non-equivalent one. We show examples of both experimental settings in Figure 1 , and an example of such a triplet with target words as follows: Original : "My last show was glorious!" Tasha said. Equivalent

Read the complete text