The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models

Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu · 2026

Read The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models by Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03425.txt.

Description

Yan Wang and five co-authors question a common belief about Mixture of Experts models, which route each input to a small set of expert networks. People often assume those experts specialize by topic. The team built COMMITTEEAUDIT, a tool that studies routing at the level of expert groups rather than single experts.

Across three models and the MMLU test set, they found a small Standing Committee of experts that takes most routing traffic in every domain. Those core experts seem to anchor reasoning and syntax, while outer experts hold topic knowledge. The result suggests these models centralize far more than expected. Training rules that force even expert use may fight the model's natural path and waste effort.

Catalog record

CREATORYan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03425

Browse the full library · About the Tower of Babel