Rethinking Sparse Mixture of Experts from a Unified Perspective

Giang Do, Hung Le, Truyen Tran · 2025

Read Rethinking Sparse Mixture of Experts from a Unified Perspective by Giang Do, Hung Le, Truyen Tran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22996.txt.

Introduction

Giang Do, Hung Le, and Truyen Tran rethink sparse mixture-of-experts models, a design that lets large neural networks grow in size without a matching rise in computing cost. Each input passes through only a few expert sub-networks. Current designs fix the budget in advance: either each token picks a set number of experts, or each expert takes a set number of tokens. Fixed budgets force poor matches and miss good ones.

The authors view the whole routing problem through linear programming, which gives one general frame for both design families. On that base they build a unified model with a single routing method and a single score for matches. Tests across clean and corrupted data, text and vision tasks, and both trained and training-free setups show clear gains over older methods. The design also allows flexible budgets, which cuts running costs while keeping quality high.

Catalog record

CREATORGiang Do, Hung Le, Truyen Tran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22996

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Sparse Mixture of Experts from a Unified Perspective Giang Do, Hung Le, Truyen Tran Abstract Sparse Mixture of Experts (SMoE) models scale the capacity of models while maintaining constant computational overhead. SMoE methods fall into two categories: Token Choice , which routes each token to a fixed number of experts, and Expert Choice , which assigns a fixed number of tokens to each expert. However, the use of fixed budgets for tokens or experts causes both approaches to select irrelevant token-expert pairs or overlook critical assignments, which degrades overall performance. To fill that gap, we rethink SMoE from a unified perspective through the lens of linear programming , which provides a general formulation for SMoE models. Furthermore, we introduce Unified Sparse Mixture of Experts (USMoE) , a novel framework comprising a unified mechanism and a unified score to overcome these limitations. We provide both theoretical justification and empirical evidence demonstrating USMoE’s effectiveness. Extensive evaluations across diverse data settings (clean and corrupted), multiple domains (including texts and vision tasks), and different learning approaches (training-free and training-based) show that USMoE not only delivers significant performance improvements over existing SMoE methods, but also enables more flexible expert selection budgets, reducing inference costs without compromising model performance. Our implementation is publicly available at https://github.com/giangdip2410/USMoE . 1 Introduction Sparse Mixture of Experts (SMoE) models have achieved notable success in natural language processing (NLP) and visual representation learning tasks ( Du et al., 2022 ; Fedus et al., 2022 ; Riquelme et al., 2021 ; Shen et al., 2023 ) . These advancements build on the Transformer architecture ( Vaswani et al., 2017 ) and its variants ( Child et al., 2019 ; Dai et al., 2019 ) , which leverage large datasets and significant compute resources. However, training large Transformer models can be prohibitively expensive, requiring extensive compute hours ( Kaddour et al., 2023 ) . To overcome this issue, SMoE models activate only a subset of experts for each input, reducing inference time compared to dense models ( Shazeer et al., 2017 ; Zoph et al., 2022 ; Artetxe et al., 2022 ; Ludziejewski et al., 2024 ) . The SMoE architecture can be categorized into two variants: Token Choice , which assigns experts to each token ( Dai et al., 2024 ; Team, 2024 ; Muennighoff et al., 2025 ; Jiang et al., 2024a ) , and Expert Choice , which assigns tokens to each expert ( Zhou et al., 2022 ) . The advantage of Token Choice lies in its ability to select experts for each token, while Expert Choice ensures a more balanced token distribution across experts. Figure 1 : Illustration of USMoE compared with Qwen3-30B-A3B-Instruct (Original) on the BoolQ dataset. We highlight the key difference between the two approaches: the Original model adopts a Token Choice strategy with a fixed number of experts per token, whereas USMoE dynamically selects the number of experts per token, allowing the model to focus on more relevant tokens and thereby improving performance. Best viewed in color. Despite their promising results, SMoE models have several limitations. The Expert Choice approach suffers from token dropping ( Zhou et al., 2022 ) , while the Token Choice approach struggles with unbalanced expert loading ( Shazeer et al., 2017 ) . The use of fixed budgets for tokens or experts causes both approaches to select irrelevant token–expert pairs or overlook critical assignments. Additionally, projecting high-dimensional representations into a lower-dimensional routing space is prone to representation collapse, where routing decisions become skewed toward a small subset of experts or multiple experts converge to similar representations ( Chi et al., 2022 ; Chen et al., 2022 ) . Recent research has explored improving router policies ( Chi et al., 2022 ; Chen et al., 2023 ; Do et al., 2023 ) to mitigate these issues. However, existing methods face three key challenges: (1) The reliance on auxiliary losses requires careful balancing between the router loss and the task loss, which introduces trade-offs; (2) Token Choice (TC) struggles to handle noisy tokens effectively; and (3) Expert Choice (EC) suffers from information leakage issues, which significantly degrade performance on autoregressive models ( Zhou et al., 2022 ; Wang et al., 2024 ; Raposo et al., 2024 ) . As a result, the question of how to optimally select experts or tokens remains open. In this paper, we revisit SMoEs through a unified perspective to better understand the criteria for expert and token selection. From this perspective, Token Choice selects experts along the expert dimension by assigning each token to its most similar expert, while Expert Choice selects tokens along the token dimension by allowing each expert to choose the most similar tokens. This view highlights a key trade-off: Expert Choice risks dropping important tokens, whereas Token Choice struggles to handle noisy or irrelevant tokens. Furthermore, both approaches suffer from the problem of representation collapse ( Chi et al., 2022 ; Do et al., 2023 ; Pham et al., 2024 ) . Building on this analysis, we propose the Unified Sparse Mixture of Experts (USMoE), a robust and efficient framework consisting of two key components: (1) the Unified Score and (2) the Unified Mechanism. The Unified Score is defined as a linear combination of two mapping functions that transform the dot product similarity between token and expert embeddings into a probability distribution. In parallel, the Unified Mechanism incorporates information from both the token and expert dimensions, as illustrated in Figure 2 . These components enable the SMoE model to dynamically prioritize tokens or experts while ensuring the selection of the most similar token-expert pair, enhancing both robustness and effectiveness. We provide both theoretical justification and empirical evidence demonstrating USMoE’s effectiveness. Extensive evaluations across diverse data settings (clean and corrupted), multiple domains (including texts and vision tasks), and different learning approaches (training-free and training-based) show that USMoE not only delivers significant performance improvements over existing SMoE methods, but also enables more flexible expert selection budgets, reducing inference costs without compromising model performance. Key contributions: (1) We present a unified perspective on Sparse Mixture of Experts (SMoE) that fomulates conventional routers as a linear programming problem, and propose USMoE , a robust and efficient framework built upon a Unified Score and a Unified Mechanism , offering improved efficiency, robustness, and flexibility over existing approaches. (2) We theoretically show that USMoE yields more stable and robust expert selection compared to baseline routing methods. (3) We conduct extensive experiments across large language and vision models, spanning both training-free and training-based settings, to comprehensively evaluate the effectiveness, robustness, and flexibility of USMoE. (4) We provide in-depth analyses that elucidate why USMoE works and demonstrate its improved interpretability and reliability. 2 Related Work Sparse Mixture of Experts (SMoE). Sparse Mixture of Experts (SMoE), an extension of the Mixture of Experts framework ( Jacobs et al., 1991 ; Jordan and Jacobs, 1994 ) , has gained traction with large language models and has since been applied in various domains, including computer vision and speech recognition ( Zhou et al., 2022 ; Riquelme et al., 2021 ) . The SMoE architecture consists of two main variants: Token Choice , where experts are assigned to each token ( Shazeer et al.,

Read the complete text