Maximum Score Routing For Mixture-of-Experts
Bowen Dong 1 , Yilong Fan 2 , Yutao Sun 1 , Zhenyu Li 1 , Tengyu Pan 1 , Xun Zhou 3 † \dagger , Jianyong Wang 1 † \dagger 1 Tsinghua University, 2 Tianjin University, 3 Seed-Foundation-Model Team, ByteDance [email protected] · 2025
Read Maximum Score Routing For Mixture-of-Experts by Bowen Dong 1 , Yilong Fan 2 , Yutao Sun 1 , Zhenyu Li 1 , Tengyu Pan 1 , Xun Zhou 3 † \dagger , Jianyong Wang 1 † \dagger 1 Tsinghua University, 2 Tianjin University, 3 Seed-Foundation-Model Team, ByteDance [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12801.txt.
Introduction
Bowen Dong and colleagues propose Maximum Score Routing, a new way to route tokens to experts inside mixture-of-experts language models. Such models send each input token to a few experts. That scales capacity while keeping speed. Old routing sets a cap on experts for hardware-friendly work. But full experts drop tokens, and little-used experts waste work on padding.
To drop the cap instead breaks load balance and speed. MaxScore models routing as a minimum-cost maximum-flow problem. It also adds a SoftTopk operator. The team reports that it fixes core limits of past rerouting and transport forms. At the same compute cost, it gave lower training losses and higher test scores than both capped and uncapped bases. Details and code are public for all.
Catalog record
| CREATOR | Bowen Dong 1 , Yilong Fan 2 , Yutao Sun 1 , Zhenyu Li 1 , Tengyu Pan 1 , Xun Zhou 3 † \dagger , Jianyong Wang 1 † \dagger 1 Tsinghua University, 2 Tianjin University, 3 Seed-Foundation-Model Team, ByteDance [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Maximum Score Routing For Mixture-of-Experts Bowen Dong 1 , Yilong Fan 2 , Yutao Sun 1 , Zhenyu Li 1 , Tengyu Pan 1 , Xun Zhou 3 † \dagger , Jianyong Wang 1 † \dagger 1 Tsinghua University, 2 Tianjin University, 3 Seed-Foundation-Model Team, ByteDance [email protected] Abstract Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable model capacity while preserving computational efficiency. Traditional MoE networks impose an expert capacity constraint to ensure GPU-friendly computation. However, this leads to token dropping when capacity is saturated and results in low hardware efficiency due to padding in underutilized experts. Removing the capacity constraint, in turn, compromises load balancing and computational efficiency. To address these issues, we propose Maximum Score Routing ( MaxScore ), a novel MoE routing paradigm that models routing as a minimum-cost maximum-flow problem and integrates a SoftTopk \mathrm{SoftTopk} operator. MaxScore resolves the fundamental limitations of iterative rerouting and optimal transport formulations, achieving lower training losses and higher evaluation scores at equivalent FLOPs compared to both constrained and unconstrained baselines. Implementation details and experimental configurations can be obtained from https://github.com/dongbw18/MaxScore.git . † † footnotetext: † indicates corresponding authors. 1 INTRODUCTION The Mixture of Experts (MoE) paradigm has emerged as a compelling architectural strategy for scaling neural networks while maintaining computational efficiency. This approach dynamically combines multiple subsets of parameters (experts) by a learnable routing network, aiming to improve model capacity and computational efficiency. The routing network of sparsely activated MoE Shazeer et al. 2017 dynamically allocates input tokens to top-k experts through differentiable sparse transformations, enabling conditional computation that scales model parameters without proportionally increasing FLOPs. Softmax is conventionally employed to compute token-expert affinity coefficients in MoE routing networks, which promotes inter-expert competition. To mitigate winner-takes-all and preserve load balance, both hard constraints using expert capacity Eigen et al. 2014 , and soft constraints using auxiliary losses Bengio et al. 2016 , are incorporated into the routing network Shazeer et al. 2017 . GShard Lepikhin et al. 2020 pioneers the integration of MoE with Transformer architectures Vaswani et al. 2017 , where expert capacity constraints enable GPU-friendly computation patterns. ExpertChoice Zhou et al. 2022 directly enables experts to select tokens based on capacity constraints. However, token dropping occurs when inputs are routed to capacity-saturated experts, while padding operations in underutilized experts create hardware inefficiencies. Empirical analysis reveals that approaches such as expanding capacity Hwang et al. 2023 or removing capacity constraints altogether Gale et al. 2022 ; Muennighoff et al. 2024 effectively eliminate token dropping, but inevitably introduce a trade-off between computational efficiency and load balancing performance. Efforts to prevent token dropping via refined routing strategies Fedus et al. 2022 ; Clark et al. 2022 have not yielded performance improvements, highlighting unresolved challenges in dynamic resource allocation. This work introduces Maximum Score Routing (MaxScore) , a novel MoE routing paradigm that formulates token-expert routing as a minimum-cost maximum-flow problem Waissi 1994 , integrated with a \mathrm{SoftTopk} operator. To the best of our knowledge, this is the first successful integration of network flow modeling and \mathrm{SoftTopk} in MoE routing. MaxScore preserves GPU-compatible expert capacity constraints and achieves better load balancing. Under the same FLOPs, MaxScore exhibits lower training loss and higher evaluation scores compared to both constrained and unconstrained baselines. Ablation studies demonstrate the necessity of both network flow modeling and the \mathrm{SoftTopk} operator, revealing fundamental limitations in the iterative rerouting mechanism of Fedus et al. 2022 and the optimal transport-based routing of Clark et al. 2022 . The synergistic combination of two methodological enhancements yields superadditive performance gains, with empirical results demonstrating that their integrated efficacy surpasses the linear summation of individual improvements. Scaling experiments show that MaxScore delivers consistent performance improvements with larger activated parameter budgets, and achieves more gains when increasing the number of experts, compared with standard MoE approaches. 2 PRELIMINARIES 2.1 Top-k Sparsely Activated MoE The top-k routing mechanism is a cornerstone of sparsely activated MoE architectures, enabling efficient scaling of model capacity while maintaining computational tractability. Originally popularized in language modeling Shazeer et al. 2017 , this paradigm dynamically routes each input token to a subset of k expert networks (where k\ll e , for e total experts). Unlike dense models that activate all parameters per input, top-k routing induces conditional computation by selecting experts based on learned gating scores, typically computed via softmax over a trainable projection of input embeddings Lepikhin et al. 2020 . For a given input x , the output y of the MoE module can be written as follows: y=\sum_{i=1}^{E}R(x)_{i}\cdot E_{i}(x),\\ (1) R(x)=\mathrm{KeepTopk}(\mathrm{Softmax}(x\cdot W_{g})),\\ (2) where R(x) is the sparsely activated routing function, \mathrm{KeepTopk}(\cdot) retains the top-k largest values while setting others to zero, W_{g} is the weight matrix of the routing function, E_{i}(x) is the output of the i -th expert network and the computation is performed only when R(x)_{i}>0 . By leveraging sparse activation, MoE decouples total capacity \mathcal{O}(e) from per-step computational cost, activating only \mathcal{O}(k) parameters during both training and inference. 2.2 Operators in Top-k MoE Routing Routings in MoE commonly use \mathrm{Softmax}(\cdot) to calculate the token-expert affinity coefficients, which encourages competition between experts. However, \mathrm{Softmax}(\cdot) serves as a smooth approximation to the one-hot \mathrm{Argmax}(\cdot) function, which can lead to inefficiencies in top- k routing, as the top-1 expert often receives a disproportionately large affinity score compared to the remaining k{-}1 experts. Alternative routing operators have also been investigated. DeepSeek-AI et al. 2024b replaces \mathrm{Softmax}(\cdot) with \mathrm{Sigmoid}(\cdot) to align with its auxiliary-loss-free load balancing strategy, while ReMoE Wang et al. 2025 explores the feasibility of using \mathrm{ReLU}(\cdot) for routing decisions. We define \mathrm{SoftTopk}(\cdot) as a smooth approximation to \mathrm{ArgTopk}(\cdot) , which represents the top-k selection in a one-hot form, formally given by: \mathrm{ArgTopk}(\mathbf{a})_{i}=\begin{cases}1,a_{i}\in\mathrm{Topk}(\mathbf{a})\\ 0,\mathrm{otherwise},\\ \end{cases} (3) where \mathbf{a}=(a_{1},a_{2},...,a_{e}) represents the affinity coefficients between the token and e experts. Martins and Astudillo 2016 and Peters et al. 2019 proposed \mathrm{Sparsemax}(\cdot) and \mathrm{Entmax}(\cdot) as differentiable approximations for top-k probability truncation. Su 2024 further introduced a broader family of \mathrm{SoftTopk}(\cdot) operators. However, their integration into MoE routing has not been investigated, leaving a promising direction underexplored. Figure 1: Different top-2 routing paradigms for 3 experts and 6 tokens. (a) sets capacity-factor c_{f}=1.0 , and token dropping occurs; (b) sets capacity-factor c_{f}=1.5 , there is no more token dropping, but more computation is wasted; (c) uses iterative rerouting mechanism, the dropped token is reassigned to expert with remaining capacity. 2.3 Expert Capacity Constrained To counteract the winner-takes-all phenomenon and maintain load balancing in the routing network, traditional routing architectures integrate dual constraint mechanisms: (i) hard limits through expert capacity Eigen et al. 2014 , and (ii) soft regularization via differentiable auxiliary losses Bengio et al. 2016 ; Shazeer et al. 2017 ; Zoph et al. 2022 . GShard Lepikhin et al. 2020 strategically harmonizes capacity-constrained MoE design with Transformer architectures Vaswani et al. 2017 . For a batch of n tokens, GShard fixes per-expert capacity with c=\frac{k*n}{e} to enable parallel-friendly computation patterns. This routing mechanism, however, poses optimization challenges due to imbalanced expert utilization. While underloaded experts incur computational overhead through padding (mathematically sound but hardware-inefficient), overloaded experts lead to token dropping. Increasing expert capacity c^{\prime}=c_{f}*\frac{k*n}{e} by a capacity-factor c_{f}