MoE-GPS: G uidlines for P rediction S trategy for Dynamic Expert Duplication in MoE Load Balancing Haiyue Ma, Zhixu Du, Yiran Chen Abstract In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Recent works improve MoE inference load balance by dynamically duplicating popular experts to more GPUs to process excessive tokens, which requires predicting the distribution before routing. In this paper, we discuss the tradeoff of prediction strategies, accuracies, overhead, and end-to-end system performance. We propose MoE-GPS , a framework that guides the selection of the optimal predictor design under various system configurations, by quantifying the performance impact to system-level model runtime. Specifically, we advocate for Distribution-Only Prediction , a prediction strategy that only predicts overall token distribution which significantly reduces overhead compared to the traditional Token-to-Expert Prediction. On Mixtral 8×7B MMLU dataset, MoE-GPS suggests Distribution-Only Prediction which improves end-to-end inference performance by more than 23 % 23\% compared with Token-to-Expert Prediction. 1 Introduction Figure 1 : MoE-GPS guidelines for selecting optimal expert prediction strategies that minimizes end-to-end inference latency based on model and hardware characteristics. Mixture-of-Experts (MoE) jacobs1991adaptive ; jordan1994hierarchical ; fedus2022switch ; shazeer2017 models reduce the computation of Large Language Models (LLMs) by activating only a subset of experts per token. In large multi-GPU datacenters, Expert Parallelism (EP) lepikhin2020gshard is typically used for Feed Forward Network (FFN) layers, where each GPU only hosts one or few experts. However, due to skewed token-to-expert distribution rajbhandari2022deepspeed ; cong2024prediction , this approach often results in load imbalance, underutilized resources, and increased end-to-end latency. The growing number of experts in modern LLM models exaggerates the imbalance liu2024deepseek . While both training and inference can have imbalance, existing works proactively balance token distribution in training, such as using auxiliary loss wang2024auxiliary which can achieve near-perfect balance. For inference, expert-to-token mappings are fixed and imbalance is unavoidable. A popular approach to mitigate load imbalance is expert duplication cong2024prediction ; wang2023prophet ; he2022fastermoe ; nie2023flexmoe , where heavily used experts are replicated across GPUs to distribute tokens more evenly. Since the token distribution changes over time, dynamic duplication is often needed, which requires a predictor for the distribution ahead of the routing stage. Higher prediction accuracy improves load balancing but also incurs greater overhead, creating a complex tradeoff between predictor complexity, accuracy, and system performance. Moreover, the effectiveness of each prediction strategy depends on multiple factors, such as workload characteristics, hardware communication schemes, and the degree of token imbalance. Despite its importance, there is currently no systematic method to model the runtime implications of MoE load imbalance, and to help choose the best predictor for different workload and system setups. We present MoE-GPS , a framework that simulates end-to-end MoE inference performance with imbalance, and guides the selection of expert prediction strategies that yield the shortest runtime. Built on top of an architectural simulator, LLMCompass zhang2024llmcompass , MoE-GPS models the runtime tradeoffs among prediction strategies, accuracy, and overhead. Given an arbitrary model architecture and hardware setup, MoE-GPS identifies the strategy that delivers the best system performance. With insights from MoE-GPS, we advocate for Distribution-Only Prediction strategy, which only predicts the coarse-grained token distribution across experts instead of exact token-to-expert mappings (Token-to-Expert Prediction). This lightweight approach is particularly effective when communication is not a bottleneck, because it reduces prediction complexity and still improves compute load balancing. Exact token-level prediction optimizes both computation and communication at the cost of higher overhead, which becomes more favorable when communication cost dominates. In addition, we observe that Distribution-Only Prediction performs better with more balanced workloads. These findings are summarized as design guidelines in Figure 1 and further elaborated in Section 4 and 5 . Our contribution can be summarized as follows: • We propose MoE-GPS , a system performance simulation framework that selects the optimal expert prediction strategy to minimize end-to-end inference latency, given a model and hardware setup. • We identify and validate the effectiveness of Distribution-Only Prediction as a lightweight alternative to Token-to-Expert Prediction, offering better scalability and efficiency under varying system bottlenecks. For example, Distribution-Only Prediction introduces more than 23\% system-level performance improvements on Mixtral 8×7B MMLU dataset compared to Token-to-Expert Prediction. Our work provides a practical tool and actionable insights for designing MoE predictors that improve inference speed across various workloads and hardware configurations. 2 Background Figure 2 : Overview of a typical Mixture-of-Expert inference based on a Transformer network, with four-way Tensor Parallelism and Expert Parallelism. Four Experts (denoted E1-E4) each resides in one GPU. Routing assigns each token to its expert, which creates load imbalance for both compute and communication, if the token-to-expert distribution is skewed. Load Imbalance in Inference Prefill Is Critical. Imbalance can occur in both training and inference in a MoE network. Current approaches mitigates training imbalance by proactively routing tokens to less popular experts wang2024auxiliary . In inference, token-to-expert mapping is fixed. Discarding or re-routing excessive tokens on popular experts may improve balance at the cost of model accuracy. In this paper, we focus on inference prefill , and do not make changes to the original token-to-expert mapping. The decode stage has much less tokens being processed at the same time and is latency-critical, and load balancing is a secondary issue. Figure 2 shows the overall flow of a Transformer Block in a Mixture-of-Expert inference workload. For illustration purposes, we assume a four-GPU system, four experts (denoted E1-E4) each residing on one GPU, and each token is routed to its top-1 experts fedus2022switch . The general concepts apply to arbitrary number of devices, experts, and tokens with top-K experts jiang2024mixtral ; zhu2024llama . Expert Parallelism Leads to Imbalance. The parallelism scheme we use is Tensor Parallelism (TP) for the Attention layers and Expert Parallelism (EP) for the FFN layers shoeybi2019megatron . Using TP for the Attention layers excludes the need to duplicate KV cache for adjacent tokens when the sequence is split to different GPUs. We used Ring All Reduce ring_all_reduce to optimize the communication runtime after TP. Using EP for the FFN layers has two benefits over TP: first, it reduces communication latency by the factor of number of devices; second, it avoids splitting the weights in each expert to narrower matrices, which can lower Tensor Core utilization and increase compute time. The introduction of EP assigns experts to certain GPUs and creates load imbalance. Industry practices suggest more advanced hybrid parallelism (TP+EP) moe_package when necessary. For simplicity, this work assumes TP-only Attention and EP-only FFN, and the insights are generalizable. Quantifying Imbalance. We assume that Expert 1 is the most popular expert which takes 75% of all tokens as shown in Figure 2 . FFN computation and communication runtime will be dominated by GPU 1 which has Expert 1. To quantify the load imbalance of each GPU, we introduced a parameter called " skewness ", the number of tokens in the most popular expert divided by the average number of tokens per expert if the workload is evenly distributed: \text{skewness}=\frac{\#\text{ of tokens in the most popular expert}}{\#\text{ of average tokens per experts}} Figure 2 ’s workload skewness is 3 since Expert 1 has 75% of the tokens but the average token per expert is 25%. Performance Impacts of Load Imbalance. Skewness only impacts the runtime of FFN compute and communication since tokens are routed to the experts for FFN. FFN computation typically fully utilizes GPU resources, and the bottleneck FFN runtime is increased by a factor of the skewness. Routing incurs inter-device communication. With a fully connected multi-GPU topology, for a perfectly balanced distribution (skewness = 1) each GPU needs to move (N-1)/N of its existing tokens to other GPUs (N = number of GPUs), assuming the token placement is completely random after scatter. This yields (N-1)/N^{2} tokens to move per GPU. However, for a workload with skewed distribution, the GPU with the most tokens will be receiving more tokens than other GPUs, and thus has longer communication latency. The overall communication time, which is bottlenecked by the GPU with the most popular expert, is scaled by skewness: (N-1)\cdot\text{skewness}/N^{2} . The same amount of communication happens for All-to-All shuffle after the FFN layers. Current Solutions. In a multi-GPU datacenter setting, many existing solutions have proposed expert duplication - mirroring popular experts in other GPUs - to offload excessive token processing. MoE-Prediction cong2024prediction observes highly skewed token-to-expert distribution and proposes predicting such distribution to guide expert placement. Prophet wang2023prophet , FlexMoE nie2023flexmoe , SE-MoE shen2022se , and FasterMoE he2022fastermoe ; he2021fastmoe propose specific strategies for dynamic expert duplication. Other techniques focus on reducing the distribution imbalance in training by imposing constraints, such as auxiliary load balancing wang2024auxiliary and expert biasing liu2024deepseek . In a different setting with fewer GPUs and the GPU memory capacity is limited for storing experts, token-to-expert mappings are also predicted to facilitate expert offloading to CPU du2024sida . Others proposes different solutions to serve as the token-to-expert mapping, such as expert buffer that stores the hottest experts huang2023towards and expert activation correlation model yi2023edgemoe . These solutions are orthogonal to the discussion of this paper. 3 Methodology This section discusses our approach to modeling and analyzing expert prediction strategies in MoE inference. We describe the integration of expert duplication into the MoE model architecture, and the two different prediction strategies (Distribution-Only and Token-to-Expert). We also show normalized system performance obtained from a performance simulator, LLMCompass zhang2024llmcompass , to illustrate high-level trends of the runtime implications of expert duplication. We defer detailed analyses of LLMCompass’s simluated performance to Section 4 . Figure 3 : Integration of expert duplication and different prediction strategies into the MoE model architecture for a single layer. Token-to-Expert Prediction eliminates communication by directly routing tokens to their predicted GPUs, while Distribution-Only Prediction assumes random token distribution during scattering. 3.1 Approach to Expert Duplication We integrate dynamic expert duplication into the MoE model by inserting a pre-trained predictor before Attention in each layer. For a chosen frequency, the predictor does inference with current batched inputs to predict distribution and guide the expert placement. Since dynamic duplication incurs extra inter-GPU communication for moving experts, existing works propose different intervals for prediction and moving, from every single batch he2021fastmoe ; wang2023prophet to every 10 minutes liu2024deepseek , to balance overhead and effectiveness. In this study, we assume single-batch prediction and placement frequency. We show calculation in Section 5 that the placement latency can be hidden within the Attention layers which is compute-bound with moderate batch sizes. Our simulator can be configured to model different frequencies of prediction and placement, by averaging out the overhead to multiple batches. Figure 3 illustrates how expert duplication helps balance token workloads. In our example, Expert 1 has the most tokens, so it is replicated across multiple GPUs to evenly distribute the load. In general, given arbitrary token-to-expert mappings, experts can be duplicated to achieve per-GPU balance by iteratively shifting experts from overloaded to underloaded devices: keep duplicating the experts on GPUs with >1/N tokens to GPUs with <1/N tokens until all GPUs process the same amount of tokens. A detailed algorithm can be found in Algorithm 1 . Input: Token–expert map f:\{1,\dots,T\}\!\to\!\{1,\dots,E\} ; Per-GPU memory capacities M\in\mathbb{N}^{G} ; Initial placement \mathcal{P}\subseteq\{1,\dots,E\}\!\times\!\{1,\dots,G\} ; Maximum copies per expert C_{\max} . Output: Balanced placement \mathcal{P} and dispatch d:\{1,\dots,T\}\!\to\!\{1,\dots,G\} . Assign each token t to any GPU g that hosts its expert: d(t)\leftarrow\min\{g\mid(f(t),g)\in\mathcal{P}\} ; 1 L_{g}\leftarrow|\{t\mid d(t)=g\}| for all g\in\{1,\dots,G\} ; 2 while \max_{g}L_{g}-\min_{g}L_{g}>1 do 3 g_{h}\leftarrow\arg\max_{g}L_{g} ; g_{c}\leftarrow\arg\min_{g}L_{g} ; 4 \Delta\leftarrow\left\lceil\frac{L_{g_{h}}-L_{g_{c}}}{2}\right\rceil ; 5 e^{\star}\leftarrow\arg\max_{e\in\mathcal{E}(g_{h})}|\{t\mid d(t)=g_{h}\land f(t)=e\}| ; 6 if (e^{\star},g_{c})\notin\mathcal{P} and copies (e^{\star})