Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
Shwai He Weilin Cai Jiayi Huang Ang Li · 2025
Read Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts by Shwai He Weilin Cai Jiayi Huang Ang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05066.txt.
Introduction
Shwai He and co-authors speed up mixture-of-experts models by capping what each expert must carry, in this 2025 paper. In these models, idle experts wait on overloaded ones, a straggler effect the authors name and target. Their first fix drops overflow tokens from busy experts, trimming imbalance at a small quality cost.
The second fix lets tokens near quiet experts add those experts as candidates before caps bite, soaking up spare capacity. Tests on language and multimodal models showed large speedups, up to nearly double, with quality intact or slightly improved.
Catalog record
| CREATOR | Shwai He Weilin Cai Jiayi Huang Ang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05066
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts Shwai He Weilin Cai Jiayi Huang Ang Li Abstract The Mixture of Experts (MoE) is an effective architecture for scaling large language models by leveraging sparse expert activation to balance performance and efficiency. However, under expert parallelism, MoE suffers from inference inefficiencies due to imbalanced token-to-expert assignment, where underloaded experts complete computations early but must wait for overloaded experts, leading to global delays. We define this phenomenon as the Straggler Effect , as the most burdened experts dictate the overall inference latency. To address this, we first propose Capacity-Aware Token Drop , which enforces expert capacity limits by discarding excess tokens from overloaded experts, effectively reducing load imbalance with minimal performance impact (e.g., 30 % 30\% speedup with only 0.9 % 0.9\% degradation on OLMoE). Next, given the presence of low-load experts remaining well below the capacity threshold, we introduce Capacity-Aware Expanded Drop , which allows tokens to include additional local experts in their candidate set before enforcing strict local capacity constraints, thereby improving load balance and enhancing the utilization of underused experts. Extensive experiments on both language and multimodal MoE models demonstrate the effectiveness of our approach, yielding substantial gains in expert utilization, model performance, and inference efficiency, e.g., applying Expanded Drop to Mixtral-8 × \times 7B-Instruct yields a 0.2% average performance improvement and a 1.85 × \times inference speedup. 1 Introduction Figure 1 : Illustration of the Straggler Effect in MoE Inference , where the most burdened experts dictate the overall latency. In recent years, the rapid evolution of Large Language Models (LLMs) [ 32 , 38 , 11 ] has driven a wave of innovations, continuously expanding the frontiers of AI research and applications. Among the model architectural innovations, the Mixture of Experts (MoE) framework has emerged as a pivotal technique for optimizing the cost-performance trade-off in LLMs. Specifically, MoE [ 35 , 14 ] enhances scalability by integrating multiple experts while activating only a subset per input. This selective activation substantially improves model performance without a corresponding increase in computational cost, effectively balancing efficiency and performance. Despite the success of MoE, a key efficiency challenge lies in the imbalanced token-to-expert distribution, which results in some experts being overloaded while others remain underutilized [ 26 , 46 ] . In distributed GPU settings, experts are typically sharded across multiple devices, with each GPU responsible for a subset of the experts. Under expert parallelism, low-load experts complete their computations earlier but must wait for overloaded experts to finish, as synchronization barriers are required before proceeding to the next stage. This expert-level straggler effect further propagates to device-level delays, where GPUs hosting lighter expert workloads are stalled by GPUs hosting heavier workloads, leading to inefficient resource utilization and increased end-to-end latency during inference. As illustrated in Figure 1 , this phenomenon is referred to as the Straggler Effect , where the heavily loaded experts determine the overall latency of imbalanced MoE inference. While auxiliary balance losses have been incorporated into the training process to alleviate imbalance [ 35 , 14 , 11 ] , these techniques remain ineffective in mitigating imbalance during inference. Specifically, as shown in Figure 1 , our findings reveal a highly uneven token distribution among experts, with the highest-load expert handling more than seven times the expected average load. Moreover, managing such imbalance during inference often incurs additional resource overhead. For example, DeepSeek-V3 mitigates this issue by duplicating high-load experts and deploying them redundantly across devices [ 11 ] . This motivates us to explore efficient token-to-expert assignment by addressing the key question: How can we prevent extreme overloading of heavily utilized experts? We propose Capacity-Aware Inference to address this challenge. Specifically, for high-load experts, we introduce Capacity-Aware Token Drop , which imposes a maximum capacity constraint and discards excess tokens from overloaded experts. This approach alleviates severe load imbalance and significantly improves efficiency, while maintaining model performance since the dropped tokens represent only a small fraction of the total workload, e.g., OLMoE achieves a 30% speedup in MoE layers with just a 0.9% performance degradation. After removing excess tokens from overloaded experts, we observe that some low-load experts remain significantly underutilized relative to the predefined capacity constraints, yet must still wait for other experts to complete their computations. This leads us to a second key question: How can we effectively leverage the available capacity of underutilized experts? For low-load experts, we extend Token Drop with Capacity-Aware Expanded Drop , which further utilizes the available capacity of underutilized experts to handle overflow tokens from high-load experts. Specifically, under expert parallelism across multiple GPUs, Expanded Drop allows each token to consider additional candidate experts on the same device while still enforcing strict local capacity constraints. This expanded selection improves the utilization of low-load experts and enhances the representational capacity of MoE models under capacity-constrained scenarios. Extensive experimental results validate the effectiveness of our proposed techniques, demonstrating significant improvements in both efficiency and performance, e.g., applying Expanded Drop to Mixtral-8 \times 7B-Instruct yields a 0.2% average performance improvement and a 1.85 \times inference speedup . Moreover, in multimodal models, we identify redundancy among image tokens and show that applying aggressive capacity constraints (e.g., setting the maximum to half of the average expert load) can still maintain performance. In short, our contributions are in four folds: • We identify the Straggler Effect caused by token imbalance at inference time in Mixture of Experts, highlighting the optimization potential for reducing latency. • Toward the high-load experts, we propose Capacity-Aware Token Drop , which enforces capacity constraints by discarding excess tokens assigned to overloaded experts, thereby mitigating extreme load imbalance. • To better utilize underloaded experts, we introduce Capacity-Aware Expanded Drop , which expands the candidate expert set to include additional local experts, further improving load balance and model performance. • Extensive experiments on both language and multimodal models validate the effectiveness of our approach, demonstrating substantial improvements in inference efficiency with comparable performance. 2 Related Works Mixture of Experts Models The Mixture of Experts (MoE) is a kind of neural network architecture with an extended set of parameters (referred to as “experts”) controlled by a router, which is first introduced in the context of conditional computation [ 21 , 23 ] . The potential of sparse activation in MoE is subsequently exploited by [ 34 ] for efficient training and inference on pretrained models with special designs, opening the door for MoE in various vision [ 33 ] and language [ 25 , 13 , 14 ] scenarios. Attributed to its exceptional efficiency, MoE has been adopted as a foundational framework in the designs of large language models [ 22 , 9 , 41 , 45 , 39 ] , achieving superior scaling laws at low computational costs. Despite these advancements, MoE still faces efficiency challenges in both training and inference [ 3 ] , and our work specifically focuses on enhancing inference-time efficiency. Imbalance in Mixture of Experts The imbalance in token-to-expert assignments [ 44 , 5 ] poses a significant challenge to the deployment of MoE. This imbalance leads to inefficiencies in computation, communication, and memory [ 18 , 37 , 42 ] , making it a