Faster MoE LLM Inference for Extremely Large Models

Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo DuWuhan UniversityWuhan, 430072, P. R. ChinaMengjia ShenWuhan Second Ship Design And Research InstituteWuhan, 430010, P. R. ChinaHai ZhaoShanghai Jiao Tong UniversityShanghai, 200240, P. R. China · 2025

Read Faster MoE LLM Inference for Extremely Large Models by Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo DuWuhan UniversityWuhan, 430072, P. R. ChinaMengjia ShenWuhan Second Ship Design And Research InstituteWuhan, 430010, P. R. ChinaHai ZhaoShanghai Jiao Tong UniversityShanghai, 200240, P. R. China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03531.txt.

Introduction

Haoqi Yang and co-authors study how to run very large mixture-of-experts language models faster in a 2025 computing paper. Such models split work among many expert subnetworks, firing only a few per query. Newer designs slice the experts finer, and serving them well is poorly understood. The team measured speed and quality while varying how many experts stay active and how many exist in total.

Cutting the active expert count bought large speed gains in some load conditions, with only small quality loss. Cutting the total count saved little speed yet hurt quality badly. The asymmetry gives operators a clear tuning rule: trim what fires per query, not what the model owns. Their method raised throughput on top of that insight.

Catalog record

CREATORHaoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo DuWuhan UniversityWuhan, 430072, P. R. ChinaMengjia ShenWuhan Second Ship Design And Research InstituteWuhan, 430010, P. R. ChinaHai ZhaoShanghai Jiao Tong UniversityShanghai, 200240, P. R. China
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03531

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Faster MoE LLM Inference for Extremely Large Models Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo DuWuhan UniversityWuhan, 430072, P. R. ChinaMengjia ShenWuhan Second Ship Design And Research InstituteWuhan, 430010, P. R. ChinaHai ZhaoShanghai Jiao Tong UniversityShanghai, 200240, P. R. China Abstract Sparse Mixture of Experts (MoE) large language models (LLMs) are gradually becoming the mainstream approach for ultra-large-scale models. Existing optimization efforts for MoE models have focused primarily on coarse-grained MoE architectures. With the emergence of DeepSeek Models, fine-grained MoE models are gaining popularity, yet research on them remains limited. Therefore, we want to discuss the efficiency dynamic under different service loads. Additionally, fine-grained models allow deployers to reduce the number of routed experts, both activated counts and total counts, raising the question of how this reduction affects the trade-off between MoE efficiency and performance. Our findings indicate that while deploying MoE models presents greater challenges, it also offers significant optimization opportunities. Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios, with only minor performance degradation. Reducing the total number of experts provides limited efficiency gains but results in severe performance degradation. Our method can increase throughput by at least 10% without any performance degradation. Overall, we conclude that MoE inference optimization remains an area with substantial potential for exploration and improvement. 1 Introduction The introduction of the Transformer ( Vaswani et al. 2017 ) has significantly improved the training efficiency of sequence models, making the training of ultra-large-scale models feasible. Building upon this foundation, GPT-style decoder-only Transformer models ( Radford et al. 2018 ; Radford et al. 2019 ; Brown et al. 2020 ; OpenAI 2023 ) , guided by scaling laws ( Kaplan et al. 2020 ) , have rapidly gained prominence ( Touvron et al. 2023a ; Touvron et al. 2023b ; Dubey et al. 2024 ) . Their exceptional scalability has made them the mainstream choice for constructing large-scale generative language models ( Chowdhery et al. 2023 ) . To further expand model size while controlling both inference and training costs, one promising direction is the sparsification of the feed-forward network (FFN), which constitutes the majority of parameters in LLMs. The FFN module in LLMs is typically implemented as either a Multi-Layer Perceptron (MLP, Murtagh 1991 ) or a Gated Linear Unit (GLU, Dauphin et al. 2017 ). Both first project the hidden state to an intermediate state via an upsampling transformation, followed by an activation function, and then downsampled to produce the new hidden state. The intermediate state in MLP and GLU can be easily partitioned: a large one can be divided into multiple smaller ones while maintaining strict equivalence. This enables a strategy where an excessively large intermediate state is split into multiple smaller states, with only a subset of them being selectively activated through gating. Parameters corresponding to each partition is referred to as an ”expert”, and this architecture is known as the Sparse Mixture-of-Experts (MoE, Shazeer et al. 2017 ). MoE was demonstrated to be effective in Transformer architectures by Fedus et al. 2022 . Mixtral-8x7B ( Jiang et al. 2024 ) is the most famous large-scale open-source MoE LLM. The deployment of Large Language Models (LLMs) has long been a significant challenge. Due to the autoregressive decoding, model parameters are repeatedly accessed during inference ( Yu et al. 2022 ) but with only a few computation, leading to low arithmetic intensity. According to the Roofline model ( Williams et al. 2009 ) , this harms the computational efficiency of the system. Several existing works have optimized this aspect by enlarging batch size ( Kwon et al. 2023 ; Zheng et al. 2024 ; Ye et al. 2024 ) . While significant progress has been made in improving the training efficiency of MoE models ( He et al. 2021 ; He et al. 2022 ; Zhai et al. 2023 ; Rajbhandari et al. 2022 ) , accelerating MoE modules during inference remains an underexplored area. In particular, the theoretical performance of MoE-based inference systems under varying batch sizes has yet to be thoroughly investigated. Figure 1 : The comparison of FFN, coarse-grained MoE, and fine-grained MoE. Traditional MoE models employ a relatively conservative number of experts, typically eight in total and two actived per token. Moreover, these experts are generally initialized from a pre-trained dense small model. Dai et al. 2024 introduced several modifications. They significantly increased the number of experts, both total and activated, while reducing the size of each expert. Additionally, all experts were randomly initialized. To mitigate the instability associated with training smaller experts, they introduced a larger shared expert which is always chosen as a backbone. The release of Deepseek V2 ( DeepSeek-AI et al. 2024a ) , featuring 236 billion parameters, demonstrated the scalability of this paradigm. DeepSeek-AI et al. 2025 further optimized the expert selection mechanism by incorporating grouping constraints and sigmoid activation, instead of softmax, leading to the development of the Deepseek V3 model. We refer to the deepseek approach as fine-grained MoE, while the conventional MoE paradigm is termed coarse-grained MoE. As shown in Figure 1 . Fine-grained MoE enhances the model’s adaptability, allowing for more flexible parameter utilization during inference. Studies have explored discarding certain parameters at inference time to improve decoding speed. These approaches include expert pruning, where specific experts are removed and not loaded into memory ( Chen et al. 2022 ; Lee et al. 2024 ; Lu et al. 2024 ; Xie et al. 2024 ; Yang et al. 2024b ) , and expert skipping, where the number of activated experts per token is reduced ( Lu et al. 2024 ) . Although some prior work has explored related topics, it has predominantly focused on designs tailored for coarse-grained MoE rather than the more promising fine-grained variant. In summary, this paper presents an efficiency analysis of fine-grained MoE models under varying batch sizes during inference. We aim to address the question of whether MoE models can achieve efficiency gains only in large-scale service scenarios. Furthermore, we investigate the impact of different expert reduction strategies on both model performance and computational efficiency to assess whether this direction warrants further exploration, particularly in fine-grained MoE and its sigmoid-activated variant, in which the model exhibits different dynamical behaviors, necessitating a distinct analytical approach. 2 Background and Related Works 2.1 FFN and MoE Feed-forward network is usually an MLP or a GLU, given in Equation 1 , where \mathrm{ACT}(\cdot) denotes the activation function, and \otimes denotes the element-wise product. \displaystyle\mathrm{MLP}(h) \displaystyle=W_{\mathrm{d}}\cdot\mathrm{ACT}\left(W_{\mathrm{u}}\cdot h\right) (1) \displaystyle\mathrm{GLU}(h) \displaystyle=W_{\mathrm{d}}\cdot\left(\mathrm{ACT}\left(W_{\mathrm{u}}\cdot h\right)\otimes\left(W_{\mathrm{g}}\cdot h\right)\right) We use GLU to represent FFN from now on since it’s the more common practice in current days ( Shazeer 2020 ; Chowdhery et al. 2023 ) . A mixture of experts layer can be represented as the weighted combination of multiple GLUs, where the weights r are given by a linear classifier called router. We define router logits as r^{\prime} . Moreover, there is usually two function to modify the logits, F_{r}(\cdot) to manipulate Top k selection for load balancing, and F_{w}(\cdot) to normalize logits into the final weights. An MoE layer with n_{e} experts and n_{a} activated is given in Equation 2 \displaystyle\mathrm{MoE}(h) \displaystyle=\sum_{i=1}^{n_{e}}r_{i}\cdot\mathrm{GLU}_{i}(h) (2)

Read the complete text