MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models ISBN: 978-1-4503-XXXX-X/2018/06
Krishna Teja Chitty-Venkata 1 , Sylvia Howland 2 , Golara Azar 2 , Daria Soboleva 2 , Natalia Vassilieva 2 , Siddhisanket Raskar 3 , Murali Emani 1 , Venkatram Vishwanath 1 1 Argonne National Laboratory, 2 Cerebras, 3 Pacific Northwest National Laboratory {schittyvenkata, memani, venkat}@anl.gov, s. · 2025
Read MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models ISBN: 978-1-4503-XXXX-X/2018/06 by Krishna Teja Chitty-Venkata 1 , Sylvia Howland 2 , Golara Azar 2 , Daria Soboleva 2 , Natalia Vassilieva 2 , Siddhisanket Raskar 3 , Murali Emani 1 , Venkatram Vishwanath 1 1 Argonne National Laboratory, 2 Cerebras, 3 Pacific Northwest National Laboratory {schittyvenkata, memani, venkat}@anl.gov, s. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17467.txt.
Introduction
Krishna Teja Chitty-Venkata and seven co-authors benchmark mixture-of-experts models in a paper from two thousand twenty-five. Such models reach huge parameter counts while staying efficient, by routing each input to a few expert subnetworks. At inference time, they face load imbalance across experts and extra routing cost.
Their bench study varies batch size, sequence length, and key model settings across several model families. They test optimization tricks on high-end Nvidia GPUs, including pruning, fused operations, speculative decoding, and quantization. Parallel strategies were also compared. Results show how performance shifts across configurations. The authors aim to guide efficient deployment of these models in practice.
Catalog record
| CREATOR | Krishna Teja Chitty-Venkata 1 , Sylvia Howland 2 , Golara Azar 2 , Daria Soboleva 2 , Natalia Vassilieva 2 , Siddhisanket Raskar 3 , Murali Emani 1 , Venkatram Vishwanath 1 1 Argonne National Laboratory, 2 Cerebras, 3 Pacific Northwest National Laboratory {schittyvenkata, memani, venkat}@anl.gov, s. |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models ISBN: 978-1-4503-XXXX-X/2018/06 Krishna Teja Chitty-Venkata 1 , Sylvia Howland 2 , Golara Azar 2 , Daria Soboleva 2 , Natalia Vassilieva 2 , Siddhisanket Raskar 3 , Murali Emani 1 , Venkatram Vishwanath 1 1 Argonne National Laboratory, 2 Cerebras, 3 Pacific Northwest National Laboratory {schittyvenkata, memani, venkat}@anl.gov, [email protected], {sylvia.howland, golara.azar, daria.soboleva, natalia}@cerebras.net Abstract . Mixture of Experts (MoE) models have enabled the scaling of Large Language Models (LLMs) and Vision Language Models (VLMs) by achieving massive parameter counts while maintaining computational efficiency. However, MoEs introduce several inference-time challenges, including load imbalance across experts and the additional routing computational overhead. To address these challenges and fully harness the benefits of MoE, a systematic evaluation of hardware acceleration techniques is essential. We present MoE-Inference-Bench, a comprehensive study to evaluate MoE performance across diverse scenarios. We analyze the impact of batch size, sequence length, and critical MoE hyperparameters such as FFN dimensions and number of experts on throughput. We evaluate several optimization techniques on Nvidia H100 GPUs, including pruning, Fused MoE operations, speculative decoding, quantization, and various parallelization strategies. Our evaluation includes MoEs from the Mixtral, DeepSeek, OLMoE and Qwen families. The results reveal performance differences across configurations and provide insights for the efficient deployment of MoEs. 1. Introduction Mixture of Experts (MoE) models have emerged as a powerful paradigm for scaling neural networks, particularly in the domain of Large Language Models (LLMs). This approach offers a way to increase model capacity without a proportional rise in computational cost. MoE differs from dense models by using multiple specialized sub-networks, where each input activates only a subset of experts (as determined by a gating network). In contrast, dense models activate all parameters for every input, making MoE architectures significantly more parameter-efficient. Architectures such as the Switch Transformer ( Fedus et al. 2022 ) and GShard ( Lepikhin et al. 2020 ) , along with more recent open-source MoE models like Mixtral ( Jiang et al. 2024 ) , Llama4 ( Meta 2025 ) , DeepSeekMoE ( Dai et al. 2024 ) , and Kimi ( Team et al. 2025 ) , exemplify the rapid advancements in MoE-based systems These models leverage sparse weight activation, enabling large networks to maintain inference efficiency. MoE models are now widely used in applications such as text generation, retrieval-augmented generation, and multimodal reasoning. However, despite their computational advantages, MoE models also pose unique challenges in inference, training stability, memory usage, and hardware utilization due to load imbalance and dynamic routing. (a) Mixtral-8x7B (b) OLMoE-1B-7B (c) Qwen1.5-MoE (d) Mixtral-8x7B (e) OLMoE-1B-7B (f) Qwen1.5-MoE Figure 1 . Layer-wise Total and Active Parameter Breakdown for Mixtral-8x7B, OLMoE-1B-7B, and Qwen1.5-MoE MoE Inference ( Liu et al. 2024b ) plays a central role in modern AI applications, as it involves executing the forward pass of a sparsely activated model where only the top- k experts per token are evaluated. Efficient inference is crucial for maximizing the benefits of sparsity in real-world deployments. As MoE models continue to grow in scale and complexity, optimizing inference is critical to achieve low latency and energy-efficien execution on modern accelerators. This includes mitigating expert load imbalance, reducing communication overhead in distributed settings, and designing scheduling strategies that fully exploit sparsity for throughput gains. The MoE ecosystem has witnessed a convergence of three key trends: the rise of open-source MoE models, advancements in AI accelerators, and the development of inference frameworks like vLLM ( Kwon et al. 2023 ) and FasterMoE ( He et al. 2022 ) optimized for sparse execution. This synergy highlights the importance of robust benchmarking to evaluate MoE performance across diverse hardware setups. Benchmarking exposes critical trade-offs between throughput, latency, and memory footprint, enabling informed decisions about model deployment and architecture optimization. The evolution of AI hardware such as GPUs and specialized AI accelerators has been instrumental for the ever-rising computational demands of MoE models. These accelerators offer high parallelism and memory bandwidth, essential for models with billions of parameters and dynamic computation graphs. However, MoE architectures also introduce new hardware challenges, such as expert placement, routing overhead, and under-utilization due to sparse activations. Addressing these hardware inefficiencies requires co-designing inference systems that are both MoE-aware and hardware-efficient. Figure 1 shows that MoE layers dominate both total and active parameters across different models, emphasizing their critical role in computational cost and memory footprint. Since MoE layer weights account for a substantial portion of the model, understanding the MoE performance is essential for optimized deployment. In this paper, we introduce MoE-Inference-Bench, a comprehensive benchmarking suite designed to systematically evaluate MoE models across a wide range of optimization techniques. Our benchmark analyzes throughput, latency, and hardware utilization for state-of-the-art MoE models, shedding light on the practical implications of sparse inference and routing dynamics. Our comprehensive study provides several insights for researchers aiming to deploy MoE models efficiently, and contributes to the broader goal of scalable and cost-effective AI deployment in the era of massive model sparsity. The main contributions of our paper are as follows: (1) Comprehensive MoE Benchmarking Suite: We propose MoE-Inference-Bench to evaluate MoE performance under diverse inference scenarios. Our suite spans models from 2B to 70B parameters, covering multiple architectures (Mixtral, DeepSeek, Qwen, Phi, OLMoE). Our study examines multiple factors that significantly influence the inference performance of MoEs, providing insights for future designs. (2) Fine-Grained Hyperparameter Scaling Analysis: We perform an extensive exploration of key MoE layer hyperparameters, which include FFN dimension, total expert count, and active expert ratio to quantify their individual and joint impact on throughput and out-of-memory boundaries on Nvidia H100 GPUs. Our results identify optimal MoE operating constraints and reveal clear trade-offs between model size, expert sparsity and hardware efficiency. (3) Inference Optimizations: We systematically assess multiple inference-time acceleration techniques such as quantization, intra and inter expert pruning, speculative decoding and Fused MoE, highlighting their effectiveness across batch sizes and sequence lengths. We also benchmark MoE inference across Nvidia H100 GPUs, analyzing the effects of tensor, pipeline, and expert parallelism strategies. 2. Background and Related Work Large Language and Vision Models Modern LLMs are predominantly built upon the transformer architecture ( Vaswani et al. 2017 ) , which comprises stacks of decoder layers. These layers incorporate core components such as token embeddings, positional encodings, multi-head self-attention, and feed-forward networks. VLMs combine vision and language capabilities to simultaneously process both visual data and textual information, enabling them to perform multimodal tasks such as image captioning and visual question answering. Mixture-of-Experts LLMs Dense architectures represent the conventional LLM, where a single, monolithic neural network activates all parameters for every token. This design facilitates comprehensive information processing but incurs substantial computational and memory costs ( Touvron et al. 2023 ) . Mixture-of-Experts (MoE) models ( Shazeer et al. 2017 ; Cai et al. 2024 ) incorporates multiple specialized subnetworks within selected layers, typically the FFN blocks, as shown in Figure 2 . A learnable routing mechanism activates only a subset of experts per token, improving parameter efficiency and potentially accelerating inference without proportionally increasing compute. Notable Examples include Mixtral-8x7B ( mistralai 2023 ) , where expert specialization enables scaling to larger total parameter counts while mitigating the runtime overhead of