Fast MoE Inference via Predictive Prefetching and Expert Replication DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06
Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar, Joseph Zuber · 2026
Read Fast MoE Inference via Predictive Prefetching and Expert Replication DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar, Joseph Zuber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11537.txt.
Introduction
Jyothish and three colleagues propose dynamic expert replication for fast mixture-of-experts inference. These models scale capacity without matching compute, but inference wastes GPU time as tokens wait on the same experts. Their method predicts which experts will overload and copies them for coming token batches. Copied experts run at once across layers, raising parallelism and cutting idle time. On Switch-base-128 and Switch-base-256 it reaches near-full GPU use and up to three times faster inference while keeping about 90 to 95 percent of baseline quality.
Catalog record
| CREATOR | Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar, Joseph Zuber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31451 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11537
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast MoE Inference via Predictive Prefetching and Expert Replication DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar, Joseph Zuber Abstract . The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-specific expertise in LLMs and scaling model capacity without proportionally increasing their computational overhead. However, MoE inference often suffers from suboptimal GPU utilization, load imbalance, and elevated latency arising from multiple tokens waiting on the same experts for their computation which arises from sparsity of expert activation. To address these challenges, we propose a dynamic expert replication strategy that predicts which experts are likely to be overloaded and replicates them for upcoming batches of tokens. The replicated experts process batch tokens concurrently across layers, which leads to improved parallelism, shorter GPU idle time, and significantly faster inference. Experimental evaluations conducted on large-scale MoE models, including Switch-base-128 and Switch-base-256, demonstrate that our method achieves near-complete GPU utilization ( ∼ \sim 100%), leading to upto 3x improvement in inference speed while preserving approximately 90-95% of the performance of baseline architectures. 1. Introduction Recent breakthroughs in machine learning, particularly in natural language processing, have primarily been enabled by exponential growth in computational resources, expansive training datasets, and increasingly larger models. However, training and deploying such massive models typically require extensive computational setups involving thousands of interconnected GPUs or accelerators running for prolonged periods, resulting in significant costs and energy consumption. Consequently, the machine learning community actively pursues more computationally efficient approaches, notably using sparse expert architectures. The Mixture-of-Experts (MoE) model is an exemplary sparse architecture that partitions neural network parameters into distinct subsets called "experts," with each expert selectively handling specific input samples. MoE architectures significantly reduce computational overhead compared to traditional dense networks, which apply all parameters to every input by conditionally routing inputs to relevant experts. Prominent language models such as GPT-4, Mixtral, Llama 4 and more highlight this evolution. This conditional computation strategy, first introduced by Shazeer ( 18 ) , has delivered state-of-the-art results in various domains, including natural language modeling, translation, computer vision, speech recognition, and multimodal learning. Recent comprehensive studies have also validated the scalability and effectiveness of MoE configurations across diverse model sizes and expert counts ( 13 ) ( 6 ) . Figure 1. Issues in MoE-based SwitchTransformers (MultiRC SuperGlue dataset) Despite these benefits, MoE architectures face substantial challenges during inference, particularly in latency, inefficient memory usage, and suboptimal GPU utilization. In practice, MoE inference can be substantially slower—up to 15 times for language modeling and approximately three times slower for machine translation—than equivalent dense models. Existing optimization techniques like distillation and GPU kernel-level enhancements typically provide limited or narrowly focused improvements, often at the expense of model accuracy or general applicability. Various SwitchTransformers have been compared in the terms of GPU utilisation, throughput and accuracy in the figure 1 for MultiRC dataset under SuperGlue when trained for the similar amount of time, values are normalized. In this paper, we address these critical inefficiencies by exploring predictive prefetching and expert replication techniques inspired by concepts of multiple producers and multiple consumers (MPMC) from distributed systems. Leveraging these insights allows us to efficiently manage concurrent expert access and resource allocation in GPU environments. Additionally, we demonstrate through rigorous experiments that predicting expert selections for tokens is not as challenging as traditionally perceived. Our findings show that accurate estimation of expert requirements is feasible when fine-tuning and extensive pretraining occur concurrently with token prediction tasks. This approach substantially enhances inference speed, optimizes GPU utilization, and maintains model accuracy, representing a significant step toward practical and economically viable deployment of MoE-based large-scale language models. Our contributions are threefold: (1) We introduce the highly efficient Simple Recurrent Unit (SRU) ( 11 ) to predict expert selections rapidly for each token, significantly speeding up inference. (2) We propose using deep physical replication of experts, which is different for each expert based on prediction, ensuring that almost no token experiences waiting delays during batch processing, thus substantially increasing GPU utilization, throughput, and also reducing inference latency. (3) We implement a practical strategy of capping expert replication at the GPU capacity, ensuring that replication is optimized within predefined GPU constraints to achieve near-optimal resource utilization. These innovations significantly enhance inference speed, optimize GPU utilization, and maintain model accuracy, representing a significant step toward the practical and economically viable deployment of MoE-based large-scale language models. 2. Preliminaries The prevalent challenges in Mixture-of-Experts (MoE) frameworks arise predominantly from the inherent sparse activation of experts during inference. In large language models (LLMs) employing MoE architectures, this sparsity induces significant inefficiencies in key performance metrics: latency, GPU utilization, and throughput. Latency, defined as the overall inference time for a given test dataset, is adversely affected by inefficient all-to-all routing, where excessive inter-expert communication and the overhead incurred by routing via softmax contribute to additional delays. The softmax-based routing mechanism, while effective in handling probabilistic assignments, introduces computational complexity that does not scale favorably with the increasing number of experts and tokens, thereby further exacerbating latency issues. Figure 2. SwitchTransformers MoE Layer with all experts, active and inactive(white) in GPU Simultaneously, GPU utilization is suboptimal because, at any moment, only a limited subset of experts is active while the majority of GPU resources remain idle. This phenomenon is compounded by the fact that the dynamic routing process—involving both softmax computation and assignment of tokens—does not sufficiently leverage the parallel processing capabilities of modern GPUs. Moreover, token stalling is observed when the routing algorithm fails to distribute the computational load evenly, which directly suppresses throughput, measured as the number of samples processed per second. This study is further motivated by the observation that traditional routing approaches, including softmax-based mechanisms, contribute substantially to the observed inefficiencies. The computational burden of softmax routing, combined with all-to-all communications and the uneven load distribution across experts, necessitates novel architectural adaptations ( 9 ) ( 8 ) ( 2 ) . MoE Overhead The growing number of experts in MoE models introduces significant sparsity, which results in substantial under-utilization of GPU resources. As more experts are added, a larger proportion of GPU memory sits idle during each forward pass, leading to severe inefficiencies during inference. This sparsity not only wastes precious compute resources but also increases the overhead associated with expert selection and communication. In large-scale models, such as those featuring 128 or 256 experts, this overhead can account for over 72% of the inference time, severely impacting overall performance as is shown in the introduction diagram ( 2 ) . Figure 3. SiDA-MoE MoE Layer with only predicted active distinct experts in GPU Figure 4. MoE-MPMC(Ours) MoE Layer with predicted active experts with their replicas in GPU Need for placing active experts in GPU with replication The selective and sparse activation of experts results in a highly imbalanced computational load distribution. For example, with a batch size of 64 and MoE layer comprising of 64 experts—where half of the tokens are routed to one expert and the remainder to another—only 2 out of 64 experts in the layer are heavily utilized,