eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference

Suraiya Tairin *, Shohaib Mahmud *, Haiying Shen, Anand Iyer · 2025

Read eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference by Suraiya Tairin *, Shohaib Mahmud *, Haiying Shen, Anand Iyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06823.txt.

Introduction

Suraiya Tairin and co-authors present eMoE, a system that serves mixture-of-experts language models without keeping every expert in GPU memory. Their 2025 paper notes that these models gain capacity from many specialist sub-networks, but storing all of them is expensive. Loading experts from slower memory on demand can stall answers instead.

The system predicts which experts upcoming prompts will need, based on repeating routing patterns. It refreshes those predictions only every few prompts, since reusing recent experts barely hurts quality. It also skips prediction for tasks that tolerate rough routing, and schedules work with deadlines and output lengths in mind. Tests cut memory use by up to four fifths and trimmed latency as well. Longer prompts and larger batches became practical on the same hardware.

Catalog record

CREATORSuraiya Tairin *, Shohaib Mahmud *, Haiying Shen, Anand Iyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference Suraiya Tairin *, Shohaib Mahmud *, Haiying Shen, Anand Iyer Abstract In recent years, Mixture-of-Experts (MoE) has emerged as an effective approach for enhancing the capacity of deep neural network (DNN) with sub-linear computational costs. However, storing all experts on GPUs incurs significant memory overhead, increasing the monetary cost of MoE-based inference. To address this, we propose eMoE, a memory-efficient inference system for MoE-based large language models (LLMs) by leveraging our observations from experiment measurements. eMoE reduces memory usage by predicting and loading only the required experts based on recurrent patterns in expert routing. To reduce loading latency while maintaining accuracy, as we found using the same experts for subsequent prompts has minimal impact on perplexity, eMoE invokes the expert predictor every few prompts rather than for each prompt. In addition, it skips predictions for tasks less sensitive to routing accuracy. Finally, it has task-aware scheduling to minimize inference latency by considering Service Level Objectives (SLOs), task-specific output lengths, and expert loading latencies. Experimental results show that compared to existing systems, eMoE reduces memory consumption by up to 80% while maintaining accuracy and reduces inference latency by up to 17%. It also enables processing prompts 40 × \times longer, batches 4.5 × \times larger, and achieves 1.5 × \times higher throughput. * * footnotetext: Equal contribution. 1 Introduction A recent trend in Deep learning (DL) has been increasing the model performance by increasing the number of model parameters [ 1 , 2 ] . However, the linear relationship between the computational cost and the model size remains as a crucial impediment to the practical applications of the large models. To circumvent this bottleneck, sparse architectures [ 3 , 4 , 5 , 6 , 7 , 8 ] have been introduced. Among these, the Mixture-of-Experts (MoE) stands out as one of the most prominent sparse models. Its adoption has substantially amplified the scalability of DNN models across various DL applications, encompassing natural language processing [ 9 , 10 , 11 ] , computer vision [ 12 ] , speech recognition [ 13 , 14 ] , and recommendation systems [ 15 ] . An MoE layer comprises a gate and a collection of experts. The gate selects only a small number (e.g., 1 or 2) of experts for computation based on each input. Its sparse expert activation enables considerable expansion of model size by several orders of magnitude without a commensurate increase in computational requirements (FLOPs). However, the scaling of such models demands an excessively large amount of scarce GPU memory [ 16 ] . This directly corresponds to substantial monetary expenses when implementing MoE-based inference systems at scale. Our data analysis, detailed in § 2 , reveals that an MoE-based model typically consumes 4x to 14x more memory compared to its dense counterpart. Due to the intricate dependence between model capacity and memory requirements, designing a memory-optimized MoE inference system is crucial for serving very large models. Existing state-of-the-art inference systems such as vLLM [ 16 ] , DeepSpeed-FastGEN [ 17 ] , Sarathi-Serve [ 18 ] , DistServe [ 19 ] are designed for the traditional transformer architecture and thus unequipped to provide memory-efficient MoE inference. In this work, we propose eMoE, a memory-efficient inference system for MoE-based large language models (LLMs). eMoE is designed by leveraging our observations from experiment measurements. eMoE integrates several components working in unison: \bullet Expert Prediction. We observe a recurring pattern in token-to-expert routing, where certain experts process more tokens than others within a given time period (i.e., popular experts). Based on this observation, we propose an on-demand loading of popular experts during inference to reduce memory footprint. By leveraging the prior distribution of expert selection, eMoE predicts and proactively loads the experts for future inputs. \bullet Periodic Expert Invocation. However, we found that time-efficient on-demand expert loading is challenging due to the increased inference latency. We observe that subsequent prompts are highly correlated, and using the same experts for subsequent prompts does not significantly affect perplexity. To reduce expert loading overhead while maintaining accuracy, eMoE invokes the expert predictor every few prompts rather than for each prompt. \bullet Task-aware Expert Loading. Additionally, our data analysis reveals that an MoE-based model can produce the same output with inaccurate experts for certain tasks as it does with the correct experts. Leveraging this observation, eMoE skips predictions for tasks less sensitive to token-to-expert-routing accuracy. \bullet Task-aware Request Scheduling. Furthermore, leveraging the above observation with another observation that certain tasks generate significantly fewer tokens than others, eMoE uses profiled task-specific token generation length, task-aware expert loading latency and user-imposed inference latency Service Level Objective (SLO) in request scheduling with an objective to minimal impact on the end-to-end inference latency. It offers an advantage over existing request schedulers [ 20 , 16 , 18 , 19 ] by considering expert loading latency. Unlike previous methods that rely on continuous expert prefetching during inference [ 21 , 22 , 23 ] , eMoE periodically loads and reuses experts, reducing the inference latency associated with these approaches. The limitations of these methods are discussed in § 2.2 . Our real experimental evaluations show that compared to existing systems, eMoE reduces memory consumption by up to 80% while maintaining accuracy and reduces inference latency by up to 17%. It also enables processing prompts 40 \times longer, batches 4.5 \times larger, and achieves 1.5 \times higher throughput. In summary, we make the following contributions: \bullet We systematically analyze token-to-expert recurrence patterns and task-specific characteristics, such as tolerance for less accurate experts and varying output lengths. Our findings reveal that existing solutions for memory-efficient MoE inference are insufficient in addressing both memory optimization and inference latency effectively. \bullet We propose eMoE, a memory-efficient inference system that uses expert prediction to load experts, incorporates periodic expert invocation, employs task-aware expert loading to minimize latency for less routing-sensitive tasks, and utilizes task-aware request scheduling to optimize both inference latency and memory consumption. \bullet We evaluate eMoE against state-of-the-art transformer-based inference systems, demonstrating significant reductions in memory consumption while preserving accuracy and minimizing end-to-end inference latency. 2 Observation and Motivation 2.1 Experiment Settings We conducted an analysis experiment on four popular publicly available MoE based LLMs: Mixtral-8x7B [ 24 ] , Mixtral-8x22B, TinyMixtral [ 24 ] and OpenMoE [ 25 ] . We used various popular natural language processing (NLP) tasks including text summarization, question answering (QA), semantic similarity comparison and semantic classification. Table 1 : Dataset summary. Dataset Contents Task XSUM [ 26 ] News articles Summarize (SUM) Tweet-sentiment [ 27 ] Tweets related to financial markets, stocks, and economic discussions Classify tweet sentiment (CLSFY) SQUAD [ 28 ] Context and question pairs Question answer (QA) Headlines similarity [ 29 ] Pairs of headlines on same topics Compare semantic similarity (COMP) Human-assistant conversation [ 30 ] Pairs of prompts and responses Conversation (CONV) Details of the datasets used for these tasks are summarized in Table 1 . Following [ 20 ] and [ 16 ] , we generated synthetic client request traces since there is no publicly available request trace for generative MoE-based language models. The generated trace follows the Poisson distribution. To generate a

Read the complete text