MoEless : Efficient MoE LLM Serving via Serverless Computing Conference: ; March 5; 2026
Hanfei Yu, Bei Ouyang, Shwai He, Ang Li, Hao Wang · 2026
Read MoEless : Efficient MoE LLM Serving via Serverless Computing Conference: ; March 5; 2026 by Hanfei Yu, Bei Ouyang, Shwai He, Ang Li, Hao Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06350.txt.
Introduction
Yu and colleagues present MoEless, a serverless MoE serving framework tackling expert load imbalance elastically. Sparse activation makes a few experts stragglers while others sit idle, inflating latency and cost. Static serverful deployments limit elasticity and force costly swapping. MoEless uses lightweight layer aware predictors to estimate expert load distributions and proactively identify stragglers. Optimized scaling and placement improve locality, GPU use, and cross expert load balance. On Megatron-LM across eight GPUs, MoEless cuts latency by 43 percent and cost by 84 percent against state of the art solutions.
Catalog record
| CREATOR | Hanfei Yu, Bei Ouyang, Shwai He, Ang Li, Hao Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51905 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06350
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoEless : Efficient MoE LLM Serving via Serverless Computing Conference: ; March 5; 2026 Hanfei Yu, Bei Ouyang, Shwai He, Ang Li, Hao Wang Abstract . Large Language Models (LLMs) have become a cornerstone of AI, driving progress across diverse domains such as content creation, search and recommendation systems, and AI-assisted workflows. To alleviate extreme training costs and advancing model scales, Mixture-of-Experts (MoE) has become a popular backbone for modern LLMs, which are commonly served in distributed deployment using expert parallelism (EP). However, MoE’s sparse activation mechanism leads to severe expert load imbalance, where a few experts become overloaded while others remain idle, resulting in expert stragglers that inflate inference latency and serving cost. Existing expert load balancing solutions assume static resource configurations on serverful infrastructures, limiting expert scalability and elasticity, and resulting in either costly real-time expert swapping or degraded generation quality. We present MoEless , the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts. MoEless employs lightweight, layer-aware predictors to accurately estimate incoming expert load distributions and proactively identify stragglers. We design optimized expert scaling and placement strategies to maximize function locality, improve GPU utilization, and balance loads across experts and GPUs. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions. 1. Introduction Motivation. LLM have revolutionized NLP ( NLP ) research and driven breakthroughs across diverse application domains, such as content generation ( 15 ; 2 ; 9 ; 57 ) , information retrieval and recommendation ( 37 ; 80 ) , and AI ( AI )-assisted decision-making ( 49 ; 36 ; 30 ) . To mitigate the extreme training costs, modern LLM increasingly adopt the MoE ( MoE ) architecture ( 29 ; 64 ; 73 ; 71 ; 14 ; 1 ) as their core design. MoE layers replace the conventional FFN ( FFN ) layers in Transformer blocks ( 69 ) with a gating network and a collection of experts, where only a small subset is activated during computation. This design significantly reduces the number of FLOP , enabling MoE -based LLM to deliver comparable or even superior performance to dense LLM at a fraction of the training cost ( 14 ; 73 ; 29 ) . Due to the immense model scale, serving MoE -based LLM requires distributed deployment under the EP ( EP ) paradigm ( 38 ) . However, expert load imbalance has been identified as a fundamental challenge in distributed serving ( 38 ; 22 ; 42 ) , where certain experts become highly popular and receive overwhelming loads compared to others. Figure 1 illustrates the expert load imbalance in two representative MoE models, Mixtral-8 \times 7B and Phi-3.5-MoE, across two real-world datasets. Such imbalance leads to the expert straggler problem ( 24 ) , which severely increases inference latency and serving cost in MoE serving. Figure 1 . Expert load imbalance across layers for different MoE models and datasets: (a) Mixtral-8 \times 7B on ShareGPT and (b) Phi-3.5-MoE on LMSYS-Chat-1M. Limitations of state-of-the-art approaches. Expert load balancing ( 38 ; 35 ; 24 ) has been extensively explored to mitigate the inherent load imbalance among experts in MoE serving. However, existing approaches assume static resource configurations on serverful infrastructures, resulting in either costly real-time expert swapping ( 38 ; 22 ; 35 ) with limited effectiveness, or lossy expert re-routing ( 24 ) that compromises generation quality. Achieving expert load balance without generation performance loss requires fine-grained, elastic, and accurate expert scaling in MoE serving. Key insights and contributions. Serverless computing offers a promising alternative and has emerged as a new computing paradigm for modern AI infrastructures. Serverless ML ( ML ) inference and LLM serving have gained significant attention from both academia ( 20 ; 68 ; 67 ; 76 ; 45 ) and industry ( 6 ; 7 ; 5 ; 50 ) . In this paper, we propose MoEless , the first serverless MoE serving framework that mitigates expert load imbalance and accelerates MoE inference via serverless experts. MoEless decouples experts from MoE models under the EP paradigm and integrates them with serverless functions to enable scalable and elastic execution. To proactively identify expert stragglers, we design lightweight predictors that accurately estimate upcoming expert load distributions with layer awareness. Based on the predicted load characteristics, MoEless dynamically scales expert replicas to eliminate stragglers and balance workloads across both the expert and GPU levels, thereby minimizing inference latency. Furthermore, we design optimized expert placement strategies to maximize function locality and GPU utilization while reducing all-to-all communication overheads in EP deployments. In summary, we make the following contributions: • We propose MoEless , the first serverless MoE serving framework that accelerates inference by mitigating expert load imbalance via serverless experts. • We design layer-aware and lightweight expert load predictors that accurately estimate incoming expert load distributions across different layers. • We develop dynamic expert scaling and placement strategies that efficiently balance workloads across experts and GPUs to eliminate straggler problems. • We prototype MoEless on top of Megatron-LM ( 63 ) , deploy it on an eight-GPU testbed, and conduct extensive evaluation against SOTA ( SOTA ) approaches. Experimental methodology. We prototype MoEless on top of the Megatron-LM framework ( 63 ) . All experiments are conducted on a testbed with eight NVIDIA A6000 GPUs, with a total of 384 GB of GPU memory, interconnected via pairwise NVLinks. We evaluate MoEless using three representative MoE -based LLM , Mixtral-8 \times 7B ( 29 ) , Phi-3.5-MoE ( 1 ) , and Llama-4-Scout ( 48 ) , across two real-world datasets, LMSYS-Chat-1M ( 81 ) and ShareGPT ( 53 ) . We compare our approach against SOTA expert load balancing methods, including Megatron-LM ( 63 ) , EPLB ( EPLB ) ( 38 ) , and an Oracle baseline ( 24 ) . Extensive experiments show that MoEless reduces inference latency by 43% and inference cost by 84% compared to SOTA baselines. Limitations of the proposed approach. In this work, we adopt standard serverless function management schemes, such as pre-warming and fixed-duration keep-alive periods, to mitigate expert function cold starts. System parameters ( e.g. , prediction distance and load-balancing thresholds) are primarily determined through offline profiling, rather than being automatically or dynamically adapted across models and datasets. We leave the design of more advanced runtime optimizations to future work. 2. Background and Motivation 2.1. Large Language Model Serving LLM have been extensively studied and deployed in both industrial production ( 54 ; 3 ; 32 ; 56 ) and academic research ( 33 ; 43 ; 16 ; 82 ) . Modern LLM typically adopt the Transformer decoder-only architecture ( 69 ) , which operates in two consecutive stages: prefill and decode . Figure 2 illustrates the auto-regressive inference process, where an LLM processes an input prompt batch and incrementally generates new tokens. During the prefill stage, the model processes all input prompt sequences in parallel and outputs the first new tokens for the batch in a single iteration . 1