Adrian Zhao

1 · 2026

Read Adrian Zhao by 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28768.txt.

Introduction

Zhao and co-authors present CRAFT, a cost-aware method for replicating experts in Mixture-of-Experts serving. Expert parallelism splits experts across devices, but token loads become uneven at inference. Replication can balance loads, yet existing schemes often over-replicate. CRAFT replicates in a fine-grained, per-layer way based on estimated benefit. It fits a fixed memory budget and needs no training or model changes. In large-scale tests with very large models, CRAFT raised serving throughput by 1.14 times on average over prior techniques.

Catalog record

CREATOR1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47413
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adrian Zhao 1 Abstract Mixture-of-Experts (MoE) has recently emerged as the mainstream architecture for efficiently scaling large language models while maintaining near-constant computational cost. Expert parallelism distributes parameters by partitioning experts across devices, but this introduces token-level load imbalance during inference. Expert replication is a widely adopted load-balancing technique in serving frameworks that alleviates load imbalance in large-scale deployments by replicating experts with high loads. In this work, we demonstrate that existing replication schemes often over-replicate , with many replicas providing marginal improvement. Replicas consume substantial GPU memory, which may lead to resource contention and throughput degradation. We present CRAFT, an efficient expert replication framework that maximizes load balance under a given memory budget by performing fine-grained, per-layer replication based on the estimated replication benefit. CRAFT can be seamlessly integrated into existing serving frameworks without any additional training or model changes. Our evaluation shows that CRAFT increases end-to-end serving throughput by 1.14 × 1.14\times on average (up to 1.2 × 1.2\times ) over existing replication techniques in large-scale deployments with models ranging from hundreds of billions to a trillion parameters. marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. CRAFT: C ost-Aware Expert R eplica A llocation with F ine-Grained Layerwise Es t imations 1 Introduction Mixture-of-Experts (MoE) alleviates the steep compute and memory scaling of dense transformers by replacing the dense feed-forward blocks with a router and a pool of experts 23 ; 18 ; 44 . By activating only a subset of experts per token, MoE models decouple total parameters from per‑token FLOPs and achieve near‑constant inference cost as capacity scales 7 . It has become the foundation for recent large models across various domains, such as LLMs 14 ; 51 ; 35 ; 24 , image and video diffusion models 49 ; 10 , and vision transformers 5 , delivering competitive performance at substantially lower compute than dense counterparts. However, deploying MoE models incurs substantial operational costs, primarily due to the massive GPU memory footprint required to host all expert parameters 42 ; 12 . Therefore, system-level optimizations that improve inference throughput yield significant cost-performance benefits by maximizing resource utilization and reducing GPU memory pressure 9 ; 48 ; 30 . To effectively distribute large expert parameters across devices, Expert Parallelism (EP) partitions experts across devices, with each device holding weights of a subset of experts. EP uses two all‑to‑all collectives to dispatch tokens to the devices that host the assigned experts and then recombines them 12 ; 28 . Despite its promising scalability, EP introduces a new system bottleneck during deployment — expert load imbalance . The MoE router routes tokens to experts based on input token distributions and often sends a disproportionate number of tokens to a few hot experts while other experts remain cold 19 ; 22 ; 8 . Thus, the cumulative token loads across GPUs become imbalanced, leading to: (i) idleness on GPUs with low load and (ii) network congestion during dispatch and combine due to imbalanced traffic 29 ; 32 ; 16 ; 33 . This degrades communication and computation efficiency during MoE inference, necessitating load-balancing strategies for EP. Expert placement and expert replication are two load-balancing techniques applicable to any MoE model and are widely adopted in modern LLM serving frameworks 56 ; 26 ; 38 ; 42 . Expert placement leverages expert load distribution to assign experts to devices such that the expected tokens per device and traffic patterns are balanced 52 ; 13 ; 29 . However, expert placement fails in layers where a few experts dominate most of the token load, making a balanced packing impossible. Expert replication addresses this by replicating hot experts 31 . This mitigates extreme load imbalance by distributing token loads from a single hot expert to multiple devices. Although expert replication is an effective load-balancing strategy that mitigates extreme expert load imbalance, it requires substantial GPU memory to store the replica weights. The most widely adopted replication technique EPLB 31 allocates one replica per layer per GPU, incurring significant GPU memory overhead on large MoE models with many MoE layers. Large-scale LLM deployments are already memory-intensive due to high parameter counts and the large KV cache necessary to efficiently process long-context requests 26 ; 6 . Therefore, the memory overhead limits the practicality of replication for large-scale MoE deployments. Expert placement does not incur additional memory overhead, but it cannot handle extreme load skew during inference. Thus, efficiently achieving balanced expert load in large-scale deployment remains an important unsolved problem for large MoE models. Figure 1: Balancedness gain (orange) and throughput gain (blue) of CRAFT on Kimi-K2-1000B deployed on 64 GPUs, normalized to the baseline with only expert placement (refer to KE8 configuration in Section 3.1 ). EPLB is configured with 60 replicas per GPU on Kimi-K2 with 60 MoE layers (at minimum one replica per layer per GPU). The X-axis (expert replicas per GPU) is presented on a base-2 logarithmic scale. In this work, we perform a detailed characterization of the benefits and overheads of replication as a means to achieve load balance. We make two key observations: (1) As the number of replicas increases, balancedness gains diminish rapidly. (2) MoE layers benefit differently from replication, which can be roughly categorized into two types: layers with few experts dominating token load (high-skew) benefit substantially from replication, while layers with balanced token load (low-skew) benefit little. Leveraging these observations, we introduce CRAFT, an end-to-end cost-aware replica allocation framework for efficient expert replication. The key idea of CRAFT is to selectively allocate replicas at fine, per-layer granularity to layers that benefit more from replication. We develop a cost model that estimates the per-layer balancedness gain with replication (i.e., replication benefit ). Based on this cost model, CRAFT computes a per-layer replication plan with optimal balancedness by assigning more replicas to layers with higher estimated gains. Figure 1 demonstrates the effectiveness of CRAFT: (1) CRAFT allocates replicas for each layer based on estimated performance benefits. This enables replication under flexible memory budgets that are much lower than EPLB (depicted by the blue curve), reducing memory usage while preserving balancedness. (2) CRAFT with the optimal number of replicas (CRAFT-OPT) achieves 1.17\times higher throughput compared to EPLB. The number of replicas can either be manually set by the user to meet specific memory constraints, or CRAFT can automatically set a value that optimizes replication efficiency. We implement CRAFT in three steps: First, we analyze offline expert load distributions and estimate per-layer benefits across different replica counts. Then, we use dynamic programming to select the optimal number of replicas per layer under the memory budget. Finally, we reserve memory and assign experts evenly across devices to minimize capacity imbalance, producing a complete replication plan for deployment. Our contributions are summarized as follows: • To our knowledge, this is the first detailed characterization of how throughput and load balancedness scale with replication.

Read the complete text