FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Qingxiu Liu 1 , Cyril Y. He 2∗ , Hanser Jiang 2 , Zion Wang 2 , Alan Zhao 2 , Patrick P. C. Lee 1 · 2026
Read FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving by Qingxiu Liu 1 , Cyril Y. He 2∗ , Hanser Jiang 2 , Zion Wang 2 , Alan Zhao 2 , Patrick P. C. Lee 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02715.txt.
Introduction
Liu and six colleagues present FluxMoE, a serving system that frees mixture-of-experts models from keeping every expert on the GPU at once. Existing engines crowd out the key-value cache in large offline jobs by holding all experts resident. FluxMoE pages experts on demand through a bandwidth-balanced hierarchy of compressed GPU memory and host DRAM. On GLM-4.5 it reaches up to 7.2 times the throughput of vLLM with no measurable quality loss.
Catalog record
| CREATOR | Qingxiu Liu 1 , Cyril Y. He 2∗ , Hanser Jiang 2 , Zion Wang 2 , Alan Zhao 2 , Patrick P. C. Lee 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63861 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02715
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving 361 Conference: ; ; Thanks: ∗ Corresponding author. Qingxiu Liu 1 , Cyril Y. He 2∗ , Hanser Jiang 2 , Zion Wang 2 , Alan Zhao 2 , Patrick P. C. Lee 1 Abstract . Mixture-of-Experts (MoE) models have become a dominant paradigm for scaling large language models, but their rapidly growing parameter sizes introduce a fundamental inefficiency during inference: most expert weights remain idle in GPU memory while competing with performance-critical runtime state such as the key-value (KV) cache. Since KV cache capacity directly determines serving throughput, this mismatch leads to underutilized memory and degraded performance. In this paper, we present FluxMoE, a new MoE inference system that decouples expert parameters from persistent GPU residency. FluxMoE introduces an expert paging abstraction that treats expert weights as streamed, transient resources, materializing them on demand and evicting them immediately after use, allowing GPU memory to be preferentially allocated to throughput-critical runtime state. We implement FluxMoE atop vLLM to enable efficient MoE inference under severe memory constraints. Experimental results demonstrate that FluxMoE achieves up to 3.0 × \times throughput gains over vLLM in memory-intensive regimes, without compromising model fidelity. 1. Introduction Mixture-of-Experts (MoE) architectures have become popular for scaling large language models (LLMs) ( 44 ; 52 ; 11 ; 7 ; 1 ) . By replacing dense feed-forward layers with large pools of expert networks, MoE models increase model capacity without proportionally increasing per-token computation. Modern MoE models contain hundreds of experts per layer across tens of layers, totaling hundreds of gigabytes or even terabytes of parameters. This capacity, however, comes with an inherent inefficiency: each expert’s weights are accessed only when its corresponding layer is executed; otherwise, they sit idle in GPU memory, consuming space proportional to the full model rather than to the active computation. This idleness directly harms inference performance. In modern LLM serving systems, the key-value (KV) cache capacity determines throughput ( 28 ) : the KV cache grows with batch size and context length, both of which increase memory demands, while larger batch sizes (when memory permits) increase the number of tokens processed per second. Maximizing the GPU memory available for the KV cache is hence critical for high-performance inference. However, existing systems ( 28 ; 60 ) treat model parameters as persistent, GPU-resident state that must remain loaded throughout the serving session , leaving the KV cache to compete for the remaining GPU memory after weights are allocated. For MoE models, this tension is severe. For example, DeepSeek-R1 ( 7 ) contains 671B parameters. Using FP8 precision, its weights occupy 671 GB (i.e., one byte per parameter), requiring at least 9 and typically 16 NVIDIA H100 GPUs (80 GB each) due to parallelism constraints and occupying roughly \tfrac{671}{80\times 16}\approx 52\% of total aggregate GPU memory. The vast majority of these weights are idle at any given decoding step: only the experts of the layer currently executed are in active use, yet all weights remain pinned in GPU memory throughout the entire inference session. Consequently, GPU memory is largely dominated by infrequently accessed model parameters, while the KV cache, whose capacity directly determines serving performance, is severely constrained. This mismatch suggests a fundamental reconceptualization of how MoE parameters should be managed. Rather than treating expert weights as persistent GPU-resident state, they can be viewed as streamed parameters , which are materialized into GPU memory only when their layer is scheduled for execution and released immediately after use. This leads to a new execution model for MoE inference: (1) \emph{model=computegraph+streamedparameters.} The MoE compute graph is static and unchanged; only the parameters fed into it are dynamic. Under this model, GPU memory is reserved primarily for performance-critical runtime state (i.e., the KV cache and activation buffers), while the full expert pool resides in a decoupled storage hierarchy and is streamed on demand. We formalize this idea as expert paging , an abstraction that decouples the logical identity of each expert from its physical GPU residency, treating expert parameters as transient resources that are paged in right before their layer is executed and evicted immediately after use. By pipelining expert transfers with layer computation, expert paging continuously overlaps parameter I/O with the forward pass, so as to keep the GPU busy while holding only a small working set of experts in physical GPU memory at any time. Building on expert paging, we design and implement FluxMoE, an MoE inference system that dynamically materializes expert parameters while remaining fully compatible with existing inference frameworks. FluxMoE adopts three tightly integrated mechanisms. ( i ) PagedTensor provides a tensor virtualization abstraction that decouples the logical identity of expert tensors from their physical GPU allocation. It reserves stable virtual addresses for all expert tensors and dynamically binds physical memory blocks to the virtual addresses of the necessary expert tensors, without requiring modifications to compute kernels (e.g., PyTorch ( 36 ) and Triton ( 47 ) ). ( ii ) The expert storage hierarchy organizes the full expert pool across compressed GPU memory and host DRAM, partitioning expert parameters across different storage backends in proportion to their bandwidth to maximize the aggregate loading rate. ( iii ) The budget-aware residency planner forms a closed-loop controller that continuously adjusts the number of expert tensors remaining in GPU memory, prioritizing the KV cache and activation buffers, and reducing expert residency as GPU memory pressure increases. In summary, this paper makes three contributions: \bullet We formalize expert paging, a new execution model that decouples the logical identity of experts from their physical GPU residency. By treating expert parameters as transient and streamed resources, we reclaim GPU memory for throughput-critical runtime state (e.g., KV cache). \bullet We design FluxMoE, which integrates three key mechanisms: PagedTensor for virtualizing tensor addresses, a bandwidth-balanced storage hierarchy that co-optimizes loading rates across compressed GPU memory and host DRAM, and a budget-aware residency planner that dynamically adjusts expert residency under memory pressure. \bullet We implement FluxMoE atop vLLM (v0.10.2) ( 28 ) and evaluate it on two state-of-the-art MoE models. In memory-intensive scenarios characterized by large batch sizes and extended context lengths, FluxMoE achieves up to 3.0 \times throughput gains over vLLM on Qwen3-Next-80B-A3B-Instruct ( 52 ) (at a batch size of 256 and context length of 4,096 tokens) without accuracy loss. 2. MoE Inference Memory Characterization We characterize the memory bottlenecks of large-scale MoE inference by profiling representative models ( 52 ; 25 ) under realistic serving conditions. We deploy models using vLLM ( 28 ) (see the testbed details in § 6.1 ), with tensor parallelism (TP) at degree 4 (i.e., a model is partitioned across a single node of four GPUs) and a batch size range of 32–256 at context lengths up to 4,096 tokens. Our analysis reveals two observations that motivate FluxMoE’s design. First, MoE models impose a disproportionately large GPU memory footprint compared to equivalent-quality dense models. Second, KV cache capacity directly governs inference throughput, yet it is tightly constrained by the GPU memory already occupied by model parameters. 2.1. Memory Footprint of MoE Models MoE architectures expand model capacity by substituting dense feed-forward layers with large pools of expert networks. This scaling comes at a steep memory cost: