FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, Lidan Shou · 2025
Read FloE: On-the-Fly MoE Inference on Memory-constrained GPU by Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, Lidan Shou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05950.txt.
Introduction
Yuxin Zhou and seven co-authors present FloE, a system that runs mixture of experts models on graphics cards with tight memory, by shrinking experts on the fly. Such models pick a few expert parts per input, but the picked parts are large to move. Their work came out in twenty twenty-five.
Offloading experts to main memory and loading them as needed chokes on the narrow bus between them. FloE squeezes the inner matrices of each expert, where much spare fat hides, and adds a cheap sparse forecast of which experts will fire. On a known model it shrank each expert hugely, fit a small card, and ran far faster than a rival, with small score loss.
Catalog record
| CREATOR | Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, Lidan Shou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05950
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, Lidan Shou Abstract With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert’s internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3 × \times compression of parameters per expert in Mixtral-8 × \times 7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5 × \times ; and delivers a 48.7 × \times inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090—all with only a 4.4% ∼ \sim 7.6% average performance degradation. † † affiliationnotice: Equal contribution 1 Introduction Mixture of Experts (MoE) models including DeepSeek-R1 ( DeepSeek-AI, 2025 ) , GPT-4 ( OpenAI, 2023 ) , Phi-4 ( Abdin et al., 2024b ) , Mixtral ( Jiang et al., 2024 ) , etc., offer a paradigm shift in the large language model (LLM) architecture by introducing sparsely activated experts. These sparse LLMs contextually activate only a subset of experts per token, significantly reducing inference costs while maintaining generative performance. However, the abundance of idle, non-activated experts during MoE inference significantly hampers efficient GPU memory utilization, making it challenging to deploy MoE models on memory-constrained GPUs. For instance, running inference for Mixtral-8 \times 7B, where two experts are activated, requires approximately 94GB of VRAM in FP16 precision. Of this, 30% of the activated parameters (27.3GB) are utilized during decoding, while the remaining 66.8GB is occupied by non-activated experts, resulting in significant inefficiency ( Shin et al., 2024 ) . To address the problem, offloading techniques ( Sarkar et al., 2023 ; Eliseev & Mazur, 2023 ; Hwang et al., 2024 ; Song et al., 2024a ; Xue et al., 2024a ; Tang et al., 2024 ) , which unmount expert parameters to CPU memory and load them into GPU memory on demand for each input, offers a natural solution. However, offloading shifts the decoding bottleneck from memory-bound to I/O-bound, as transferring billions of parameters through the low-bandwidth PCIe bus incurs substantial data transfer delays. For comparison, the DRAM-to-VRAM bandwidth (32GB/s for PCIe 4.0) is orders of magnitude lower than the bandwidth between GPU memory and on-chip computation units (300GB/s). Consequently, existing MoE inference systems with expert offloading, designed for edge-side continuous serving scenarios (i.e., single-batch latency-sensitive inference) ( Kong et al., 2024 ; Eliseev & Mazur, 2023 ; Hwang et al., 2024 ; Sarkar et al., 2023 ; Tang et al., 2024 ) , still fail to support on-the-fly inference, where the loading process is perceptible to the user because its overhead cannot be hidden by the model computation. Ultra-low-bit quantization effectively reduces the size of transmitted parameters to mitigate the latency of activated expert loading ( Eliseev & Mazur, 2023 ; Sarkar et al., 2023 ) , but at the cost of significantly degraded generation performance. Thus, a pressing question emerges: How can we hide the I/O overhead of activated experts within model computation to enable on-the-fly MoE inference on the memory-constrained GPU while minimizing generation performance degradation? In this paper, we present an on-the- fl y M oE inference system, coined FloE , for consumer-grade devices. FloE reduces the I/O overhead of the experts, namely the transfer cost of the matrices for up, gate, and down projections, via a hybrid compression mechanism ( Section 3.2 ). Despite the utilization of the well-known inter-expert sparsity, the compression exploits the vast, untapped intra-expert sparsity in MoE models with a novel contextual sparsification scheme ( Section 3.2.1 ), balancing the transfer cost and the downstream performance. In detail, the system first identifies low-magnitude, no-salient output activations of up projection and then removes the corresponding channel weights from the gate and down projections. Meanwhile, we observe that the up projection matrix has limited sensitivity on performance against quantization, motivating us to enable the ultra-low-bit quantization in FloE to reduce the transfer overhead further ( Section 3.2.2 ). Although the hybrid compression reduces per-transfer cost for MoE models, the pipelining between transfer and computation is prevented due to the sequential execution of routing, quantized up projection computation, and DRAM expert fetching, inhibiting on-the-fly inference. Therefore, we investigate the weights and the input during the computation and locate a high similarity between the shared hidden state input before routing and up projection of the MoE model. Based on the finding, we devise two efficient yet effective sparsity predictors: (1) an inter-expert learning-based predictor to guide the routing of the activation expert of the next layer with the hidden state of the current layer; (2) an intra-expert reused-based predictor precomputing the context sparsity distribution with the hidden state of the current layer and the reused up projection. The two predictors, with the help of prefetching, enable the pipelining of transfer and computation for on-the-fly inference. ( Section 3.3 ) To integrate all the techniques above, we at last propose an efficient sparse kernel and compact asynchronous transfer from DRAM to VRAM to achieve system-wide efficiency ( Section 3.4 ). The experimental study on various GPU specs and downstream tasks evidence the efficiency and efficacy of FloE ( Section 4 ). Notably, for the popular Mixtral-8 \times 7B, FloE achieves 9.3 \times parameter compression per expert, enables deployment on a GPU with just 11GB VRAM , and delivers a 2.6 \times inference speedup on an RTX 3090, with only 4.4% \sim 7.6% average performance degradation. Figure 1: Comparison of MoE inference offloading systems for memory-constrained GPUs: (a) Naive MoE Inference with Offloading; (b) Advanced MoE Inference with Offloading; (c) On-the-Fly MoE Inference (FloE). 2 Related Work Experts Offloading. Efficient deployment of MoE models faces substantial challenges due to their parameter counts, particularly under resource constraints. Current inference frameworks like Llama.cpp ( llama.cpp, ) , HuggingFace Accelerate ( Gugger et al., 2022 ) , and DeepSpeed Inference ( Aminabadi et al., 2022 ) employ experts offloading by selectively transferring VRAM-dominant expert weights to DRAM ( Sheng et al., 2023 ) . However, constrained PCIe bandwidth creates transfer bottlenecks during CPU-GPU expert transfers ( Kamahori et al., 2024 ) . To mitigate this, prefetching strategies predict and preload required experts through two paradigms: experience-based statistical methods using offline activation traces ( Sarkar et al., 2023 ) (limited to top-1 expert activation strategy ( Fedus et al., 2022 ) ), and intermediate result-driven approaches leveraging hidden states ( Eliseev & Mazur, 2023 ; Hwang et al., 2024 ; Song et al., 2024a ; Tang et al., 2024 ) or prior expert indices ( Xue et al., 2024a ) . The former fails under multi-expert activation due