DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng · 2026
Read DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge by Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19172.txt.
Introduction
Huang and co-authors speed up mixture-of-experts models on small edge devices, where memory and data flow are tight. Their DyMoE method sets the numeric precision of each expert on the fly. It favors key experts, guards deep layers that carry more meaning, and loads experts ahead of time to hide wait. On real edge hardware, first-token wait drops by up to 22.7 times. Speed per token rises by up to 14.58 times. Accuracy stays close to the full model.
Catalog record
| CREATOR | Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34196 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng Abstract Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44 × 3.44\times – 22.7 × \times and up to a 14.58 × 14.58\times speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices. 1 Introduction Large Language Models (LLMs) are increasingly transitioning from centralized cloud services to local deployment on resource-constrained edge platforms, driven by imperatives such as data privacy, zero-latency availability, and inference cost sustainability ( 27 ; 29 ; 22 ) . Simultaneously, the Mixture-of-Experts (MoE) ( 19 ) architecture has emerged as the dominant paradigm for scaling LLMs by leveraging sparse activation to decouple model capacity from computational cost. By dynamically routing tokens to a sparse subset of experts, MoE delivers massive-model reasoning capabilities with significantly lower FLOPs than dense models of equivalent scale. Although originally designed for cloud-scale efficiency, this sparse, low-compute characteristic incidentally makes high-performance inference theoretically feasible on hardware-limited edge devices. Figure 1 : Pipeline Comparison: DyMoE vs. Two Conventional MoE Baselines. However, despite their computational efficiency, deploying high-performance MoE models on edge devices faces a daunting storage and bandwidth wall. The massive inactive parameter set creates a footprint that far exceeds the physical memory of typical edge hardware; for instance, Mixtral-8 \times 7B requires approximately 87 GB in BF16 format, whereas consumer laptops or embedded AI accelerators often possess significantly less memory. To mitigate this, offloading inactive experts to host memory is a primary strategy. However, as illustrated in Figure 1 , a naive load-on-demand approach incurs prohibitive latency. Even with prefetching ( 26 ; 20 ; 32 ; 6 ) , the system still suffers from substantial I/O bubbles, as the time to load massive expert weights typically dwarfs the narrow computation window, leaving the GPU idle in Wait-for-Weight stalls. To alleviate this bandwidth pressure, existing optimizations focus on reducing data volume through weight compression or expert skipping. However, these methods often suffer from a lack of fine-grained adaptivity to both the model’s intrinsic structure and the dynamic nature of inference. Uniform quantization ( 8 ; 16 ; 2 ; 21 ) treats all parameters with equal importance, disregarding their varying sensitivity across layers and leading to disproportionate accuracy loss at extreme bit-widths (e.g., Int2). Meanwhile, static mixed-precision ( 26 ; 32 ) and dynamic skipping frameworks ( 30 ; 20 ; 17 ) remain constrained by offline-derived statistics or pre-determined thresholds. Because these strategies are frozen prior to deployment, they are incapable of adapting to the fluid dynamics of real-time input. The limitations of static approaches motivate a more granular exploration of MoE inference dynamics. In this work, we identify three synergistic properties that bridge algorithmic sparsity with potential system-level efficiency: (1) Dynamic Skewness: expert importance is primarily driven by a small subset of highly influential tokens—often referred to as heavy-hitters—whose activation patterns vary significantly across inputs, suggesting that uniform expert treatment is inherently sub-optimal; (2) Depth-Dependent Sensitivity: model layers exhibit non-uniform tolerance to information loss, with deeper layers demonstrating significantly higher quantization robustness; and (3) Inter-layer Predictability: the inherent activation similarity across adjacent layers enables the accurate look-ahead identification of critical experts for subsequent stages. Guided by these insights, we propose DyMoE, an algorithm-system co-designed MoE inference framework that requires zero re-training or calibration overhead. DyMoE introduces a dynamic precision scheduler that assigns experts to a spectrum of mixed-precision states (e.g., 8-bit, 4-bit, and “0-bit”) based on their runtime importance. Under this unified representation, a 0-bit assignment corresponds to expert skipping, effectively eliminating both memory I/O overhead and computational costs for redundant parameters. To mitigate the I/O bottlenecks inherent in edge devices, DyMoE leverages inter-layer predictability to implement a look-ahead prefetching mechanism, which overlaps expert loading with active computation. As illustrated in Figure 1 , by dynamically modulating the precision of expert weights, DyMoE significantly accelerates inference throughput on resource-constrained commodity hardware while maintaining competitive model performance. Our specific contributions are as follows: • Dynamic Expert Importance Classification Method: We propose a runtime-adaptive scheme that classifies experts into Critical and Sub-critical tiers based on heavy-hitter tokens and gate score. By incorporating depth-dependent sensitivity, our method ensures that limited resources are strictly prioritized for experts most vital to model performance. • A High-Performance Inference System: Leveraging the proposed classification, we implement a Dynamic Mixed-Precision Expert Orchestration system that dynamically assigns experts to optimal execution paths. Specifically, sub-critical experts are transitioned into lower-precision quantization to minimize resource consumption. This system is powered by two synergistic components: (i) a Look-ahead Prefetching Engine that exploits inter-layer predictability to prefetch critical weights, effectively overlapping I/O with computation; and (ii) Mixed-precision Cache Management that orchestrates limited VRAM. • Extensive Empirical Validation: We evaluate DyMoE on representative MoE architectures (e.g., Mixtral-8 \times 7B ( 12 ) , Qwen3-30B-A3B ( 24 ) ) across diverse edge-side memory constraints (12 – 24 GB). Experimental results demonstrate that DyMoE achieves a 3.44\times – 22.7 \times reduction in TTFT and up to a 14.58\times speedup in Time-Per-Output-Token latency (TPOT) compared to state-of-the-art offloading baselines. Crucially, these gains are achieved with marginal accuracy degradation. 2 Background and Related Work 2.1 Mixture-of-Experts Architecture (a) MoE Architecture (b) Memory Demands of SOTA MoEs Figure 2 : Overview of MoE Structure and its Memory Demands. The Mixture-of-Experts (MoE) ( 19 ) architecture scales model capacity by replacing dense Feed-Forward Networks (FFNs) with a sparse layer comprising multiple independent experts and a gating network ( 2(a) ). During inference, the router selects a small subset of experts per token, effectively maintaining a constant computational cost (FLOPs) despite a massive parameter footprint. However, this architectural efficiency in computation does not translate to memory savings. As illustrated in 2(b) , modern MoE models, such as Mixtral- 8\times 7B and Qwen3-30B-A3B, possess parameter footprints that far exceed the VRAM capacity of common edge hardware (e.g., 12 GB, 16 GB, or 24 GB). Paradoxically, while these models require prohibitive storage, their runtime utilization is remarkably sparse: Mixtral- 8\times 7B activates only \sim 27% of its parameters per token, while Qwen3-30B-A3B activates as little as 10%. This vast memory-utilization gap—where 70–90% of parameters remain idle at any given step—motivates our pursuit of a runtime-adaptive system. 2.2 MoE Compression and Offloading To alleviate the severe memory bottlenecks, researchers have explored various model compression methods and system-level optimization techniques. Quantization and Pruning. Standard Post-Training Quantization (PTQ) methods like GPTQ ( 8 ) and AWQ ( 16 ) apply uniform bit-widths to reduce