D 2 MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving Conference: The 31st Annual International Conference on Mobile Computing and Networking; November 4–8, 2025; Hong Kong, China The 31st Annual International Conference on Mobile Computing and Networking (ACM MO

Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo · 2025

Read D 2 MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving Conference: The 31st Annual International Conference on Mobile Computing and Networking; November 4–8, 2025; Hong Kong, China The 31st Annual International Conference on Mobile Computing and Networking (ACM MO by Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15299.txt.

Introduction

Haodong Wang and co-authors speed up large language models on small devices in a 2025 paper. Their target is the mixture-of-experts model, which holds many expert parts but uses a few per task. Such models still strain phones and edge gear. Past fixes shrink the model once and for all, which loses flex when tasks vary.

Their system, D2MoE, fits the shrink to each task as it runs. A nested shrink method stores expert weights at stacked bit widths, so memory drops. A scheduler loads the hottest experts first and overlaps load with compute to cut idle time. On real edge gear the system raised throughput by up to 1.39 times. Peak memory fell by up to 53 percent, while answer quality held near the full eight-bit form.

Catalog record

CREATORHaodong Wang, Qihua Zhou, Zicong Hong, Song Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D 2 MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving Conference: The 31st Annual International Conference on Mobile Computing and Networking; November 4–8, 2025; Hong Kong, China The 31st Annual International Conference on Mobile Computing and Networking (ACM MOBICOM ’25), November 4–8, 2025, Hong Kong, China DOI: 10.1145/3680207.3723493 ISBN: 979-8-4007-1129-9/2025/11 CCS: Human-centered computing Ubiquitous and mobile computing CCS: Computing methodologies Artificial intelligence Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo Abstract . The mixture of experts (MoE) model is a sparse variant of large language models (LLMs), designed to hold a better balance between intelligent capability and computational overhead. Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices, especially with the demands of on-device inference services. Recent research efforts often apply model compression techniques, such as quantization, pruning and merging, to restrict MoE complexity. Unfortunately, due to their predefined static model optimization strategies, they cannot always achieve the desired quality-overhead trade-off when handling multiple requests, finally degrading the on-device quality of service. These limitations motivate us to propose the D 2 MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert. Specifically, inspired by the nested structure of matryoshka dolls, we propose the matryoshka weight quantization (MWQ) to progressively compress expert weights in a bit-nested manner and reduce the required runtime memory. On top of it, we further optimize the I/O-computation pipeline and design a heuristic scheduling algorithm following our hottest-expert-bit-first (HEBF) principle, which maximizes the expert parallelism between I/O and computation queue under constrained memory budgets, thus significantly reducing the idle temporal bubbles waiting for the experts to load. Evaluations on real edge devices show that D 2 MoE improves the overall inference throughput by up to 1.39 × \times and reduces the peak memory footprint by up to 53 % \% over the latest on-device inference frameworks, while still preserving comparable serving accuracy as its INT8 counterparts. Figure 1 . Traditional MoE single routing (expert ID only) vs. our D 2 MoE dual routing (ID and bit-wdith). 1. INTRODUCTION The Large Language Models (LLMs) ( Touvron et al. 2023 ; Vaswani et al. 2017 ; Zhang et al. 2022 ; Jiang et al. 2024 ) are increasingly embedded in our work and everyday activities, supporting tasks like summarization, code completion, and decision making ( Chen et al. 2021 ; Brown et al. 2020 ; Github 2022 ; OpenAI 2022 ) . To further scale these models without incurring prohibitive training and inference costs, recent research has introduced Mixture-of-Experts (MoE) models, where feedforward network (FFN) layers are often replaced with MoE layers. These MoE layers enable sparse activation for each token, dynamically routing computation to a subset of model components (i.e., experts). As model sizes continue to grow and task complexity increases, the sparse activation architecture of MoE models becomes increasingly critical. However, compared to other LLMs, MoE-based models face more severe memory bottlenecks during inference due to their large parameter sizes. This challenge is particularly pronounced in resource-constrained edge environments. For instance, when stored in a compact float16 format, the parameters of Mixtral 8 \times 7B ( Jiang et al. 2024 ) require over 90GB of memory, whereas consumer-grade GPUs such as the NVIDIA RTX 3060 laptop offer only 6GB of memory, highlighting the resource limitations of edge devices. Quantization is one of the most prominent solutions for memory optimization during LLM inference on edge devices. Applying quantization to MoE models is particularly effective, as experts account for almost 90% of the parameters (e.g., in Mixtral-8 \times 7B, experts contribute 89.9% of the parameters, while attention mechanisms account for less than 11.1%). Consequently, quantizing experts significantly reduce memory usage and the size of parameter transfers between storage hierarchies during inference. Recent efforts have proposed quantization strategies tailored to MoE-based models, where bit-width is assigned to different experts based on their importance on calibration dataset and fix bit-width assignments throughout inference. For instance, EdgeMoE ( Yi et al. 2023 ) assigns bit-width offline by quantifying the accuracy loss of experts on a calibration dataset, while MC-MoE ( Huang et al. 2025 ) uses expert activation frequency and confidence to design a fixed bit-width allocation strategy during inference. However, due to the dynamic sparse activation characteristic of MoE-based models, the importance of an expert can vary with different input tokens. Inspired by related work on dense models ( Raposo et al. 2024a ; Gong et al. 2024 ) , a straightforward approach involves introducing a lightweight, trainable adapter before each transformer block to dynamically decide whether to skip the current block’s computation. This approach significantly reduces overall FLOPs and accelerates inference. Specifically, the adapter dynamically allocates more computational blocks to important tokens while skipping less significant tokens, optimizing computational efficiency. Nevertheless, incorporating dynamic bit-width selection for activated experts inevitably increases memory overhead. This is because the weights for different bit-width in existing quantization methods are stored independently. For instance, in LLaMA-MoE ( Zhu et al. 2024 ) , when quantized using llama.cpp ( Gerganov 2023 ) , INT4 experts require 3.81 GB of memory, while storing INT2/3/4 experts simultaneously demands 9.62 GB. This significantly increases the memory footprint in resource-constrained edge devices. Although parameters of inactive experts can be offloaded to lower-tier storage units (e.g., CPU or SSD), the I/O latency associated with quantized experts remains substantial (e.g., at Mixtral \times 7B, INT4 quantization experts still account for more than 70% of the parameters), leading to significant computational bubbles caused by waiting for I/O. For example, on an NVIDIA RTX 3060, the average computation time for a single expert in LLaMA-MoE is 3.1 ms, whereas the average data transfer time is approximately 20 ms. Existing methods attempt to overlap I/O and computation by processing multiple batches simultaneously ( Guo et al. 2023 ; Cao et al. 2025 ) . However, due to the token variability inherent in gating mechanisms, the total number of activated experts and their associated bit-width can increase significantly. Furthermore, when the gating mechanism selects more than one expert per token (as in Mixtral-8×7B and LLaMA-MoE), the I/O overhead for transferring expert parameters is multiplied, causing GPU to experience more frequent idle periods while waiting for expert parameters. To tackle these challenges, we propose a dually sparsely-gated Mixture-of-Experts paradigm. Specifically recognizing that expert importance varies dynamically with different tokens, we first introduce an end-to-end fine-tuning strategy for the gating network, called token-adaptive bit-width selection . This strategy enables dynamic bit-width decisions for each token, achieving a better trade-off among model accuracy and peak memory usage for MoE-based LLMs. Second, we present matryoshka weight quantization (MWQ), which compresses expert weights into a structure where bit-width can be shared hierarchically. In this approach, higher bit-width always encapsulates lower bit-width, resembling the nested structure of matryoshka dolls. 1 1 1 matryoshka dolls are a set of wooden dolls of decreasing size, nested within one another. Finally, we employ a bit-width-aware I/O-compute pipeline that dynamically reorganizes the I/O and computation order for different bit-width. This pipeline processes multiple requests in batches, optimizing the parallel efficiency of expert I/O and computation. Together, these innovations improve the overall performance and efficiency of MoE-based LLMs,

Read the complete text