ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference

Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang, Yusen Zhang, Liang Wang, Limin Xiao · 2026

Read ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference by Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang, Yusen Zhang, Liang Wang, Limin Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27081.txt.

Introduction

Zhu and five co-authors present ReMoE, a tuning method that boosts expert reuse in memory-constrained mixture-of-experts inference. Since few experts fit in cache, missing experts must be fetched from slow storage, causing evictions and I/O overhead. ReMoE biases routing toward selected experts, producing stable routing that matches cache locality, and it cuts expert fetches without adding inference-time compute. Tests on DeepSeek and Qwen models lift expert reuse by 26 percent while holding performance, and real trials raise throughput and speed decoding.

Catalog record

CREATORXiongwei Zhu, Xiaojian Liao, Tianyang Jiang, Yusen Zhang, Liang Wang, Limin Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84687
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang, Yusen Zhang, Liang Wang, Limin Xiao Abstract Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in memory-constrained inference scenarios, only a small set of experts can be cached. Experts not in the cache must be fetched from slow external storage (e.g., UFS), leading to frequent evictions and substantial I/O overhead. We propose ReMoE, a router fine-tuning framework designed to boost token-wise expert reuse. ReMoE biases the router toward recently selected experts, producing temporally stable routing that better matches cache locality constraints. By increasing short-horizon expert reuse, ReMoE reduces expert fetches from storage without adding inference-time computation. Experiments on DeepSeek and Qwen models show that ReMoE improves expert reuse by 26% while maintaining downstream task performance. Real-system evaluations further confirm these benefits, improving output throughput by 8.4% under vLLM GPU–CPU expert offloading and reducing TPOT by 43.6–49.8% under llama.cpp on Jetson Orin NX, corresponding to a 1.77–1.99 × \times decode speedup across diverse workloads. Checkpoints and usage instructions are available at https://github.com/BUAA-OSCAR/ReMoE . 1 Introduction Edge-side deployment of Large Language Models (LLMs) is an emerging trend ( 62 ; 68 ; 58 ) , with on-device AI applications such as real-time translation and advanced photo editing already demonstrating its practical value ( 64 ) . While Mixture-of-Experts (MoE) architectures are central to scaling LLM capacity, their deployment has so far remained predominantly cloud-based, with limited use on edge devices. Recent efforts on on-device MoE models, edge-oriented inference runtimes, and mobile MoE applications suggest that this situation is beginning to change, making MoE increasingly relevant to edge and memory-constrained deployments ( 43 ; 41 ; 42 ; 39 ; 1 ; 23 ) . This trend is supported by two factors. First, MoE’s sparse activation keeps only a subset of experts active during inference, preserving large model capacity while limiting the activated parameter footprint. Second, advances in mobile storage make local weight storage increasingly feasible. For example, Samsung’s UFS 4.0 offers read speeds up to 4 GB/s and capacities up to 1 TB, making local storage of large model weights increasingly feasible ( 47 ) . However, deploying MoE LLMs on edge devices introduces the challenge of frequent expert switching. During the decoding phase, where each token may activate a different set of experts, this results in frequent, irregular I/O requests to load the required expert weights from storage, prolonging the inference latency ( 44 ) . Current solutions typically attempt to hide this latency through system-level techniques such as prefetching or caching algorithms. A key upstream factor is the routing trace produced by the MoE router, i.e., the sequence of selected expert sets \{E_{t}\}_{t=1}^{T} across decoding steps. Many standard MoE training recipes include load-balancing objectives that spread tokens across experts to improve training-time utilization under expert parallelism. While useful for large-scale training, such dispersion can be misaligned with memory-constrained single-request decoding, where inference benefits when adjacent tokens reuse part of the same expert working set. ReMoE addresses this training–deployment mismatch by shaping the routing trace itself, complementing runtime caching and prefetching. Figure 1 : Comparison of expert I/O dynamics. Baseline: Standard routers select disjoint experts across steps, causing frequent cache replacements and I/O thrashing. ReMoE: Our method encourages temporal locality by increasing expert reuse across adjacent steps, thereby reducing cache turnover and I/O overhead. To bridge this gap, we propose ReMoE, a lightweight router fine-tuning framework that aligns routing behavior with memory capacity constraints on edge devices. ReMoE adapts the router using two complementary objectives: (i) a temporal locality loss that encourages expert reuse across adjacent tokens, and (ii) a Trust-KL loss that softly anchors the updated routing distribution to the pretrained router. This biases the router toward temporally stable reuse while constraining distributional drift, transforming fragmented routing traces into more cache-friendly sequences. Crucially, ReMoE does not modify the model architecture, hardware, or inference kernels, and adds no runtime policy beyond using the fine-tuned router weights. We evaluate ReMoE across fine-grained MoE models and heterogeneous serving platforms. ReMoE increases expert overlap by 26.4% on DeepSeek-V2-Lite and by 27.2% on Qwen1.5-MoE-A2.7B. We further perform trace-driven offline cache analysis using unique hit rate (uHR), the cache-hit fraction over distinct expert requests, and total unique misses (#uMiss), the number of distinct expert loads. ReMoE improves uHR and reduces #uMiss across cache capacities and replacement policies. In real-system evaluations, ReMoE improves output throughput by 8.4% and reduces TPOT by 4.5% under vLLM GPU–CPU expert offloading. On Jetson Orin NX, where expert misses are more expensive under SSD-backed edge inference, ReMoE reduces TPOT by 43.6–49.8% across ShareGPT, GSM8K, and HumanEval, corresponding to a 1.77–1.99 \times decode speedup. 2 Related Work MoE routing, load balancing, and locality. Mixture-of-Experts (MoE) scales model capacity by activating only a few experts per token ( 49 ; 32 ; 18 ; 15 ) . Most MoE models use Top- K token-choice routing with auxiliary balancing losses to prevent collapse and improve utilization ( 32 ; 18 ) . Recent work explores alternative routing objectives and mechanisms, such as router z-loss for stability ( 72 ) , expert-choice routing for better balance and efficiency ( 71 ) , and auxiliary-loss-free balancing strategies ( 57 ) . Prior analyses study when learning to route helps and how routing variants affect quality ( 13 ) , with surveys summarizing modern MoE routing and training practices ( 4 ) . Fine-grained MoEs, including DeepSeek-V2/V3 and Qwen MoE, increase specialization but also amplify token-wise expert switching ( 10 ; 45 ; 11 ; 65 ) . Oracle-MoE addresses the resulting locality problem by redesigning the routing architecture and training from scratch to preserve expert activation consistency ( 70 ) . ReMoE targets the same locality bottleneck from a different angle: rather than modifying the architecture or requiring pretraining, it fine-tunes only the existing gate parameters of an already-trained MoE checkpoint, making it a lightweight post-training adaptation that leaves the model architecture and expert weights unchanged. System support for memory-constrained MoE inference. System efficiency for MoE inference depends on dispatch, communication, expert parallelism, and expert-weight movement; frameworks such as DeepSpeed-MoE and Tutel/MegaBlocks optimize dispatch and expert-parallel execution costs ( 46 ; 26 ; 20 ; 38 ) . Under memory constraints, MoE offloading systems reduce expert-transfer overhead through caching, offloading, and CPU–GPU orchestration, as shown by MoE-Infinity, HOBBIT, FineMoE, KTransformers, MoE-Lightning, and Fiddler ( 63 ; 54 ; 67 ; 33 ; 6 ; 5 ; 29 ) . CoServe shows that expert-based collaborative inference also suffers from memory-tier switching overhead, motivating dependency-aware scheduling and expert management ( 53 ) . Some cache-aware routing methods, such as Mixture of Cache-Conditional Experts, bias expert selection using cache residency at inference time, directly trading off cache hits and routing choices during decoding ( 52 ) . ReMoE is complementary to these runtime methods: it reshapes the routing trace offline through router fine-tuning, so the deployed model can use the same inference graph and standard cache policies. Although related memory-constrained serving systems have also been studied for dense LLMs ( 51 ; 2 ; 64 ; 28 ;

Read the complete text