MobileMoE: Scaling On-Device Mixture of Experts

Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi · 2026

Read MobileMoE: Scaling On-Device Mixture of Experts by Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27358.txt.

Introduction

Yanbei Chen and colleagues present MobileMoE, a family of on-device mixture-of-experts models under a billion active parameters. A scaling law tuned to mobile memory and compute picks moderate sparsity with fine-grained shared experts. The models match or beat leading dense on-device rivals with far fewer inference FLOPs and up to 60 percent fewer parameters.

Catalog record

CREATORYanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101193
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MobileMoE: Scaling On-Device Mixture of Experts Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi Abstract Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE , a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot – moderate sparsity with fine-grained and shared experts – that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4 × \times fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers 1.8 1.8 - 3.8 × 3.8\times faster prefill and 2.2 2.2 - 3.4 × 3.4\times faster decode than the dense baseline MobileLLM-Pro. † † affiliation: Meta AI † † date: August 10, 2026 † † correspondence: Yanbei Chen at [email protected] Detailed author contributions can be found in the Author Contributions section. 1 Introduction Mixture-of-Experts (MoE) architectures increasingly dominate state-of-the-art Large Language Models (LLMs), as represented by both open-source models (e.g., DeepSeek V3 Liu et al. 2024a , Qwen3 MoE Yang et al. 2025 ) and proprietary frontier models (e.g., Gemini Team et al. 2024 , Grok xAI 2024 ). However, on-device LLMs remain overwhelmingly dense (e.g., MobileLLM Liu et al. 2024b , MobileLLM Pro Huber et al. 2025 ), and scaling MoE in the sub-billion active-parameter regime, where on-device LLMs typically operate, remains largely unexplored. Addressing this gap is increasingly crucial for next-generation edge AI: efficient on-device LLMs reduce reliance on cloud compute and enable low-latency, cost-effective, privacy-preserving applications on smartphones, wearables, and embodied agents. Unlocking the potential of LLMs on edge devices requires overcoming severe compute and memory constraints. MoE architectures address these constraints through three complementary efficiencies. First, parameter efficiency : an MoE model expands total capacity through many expert networks while activating only a sparse fraction per token, matching the performance of a much larger dense counterpart at significantly less inference compute Yang et al. 2025 . Second, runtime efficiency : sparse activation reduces inference FLOPs, lowering runtime latency and conserving mobile battery life. Third, learning efficiency : expert networks specialize across distinct domains (e.g., knowledge, code, math) Shazeer et al. 2017 , packing broad multi-task capability into one unified model. Crucially, the recent growth of smartphone DRAM in the past few years (e.g., from 4 GB on iPhone 13 to 12 GB on iPhone 17, from 8 GB on Samsung Galaxy S21 to 12 GB, 16 GB on S25 and S25 Ultra) provides the memory headroom to host these efficient and capable sparse LLMs directly on mobile devices. Yet the scaling methodology of on-device MoE, from architectures to training recipes and practical on-device deployment, has yet to be established. While scaling laws have long served as the north star guiding the development of dense LLMs Kaplan et al. 2020 ; Hoffmann et al. 2022 and MoEs Clark et al. 2022 ; Krajewski et al. 2024 , existing frameworks overwhelmingly focus on scaling models up to tens or hundreds of billions of parameters for deployment on cloud servers. To address practical edge constraints, we formulate a novel MoE scaling law tailored to the sub-billion active-parameter regime, providing a principled foundation to guide architectural design under joint memory and compute constraints. Building upon this scaling law, we derive MobileMoE , the first sub-billion-active MoE family optimized for the edge across three scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total) with < 3 GB INT4 weight footprints to fit in mobile DRAM. To realize the MoE architectural advantages at scale, we design a comprehensive four-stage training recipe: pre-training, mid-training, instruction fine-tuning, and 4-bit quantization-aware training. Our pipeline explicitly addresses MoE-specific training stability and efficiency, and scales up MobileMoE training with exceptional token efficiency. With only \sim 6T pre-training tokens, MobileMoE matches or surpasses dense baselines trained on 1.5-2 \times more tokens (e.g., 9T for Llama 3.2 1B Grattafiori et al. 2024 , 11T for SmolLM2 1.7B Lozhkov et al. ), validating the learning efficiency of MoE at the sub-billion active scale. Notably, our scaling-law-derived architecture and training recipe enable MobileMoE to establish a new Pareto frontier for on-device LLMs across 14 foundational benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning (Figure 2 ). Smaller MobileMoE-S/M match or exceed dense baselines using 2-4 \times fewer inference FLOPs at comparable memory, while MobileMoE-L pushes the frontier further to state-of-the-art accuracy at sub-billion active scale. Furthermore, compared to the state-of-the-art MoE OLMoE-1B-7B Muennighoff et al. 2024 , MobileMoE-M matches its accuracy with \sim 60% fewer active and total parameters, while MobileMoE-L achieves much higher accuracy with 30% fewer active parameters and 23% smaller model memory footprint. (a) (b) Figure 1: MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute F_{\text{inf}}=2N_{\text{act}} (GFLOPs) and (b) total parameters N_{\text{total}} (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB) 2 2 2 Accuracy is reported from public model releases at full precision (BF16). In subplot (b), the x-axis shows total parameters N_{\text{total}} in billions alongside the corresponding projected INT4 weight memory, computed as \mathcal{M}_{\text{weight}}=\tfrac{1}{2}N_{\text{total}} GB under 4-bit weight quantization that is commonly used for on-device LLM deployment (as detailed in Section 3.4 , Section 4.3 ). . Beyond benchmark performance, we demonstrate the practical on-device runtime benefits by deploying MobileMoE on flagship smartphones: Samsung Galaxy S25 and iPhone 16 Pro. Since most existing mobile inference stacks lack native MoE support, we develop a custom fused MoE kernel to enable efficient MoE inference, providing the first MoE runtime support on commodity smartphone CPUs, with comprehensive runtime profiling across CPU and GPU backends. Powered by this kernel, at comparable INT4 weight memory, MobileMoE-S achieves 1.8 – 3.8\times faster prefill and 2.2 – 3.4\times faster decode than the dense baseline MobileLLM-Pro Huber et al. 2025 while consuming up to 22% less peak RSS at 8k context. Concurrently, MobileMoE-M matches or outperforms MobileLLM-Pro on runtime with higher accuracy, while MobileMoE-L delivers substantially higher accuracy with moderate runtime cost. The consistency of this Pareto pattern on mobile devices confirms the compute and memory efficiency of MobileMoE holds on real hardware. Our contributions are three-fold: 1. We introduce MobileMoE-S/M/L, the first sub-billion-active MoE family for on-device deployment, derived based on a generalized on-device MoE scaling law under joint memory and compute constraints. Guided by this scaling law, we identify the sweet-spot MoE design choices for on-device use cases (moderate sparsity, fine-grained granularity, and a shared expert) that together define MobileMoE. 2. We

Read the complete text