Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
Songchen Ma 1,2† , Hongyi Li 2† , Weihao Zhang 1,2∗ , Yonghao Tan 1,2 , Pingcheng Dong 1,2 , Yu Liu 1 , Lan Liu 3 , Yuzhong Jiao 1 , Xuejiao Liu 1 , Luhong Liang 1 , Kwang-Ting Cheng 1,2∗ · 2026
Read Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling by Songchen Ma 1,2† , Hongyi Li 2† , Weihao Zhang 1,2∗ , Yonghao Tan 1,2 , Pingcheng Dong 1,2 , Yu Liu 1 , Lan Liu 3 , Yuzhong Jiao 1 , Xuejiao Liu 1 , Luhong Liang 1 , Kwang-Ting Cheng 1,2∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27624.txt.
Introduction
Ma and co-authors speed up mixture-of-experts inference on chiplet hardware. Edge use means small batches and tight on-chip memory. Expert loads also swing sharply at runtime. Their FSE-DP plan shards experts fully across chiplets. Fine streams of expert work follow dynamic paths. High die-to-die bandwidth carries those streams. Simple virtualization rules tame the dataflow. A light scheduler keeps loads balanced. Gains reach about 1.22 times in their tests.
Catalog record
| CREATOR | Songchen Ma 1,2† , Hongyi Li 2† , Weihao Zhang 1,2∗ , Yonghao Tan 1,2 , Pingcheng Dong 1,2 , Yu Liu 1 , Lan Liu 3 , Yuzhong Jiao 1 , Xuejiao Liu 1 , Luhong Liang 1 , Kwang-Ting Cheng 1,2∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66518 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27624
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling Songchen Ma 1,2† , Hongyi Li 2† , Weihao Zhang 1,2∗ , Yonghao Tan 1,2 , Pingcheng Dong 1,2 , Yu Liu 1 , Lan Liu 3 , Yuzhong Jiao 1 , Xuejiao Liu 1 , Luhong Liang 1 , Kwang-Ting Cheng 1,2∗ Abstract Mixture-of-Experts (MoE) is a promising approach for edge AI with low-batch inference. Yet, on-device deployments often face limited on-chip memory and severe workload imbalance; the prevalent use of offloading further incurs off-chip memory access bottlenecks. Moreover, MoE sparsity and dynamic gating shift distributed strategies toward much finer granularity and introduce runtime scheduling considerations. Recently, high die-to-die (D2D) bandwidth chiplet interconnects create new opportunities for multi-chiplet systems to address workload imbalance and offloading bottlenecks with fine-grained scheduling. In this paper, we propose Fully Sharded Expert Data-parallelism (FSE-DP), a parallelization paradigm specifically architected for low-batch MoE inference on multi-chiplet accelerators. FSE-DP attains adaptive computation–communication overlap and balanced load by orchestrating fine-grained, complementary expert streams along dynamic trajectories across high-bandwidth D2D links. The attendant dataflow complexity is tamed by a minimal, hardware-amenable set of virtualization rules and a lightweight scheduling algorithm. Our approach achieves 1.22-2.00 × \times speedup over state-of-the-art baselines and saves up to 78.8% on-chip memory. I Introduction The increasing demand for real-time, privacy-preserving AI services is driving the deployment of Large Language Models (LLMs) onto edge devices. To meet the escalating resource requirements of on-device scenarios such as AI PCs, robotics, and autonomous driving, chiplet-based multi-chiplet accelerators are emerging as a more scalable and cost-effective solution than monolithic designs [ 80 ] . A multi-chiplet package designed for on-device AI typically integrates multiple accelerator dies interconnected by high-bandwidth links and is usually coupled with large off-package memory, such as DRAM [ 52 , 61 , 33 ] (Figure 1 (a)). Concurrently, the Mixture-of-Experts (MoE) architecture has gained prominence for its ability to reduce the number of parameters activated per inference while maintaining a large overall model capacity [ 11 , 22 , 34 , 75 , 72 , 39 ] (Figure 1 (b)). The combination of multi-chiplet and MoE presents a powerful paradigm for high-performance on-device AI [ 28 ] , where the sparsely activated large model can be distributed across chiplets to leverage spatial parallelism and high-speed inter-die communication. Despite this promising synergy, effectively deploying MoE models on multi-chiplet packages, particularly in on-device scenarios characterized by low batch sizes, introduces significant challenges. First, limited on-chip memory : Although chiplet technologies significantly enhance the scalability of computational resources, on-chip SRAM-based cache remains a critical asset for on-device systems relative to LLM capacity [ 59 ] . Moreover, even devices with large memory dies or GPUs equipped with high-bandwidth HBM still widely adopt off-loading strategies in edge deployments [ 9 , 63 , 10 , 81 , 3 ] , owing to the persistent demand for ever-larger models. It is therefore essential to minimize redundant storage and maximize memory efficiency [ 68 , 25 ] . Second, external memory access bottleneck : Constrained on-chip capacity forces edge systems to off-load models, necessitating frequent, high-volume off-chip traffic for both KV caches and expert weights. The reduced weight reuse in low-batch scenarios further exacerbates this issue. Third, dynamic workload imbalance : During each forward pass, the number of tokens assigned to each expert varies—some experts process many tokens (hot experts), while others handle few or none (cold experts) [ 27 , 25 , 10 , 85 ] . This long-tail distribution introduces two aspects of workload imbalance: (1) the compute-to-data-transfer ratio differs across experts, posing additional obstacles to overlapping off-chip memory access with computation; (2) the storage footprint and compute load diverge among chiplets, reducing overall utilization. Furthermore, low-batch conditions render traditional expert-balancing training [ 11 , 71 ] or elastic containers [ 60 , 38 ] used in the cloud ineffective. These three challenges are tightly coupled with the characteristics of multi-chiplet architectures, and the latter two are further amplified by low-batch, on-device MoE workloads. Fig. 1: Typical template of (a) multi-chiplet-based AI accelerator. (b) Mixture-of-Experts network. Commonly-adopted parallel strategies—such as expert parallelism (EP) [ 20 , 34 , 73 ] or hybrids that combine data parallelism (DP) [ 16 ] , tensor parallelism (TP) [ 56 , 14 , 19 ] , and pipeline parallelism (PP) [ 4 ] —fail to adequately address the aforementioned challenges in multi-chiplet inference settings. Most prior optimizations for edge devices target GPUs and seldom exploit the distinctive characteristics of chiplet-based packages. Recently, several methods specialized for MoE inference on multi-chiplet have been proposed [ 17 , 78 ] . These approaches generally aim to optimize inter-die communication, especially all-to-all communications in MoE, but they place less emphasis on addressing the long-tail issue and the external memory access bottleneck. Propelled by rapid advancements in advanced packaging and high-speed communication technologies, the emergence of cost-effective and power-efficient die-to-die (D2D) interconnects with massive bandwidth and low latency—now being standardized by protocols like UCIe [ 53 ] —is fundamentally recasting inter-die data transfer from a performance bottleneck into a rich architectural resource. Capitalizing on this significant opportunity, a novel parallelization strategy is expected to not only fully exploit the intra-package communication resource to enhance on-chip memory efficiency and curtail off-chip traffic, but also replace the expensive collective communication, inherent to traditional EP/TP, with highly efficient point-to-point transfers. Complementing the interconnection benefit, each chiplet typically possesses an independent control path, enabling chiplet arrays to exhibit a Multi-Instruction-Multi-Data (MIMD) character that supports asymmetric execution patterns. Consequently, load balance can be systematically engineered through a non-uniform yet mutually complementary mapping space [ 51 , 80 ] . Collectively, these two factors unlock opportunities for fine-grained, synchronization-free dataflow among chiplets [ 79 ] . Grounded in these observations, we propose Fully Sharded Expert Data-parallelism (FSE-DP), a specialized parallelization strategy tailored for MoE inference on multi-chiplet packages that delivers the following advantages: Save on-chip memory and reduce off-chip memory traffic. The root cause of duplicated on-chip memory is the rigid ”one-chip-one-slice” mapping assumed by EP or TP. FSE-DP breaks this assumption by treating the whole chiplet array as a single, pooled buffer: only one physical copy of any token/expert slice is kept in the entire package. Leveraging high D2D bandwidth, FSE-DP streams the expert slice along a scheduled trajectory. The saved on-chip memory provides more data-reuse opportunities to reduce external memory access. Dynamic computation-communication overlap and load balancing. The long-tail distribution and dynamic features of MoE render fixed scheduling infeasible. FSE-DP introduces a dynamic fine-grained dataflow to realize each expert’s trajectory, which turns this heterogeneity into an opportunity. By fusing and complementing the dataflow of expert trajectories with different load characteristics at fine granularity, FSE-DP achieves dynamic computation-communication overlap under non-unified memory access (D2D and die-to-DDR), minimizes on-chip memory, and enables load balancing. Streamline fine-grained complexities with hardware-efficient rules and algorithms. The fine-grained dataflow fusion ostensibly introduces intricate memory-access and communication patterns, significantly elevating execution complexity. Nevertheless, FSE-DP’s dataflow is steered by a handful of lightweight, self-acting rules. Each chiplet receives the slice, computes its local token batch, and immediately forwards the slice to the next chiplet. These rules transparently abstract away memory and inter-die communication