Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling

Yan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, Pengfei Zheng · 2025

Read Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling by Yan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, Pengfei Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04398.txt.

Introduction

Yan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, and Pengfei Zheng attack a serving bottleneck for giant AI models in this two thousand twenty-five paper. Mixture-of-experts models split work among specialist subnetworks spread across devices. Expert parallelism routes tokens to the device that hosts the needed expert, setting off all-to-all communication. Past systems set expert placement and token scheduling apart, wasting bandwidth between the two calls.

Their paradigm, Semantic Parallelism, schedules the model and the data as one. Experts and the tokens that wake them land on the same device. The build, Sem-MoE, clusters experts offline by co-activation tendencies. Online, it rebatches new requests toward devices that host their likely experts. It reshuffles tokens in requests for tensor-parallel setups. Modeled activation likelihoods drive all three techniques, ahead of time. The goal is MoE inference whose cost is set by computation, not communication.

Catalog record

CREATORYan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, Pengfei Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling Yan Li, Zhenyu Zhang, Zhengang Wang, Pengfei Chen, Pengfei Zheng Abstract Prevailing LLM (Large Language Model) serving engines employ expert parallelism (EP) to implement multi-device inference of massive Mixture-of-Experts (MoE) models. However, the efficiency of expert parallel inference is largely bounded by inter-device communication, as EP embraces expensive all-to-all collectives to route tokens to the remote experts if not collocating on the same GPU/NPU device. Nevertheless, state-of-the-art schemes treat expert device-placement and request (or token) device-scheduling as separate concerns, triggering excessive communication between them and compromising inference efficiency This paper proposes Semantic Parallelism , a novel parallelism paradigm that minimizes the steep communication costs in EP-centric MoE serving via model-data collaborative scheduling. We implement Semantic Parallelism in a framework called Sem-MoE. Sem-MoE maximally collocates experts and their activating tokens onto the same device using proactively modeled activation likelihood between them and introduces three key techniques: (1) Offline model scheduling, which preliminarily clusters and collocates experts onto devices based on their co-activation tendencies for certain classes of input. (2) Online inter-request data scheduling for Attention-DP setups, which proactively rebatches incoming requests onto the device that hosts experts most likely and frequently activated by the corresponding requests. (3) Online intra-request data scheduling for Attention-TP setups, which seamlessly fuses a token reshuffling procedure into the original inference pipeline and proactively reschedules tokens to devices to reduce dispersed remote routing. We build Sem-MoE into a prevailing LLM serving engine SGLANG. Experiments show our collaborative scheduling approach can effectively reduce the all-to-all communication volume in EP and achieve superior inference throughput compared to existing solutions. 1 Introduction The democratization of large language models (LLMs) has been largely driven by continuous model scaling. Over the past five years, the parameter count of the largest trained LLMs has increased by three orders of magnitude, posing significant challenges to the scalability and economic viability of both training and inference under modern AI hardware constraints. To mitigate these challenges, the Mixture-of-Experts (MoE) architecture Fedus et al. (2022) ; Artetxe et al. (2022) ; Jiang et al. (2024) has been introduced. Unlike dense models, MoE models sparsely activate one or more expert sub-networks per input, enabling training of trillion-parameter models without compromising accuracy, while maintaining a sub-linear increase in computational cost. This approach has gained widespread adoption in recent industrial-strength LLMs, including DeepSeek-V3 DeepSeek-AI (2024b) /DeepSeek-R1 DeepSeek-AI (2025) , GPT-OSS OpenAI et al. (2025) , the Qwen3-Series Yang et al. (2025) , and Kimi-K2 Team et al. (2025) . However, at inference time, massive MoE models still require substantial GPU/NPU 1 1 1 We use GPU and NPU interchangeably in this paper. resources to compute, store, and load both expert and attention parameters. To achieve scalability and meet latency requirements, existing inference frameworks deploy multi-dimensional parallelism strategies that distribute experts and attention blocks across interconnected devices. An efficient parallelization scheme must effectively partition input tokens and model parameters, maximize resource utilization, and minimize communication overhead. To address the memory demands of large-scale MoE deployment and leverage aggregate memory bandwidth, modern inference engines such as SGLang Zheng et al. (2024) and vLLM Kwon et al. (2023) employ expert parallelism (EP), whereby experts are distributed across devices. Attention layers are typically parallelized via data parallelism (DP) or tensor parallelism (TP). While EP enables parallel computation of experts across GPUs, it introduces significant communication overhead: intermediate activations must be dispatched from the gating module on a source GPU to the destination GPUs hosting the routed experts, and later combined back after expert computation. These operations often result in cluster-wide any-to-any token shuffling, typically implemented via two all2all collective operations (e.g., NCCL/HCCL’s all2all ). Figure 1: Latency breakdown for DeepSeek-V2-Lite inference over a single MoE layer. Hardware: 8-GPU server with fast inter-GPU network (specialized connection with over 400GB/s bandwidth). Our analysis reveals that the inference performance of MoE models remains severely constrained by these costly all2all operations. For instance, a preliminary experiment running SGLang on the DeepSeek-V2-Lite model with 8 GPUs shows that EP communication accounts for up to 59.2% of the forward-pass latency in the MoE layers, respectively—even on high-speed interconnects (see Figure 1 ). This bottleneck is further exacerbated on slower interconnects such as PCI-e or Ethernet. Therefore, systematically reducing EP communication has become a critical task for improving the efficiency and scalability of MoE inference. In this paper, we demonstrate that the communication overhead of EP can be substantially reduced through a novel semantic-aware model–data collaborative scheduling approach, namely Semantic Parallelism . This method forecasts expert routing paths for both requests and individual tokens, and proactively co-schedules tokens and experts to eliminate redundant communication. We implement the above idea in a system Sem-MoE , including two key techniques: First, Sem-MoE performs offline model scheduling to reduce expert dispersion. Experts that are frequently activated together are clustered and placed on the same device or server based on predicted token-expert affinities. This grouping is performed periodically offline to avoid runtime overhead. Second, Sem-MoE employs online data scheduling to align input tokens with their corresponding expert groups. This includes: (1) Inter-request scheduling for DP-based attention: dynamically batching requests to maximize expert affinity and minimize cross-device transfers. (2) Intra-request scheduling for TP-based attention: proactively shuffling token activations during the TP communication phase. Specifically, Sem-MoE replaces the standard post-attention allreduce with a shuffled-reduce-scatter and a deferred shuffled-allgather , effectively merging proactive token routing with necessary data transformation. Through collaborative model-data scheduling, semantic parallelism significantly reduces communication volume and improves inference throughput, as demonstrated through extensive experiments implemented on top of SGLang. We list the contributions of this paper as follows. 1. We conduct a comprehensive data analysis and reveal a significant context-independent correlation between tokens and experts in large-scale MoE models, which provides a foundational insight for optimizing expert placement and token routing. 2. We design and implement an efficient model-data collaborative scheduling algorithm that leverages the observed token–expert affinity. Our scheduler improves local activation rate by 15.4% compared to baseline methods, substantially reducing unnecessary cross-device communication. 3. We implement semantic parallelism in Sem-MoE on top of the state-of-the-art inference engine SGLang and perform extensive evaluations. The results demonstrate that Sem-MoE achieves a throughput improvement of up to 2.78x under specific SLOs in Attention-DP scenarios and up to 24.9% latency reduction under Attention-TP setups, validating the practical effectiveness of our approach. 2 Background Mixture-of-Experts The Mixture-of-Experts (MoE) architecture is a conditional computation paradigm designed to scale model capacity without a proportional increase in computational cost [1]. Unlike dense models, where all parameters are activated for every input, an MoE model consists of a multitude of expert sub-networks (typically Feed-Forward Networks, FFNs) and a gating network (or router). For each input token, the gating network predicts a sparse combination of experts (e.g., the top- k experts) to which the token is dispatched. Only the selected experts are activated for computation. The most common gating function is the Top-K Gating, which selects the k experts with the highest scores. This design enables models to possess a vast number of parameters (e.g., trillions) while keeping the FLOPs per token roughly constant, as only a small, fixed number of experts (e.g., k=2 ) are active per token. This has made MoE

Read the complete text