XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference Daniil Vankov, Nikita Ivkin, Kyle Ulrich, Xiang Song, Ashish Khetan, George Karypis Abstract Mixture-of-Experts (MoE) architectures are increasingly used to efficiently scale large language models. However, in production inference, request batching and speculative decoding significantly amplify expert activation, eroding these efficiency benefits. We address this issue by modeling batch-aware expert selection as a modular optimization problem and designing efficient greedy algorithms for different deployment settings. The proposed method, namely XShare , requires no retraining and dynamically adapts to each batch by maximizing the total gating score of selected experts. It reduces expert activation by up to 30% under standard batching, cuts peak GPU load by up to 3x in expert-parallel deployments, and achieves up to 14% throughput gains in speculative decoding via hierarchical, correlation-aware expert selection even if requests in a batch drawn from heterogeneous datasets. 1 Introduction Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks to unprecedented sizes while maintaining computational efficiency during training. The key insight, introduced by 17 , is to replace feed-forward layers with a collection of fully connected networks, where only a sparse subset is activated for each input token. This conditional computation approach fundamentally changes the scaling characteristics of neural networks: by decoupling total model capacity (parameters) from per-token computation (FLOPs), MoE models achieve dramatically better performance-to-FLOPs ratios compared to dense models. Recent work on scaling laws for MoE models ( 1 ) demonstrates that for a fixed computational budget, sparse MoE models with 10-100 \times more parameters than their dense counterparts can achieve superior accuracy while maintaining similar training costs. For instance, Switch Transformers ( 5 ) scale to trillion-parameter models while requiring only a fraction of the FLOPs needed to train equivalent-quality dense models. This favorable parameter-to-FLOPs trade-off during training has made MoE the architecture of choice for state-of-the-art large language models, with architectures like GShard ( 9 ) successfully demonstrating that sparse expert activation enables efficient training of models at unprecedented scales. However, the training-time efficiency gains of MoE architectures do not directly translate to inference-time efficiency in production serving systems. Production LLM serving systems can be categorized by their optimization objective: latency-optimized systems minimize per-request latency (typically using batch size 1), throughput-optimized systems maximize requests per second, and goodput-optimized systems maximize throughput subject to service level objectives for latency. Our work primarily targets goodput-optimized deployments, which must balance batching for hardware efficiency against latency constraints. These systems introduce two key challenges that make MoE inference inefficient. First, to maximize hardware utilization and amortize the cost of memory transfers, serving systems batch multiple requests together. While each individual token activates only k experts (typically from k=1 , k=4 , or k=8 ), the union of experts activated across all tokens in a batch grows substantially with batch size. Second, virtually all modern serving systems employ speculative decoding to improve throughput, where a small draft model generates candidate tokens that are verified in parallel by the target model. For a speculative length of s , each request effectively multiplies the batch size by (s+1) : a batch of B requests becomes B(s+1) tokens that must be processed simultaneously. The combination of request batching and speculative decoding results in expert activation patterns, where, for moderate batch sizes, the majority of experts must be loaded into memory, and for large batches, nearly all experts are activated. To quantify this effect, the expected number of activated experts in a batch follows \mathbb{E}[N_{a}]=N\left(1-(1-k/N)^{B}\right), where N is the total number of experts, k is the top- k routing, and B is the effective batch size (including speculative tokens). For DeepSeek-R1 ( N=256 , k=8 ), this predicts that at batch size 8, approximately 57 experts are activated, and at batch size 32, approximately 163 experts—closely matching our empirical observations. With speculative decoding of length s , the effective batch becomes B(s+1) , further accelerating expert activation growth. While a sparse MoE model remains computationally more efficient than a dense model with equivalent parameters (only activated experts consume FLOPs), the memory bandwidth bottleneck dominates during the autoregressive decode phase: all activated experts must be loaded from memory regardless of their utilization, making inference memory-IO-bound rather than compute-bound. Figure 1 : Average number of activated experts Recent interpretability work on MoE models ( 20 ) demonstrates that stronger models achieve better performance with lower neuron utilization — suggesting that efficient sparse activation is a hallmark of model quality rather than a compromise. However, these insights have not been translated into practical inference optimization strategies that can dynamically adapt expert selection to batch characteristics. Related works. Traditional network pruning techniques such as MetaPruning ( 12 ) and BilevelPruning ( 6 ) have been adapted to MoE architectures. More recently, 11 demonstrates that not all experts contribute equally to model accuracy, enabling aggressive static pruning of less important experts. DiEP ( 3 ) advances this direction by learning non-uniform pruning ratios across layers. 19 proposes MoE-Pruner, which uses router-weighted pruning metrics for one-shot weight-level pruning on calibration data, but produces static pruning decisions dependent on the calibration datasets and requires knowledge distillation to recover from moderate accuracy degradation. While these methods reduce the total parameter count, they require retraining or fine-tuning and produce fixed, static pruning decisions that cannot adapt to different workload characteristics or speculative decoding patterns. Most closely related to our work, 13 proposes Dynamic Skipping method, which operates at the token level, skipping expert e_{1} if w_{e_{1}}<\beta w_{e_{0}} , where \beta is calibrated per-layer. Intuitively, an expert is skipped when there is a large diminishing return, reflected by a sharp drop in gating scores between consecutively ranked experts. However, it processes each token independently and does not account for batch composition, so the batch-level expert explosion problem in production remains unresolved. In contrast, Lynx ( 7 ) was the first to recognize the batch-level expert explosion problem and to propose a batch-aware expert selection policy, LYNX-Lat. LYNX-Lat aggregates per-token expert preferences across a batch to identify the least-used experts, then drops a fixed, pre-selected number of them. The number of experts dropped is determined offline by tuning the accuracy–performance tradeoff. However, dropping the least frequently used experts ignores how important those experts were when they were selected (e.g., their rank for the tokens that used them). As a result, it can remove highly ranked experts for some tokens and severely hurt accuracy. Authors describe this policy at a conceptual level, without providing precise algorithms or pseudocode, making the approach difficult to reproduce or directly compare against. During the preparation of this manuscript, we became aware of concurrent work by 14 where authors propose building a shared pool of candidate experts by taking the top- k^{\prime} experts (with k^{\prime}