Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan Abstract Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) architectures is constrained by an “ expert explosion ”: as the number of tokens generated in parallel increases, the number of distinct experts activated grows nearly linearly. This results in substantial memory traffic that pushes inference into a memory-bound regime, negating the efficiency gains of both MoE and parallel decoding. To address this challenge, we propose Dynamic Expert Sharing (DES) , a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection. To maximize expert reuse, DES identifies a compact, high-utility set of experts to satisfy the requirements of an entire parallel decoding block. We introduce two innovative selection strategies: (1) Intra-Sequence Sharing (DES-Seq) , which adapts optimal allocation to the sequence level, and (2) Saliency-Aware Voting (DES-Vote) , a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights. Extensive experiments on MoE dLLMs demonstrate that DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy, effectively decoupling memory overhead from the degree of parallelism. 1 Introduction Figure 1 : From left to right: (a) Growth of unique activated experts relative to the degree of parallelism (input block size). (b) Roofline model demonstrating that MoE dLLMs are more memory-bound than their dense counterparts and how DES mitigates this bottleneck. (c) Trade-off between activated experts and relative accuracy, where DES-Vote notably extends the Pareto frontier. (d) MoE kernel latency versus relative accuracy. For (c) and (d), accuracy is reported relative to the vanilla baseline. Results in (c) are averaged across benchmarks, namely HumanEval, MBPP, MATH500, and GSM8K, while (d) evaluates DES-Vote ’s performance on MBPP. The average activated experts is defined as the mean number of active experts across all MoE layers in the respective model. Recent advances in large-scale foundation models have seen the emergence of native parallel decoding paradigms, most notably diffusion large language models (dLLMs) ( 35 ; 32 ) . The state-of-the-art dLLMs typically leverage block-based parallel decoding ( 3 ; 30 ) to achieve performance parity with traditional autoregressive (AR) models through concurrent token generation. To scale these architectures effectively, dLLMs increasingly integrate Mixture-of-Experts (MoE) ( 15 ; 31 ; 1 ) as a core architectural component ( 5 ; 35 ; 10 ) . The primary advantage of MoE lies in its ability to decouple the model’s overall knowledge capacity from its per-token computational cost. By activating only a sparse subset of experts for each input, MoE allows dLLMs to scale to massive parameter counts, significantly enhancing representational power without a proportional increase in FLOPs. The integration of MoE into dLLMs, however, introduces an efficiency paradox, illustrated by our roofline analysis (Figure 1 ). In particular, while transitioning from AR to parallel dLLM increases operational intensity, the sparse activation of MoE dLLMs makes them more memory-bound than their dense counterparts. In these memory-bound scenarios—typical of multi-GPU and CPU-offloading environments ( 6 ; 33 ) —latency is dominated by the traffic from High Bandwidth Memory (HBM) to Static RAM (SRAM) required to load unique expert weights. This bottleneck is intensified by an “ expert explosion ” inherent to parallel generation: whereas AR decoding processes a single token per step, dLLMs process N tokens simultaneously. Consequently, if experts are selected independently, the unique expert load scales nearly linearly with the block size N , rapidly exceeding the steady-state memory traffic of AR decoding (Figure 1 ). Crucially, as the parallel block scales, the overhead of transferring divergent experts from HBM to on-chip memory can negate the compute savings of MoE, potentially rendering these models slower than their smaller dense equivalents. This memory traffic overhead imposes a systemic ceiling on the throughput gains promised by parallelization. Existing dynamic efficiency optimizations for MoE, such as expert skipping ( 22 ; 16 ; 2 ) , are primarily designed for AR models and remain strictly token-centric. While effective at reducing per-token computational FLOPs, these methods prioritize local sparsity rather than cross-token expert redundancy. Consequently, they fail to mitigate the global HBM traffic bottleneck, as the unique expert weights loaded per layer are not optimized at the sequence level. Crucially, these approaches do not directly address the memory traffic overhead encountered in block-wise parallel decoding, where the collective expert footprint, rather than per-token compute, becomes the primary constraint on throughput. To address these challenges, we propose Dynamic Expert Sharing (DES) , a novel method designed to minimize the expert footprint by forming consensus across parallel tokens. Our approach shifts the optimization objective from per-token pruning to sequence-level sharing. We hypothesize that, unlike standard AR batching where independent tokens may activate disparate experts, parallel decoding in dLLMs inherently involves semantically coupled tokens that exhibit significant overlap in expert demand . To exploit this, we introduce coreset selection , where a minimal, high-utility set of experts is dynamically identified at runtime to serve the entire parallel block. We explore two specific strategies: • Intra-Sequence Sharing (DES-Seq): We take the union of experts chosen independently by each token in the block. This allows for vectorized execution, eliminating the need to handle divergent expert assignments for individual positions. • Saliency-Aware Voting (DES-Vote): A novel mechanism where tokens collectively vote for experts based on their weighted router saliency. This prioritizes the most critical experts by focusing on the holistic semantic complexity of the sequence. By restricting the expert selection to a shared coreset, we significantly reduce the HBM-to-SRAM traffic, effectively mitigating the memory traffic overhead of parallel MoE decoding without compromising generative quality. As shown in Figure 1 , DES reduces the active experts by up to 55% and the MoE kernel latency by up to 38.0% while achieving performance comparable to the vanilla baseline. Our contributions are summarized as follows: 1. We identify and quantify the “expert explosion” bottleneck in parallel MoE decoding, demonstrating how independent expert selection across parallel tokens induces a critical memory traffic overhead (Section 3 ). 2. We propose Dynamic Expert Sharing , a method that shifts MoE optimization from token-centric pruning to sequence-level coreset selection, leveraging a novel saliency-aware voting mechanism to maximize expert reuse (Section 4 ). 3. Through extensive benchmarking on MoE dLLMs, we demonstrate that DES achieves significant memory saving and speedup with negligible accuracy drop, effectively decoupling memory traffic from the degree of parallelism (Section 5 ). 2 Background Parallel Decoding. Non-AR models first emerged in the field of neural machine translation ( 14 ; 28 ) as a means to reduce generation latency. More recently, parallel decoding has been adopted for LLMs to reduce the number of memory-bound forward passes. These approaches typically utilize either specially trained tokens ( 7 ; 20 ) or n -grams retrieved from context ( 12 ) as inputs to a frozen AR model, enabling the generation of multiple output tokens in parallel. However, because the AR backbone undergoes limited re-training, the resulting generation quality is often constrained; consequently, these parallel models are primarily employed within speculative decoding settings that require an additional verification step. Diffusion Large Language Models. To fully realize the acceleration potential of parallel generation and eliminate the overhead of slow verification, natively parallel models ( 13 ; 35 ) are being trained from scratch with multi-token prediction objectives. In particular, Diffusion Large Language Models (dLLMs) have emerged as a promising candidate to balance generation quality and efficiency ( 35 ; 32 ; 3 ) . These models ingest multiple masked tokens and process them concurrently, enabling parallel decoding without the need for supplementary verification steps. To further enhance dLLM throughput, recent research has introduced approximate KV cache and sampling strategies ( 30 ; 29 ; 21 ) alongside specialized inference frameworks ( 24 ) . Efficient Mixture-of-Experts. Mixture-of-Experts (MoE) has emerged as the de facto architecture to scale up dLLMs ( 5 ; 35 ; 10 ) . Formally, an MoE layer replaces the standard feedforward network (FFN) of a Transformer block with a set of M independent experts \{E_{1},\dots,E_{M}\} , where each expert E_{i} is a sub-network and D is the hidden dimension. A gating or routing function G:\mathbb{R}^{D}\to\mathbb{R}^{M} assigns a score to each expert for a given input hidden state x . The output of the MoE module is computed as the weighted sum of the K highest-scoring experts: \text{MoE}(x)=\sum_{i\in\mathcal{S}}\frac{G(x)_{i}}{\sum_{j\in\mathcal{S}}G(x)_{j}}E_{i}(x) (1) where \mathcal{S}=\text{TopK}(G(x),K) is the set of indices corresponding to the K experts with the highest routing scores. In the context of dLLMs, the total number of unique experts required across a batch of N tokens is |\bigcup_{n=1}^{N}\mathcal{S}_{n}| , which typically grows as the degree of parallelism increases. Existing strategies for MoE efficiency primarily follow two paradigms: offline expert pruning and dynamic expert skipping. Offline approaches, such as expert pruning ( 9 ; 34 ) and merging ( 18 ) , aim to reduce the static parameter footprint through structural sparsity, quantization, or weight consolidation. In contrast, dynamic expert skipping ( 17 ; 22 ; 16 ; 2 ) leverages runtime information to bypass redundant expert computations on-the-fly. Additionally, OEA ( 27 ) first attempted expert sharing across batches within the context of AR decoding. While our proposed method is orthogonal to offline pruning strategies, it addresses a critical gap in existing research: current online optimizations are largely tailored for AR models and are not designed to optimize for the memory-traffic bottlenecks inherent to dLLMs. Figure 2 : Latency characterization of MoE decoding under varying block lengths. Left: Latency breakdown across MoE FFN, attention, and other components, showing that MoE FFN dominates end-to-end latency. Right: MoE kernel latency versus activated experts for different block lengths (16, 32, 64), illustrating a linear increase in latency with more activated experts. Figure 3 : Overview of the Dynamic Expert Sharing method. (a) Vanilla MoE dLLM independently routes multiple tokens, leading to a high count of unique activated experts. (b) Dynamic Expert Skipping reduces local per-token computation (indicated by dotted lines) but often fails to optimize the global unique expert load. (c) Dynamic Expert Sharing employs sequence-level coreset selection (via DES-Seq or DES-Vote) to identify high-utility experts globally, significantly minimizing the unique expert weights transferred from HBM. (d) DES-Vote enforces sequence-level consensus by aggregating router weights across the parallel block, contrasting with the independent per-token selection utilized in DES-Seq. Greyed out boxes and bars represent experts not selected and not fetched to on-chip memory. 3 Motivation The primary motivation for parallel decoding in dLLMs is to overcome the sequential bottleneck of AR generation, which is notoriously memory-bound due to the low arithmetic intensity of processing a single token per forward pass. While dLLMs scale concurrent tokens to increase arithmetic intensity, the integration of MoE reintroduces the memory-bound bottleneck (Figure 1 ). This is exacerbated by both algorithmic constraints, where smaller parallel blocks ( e.g. , 16/32 tokens) are necessary to maintain competitive accuracy ( 3 ) , and hardware trends, where the FLOPs/byte ratio of modern GPUs continues to outpace memory bandwidth ( 23 ) . Consequently, MoE dLLM inference is still usually memory-bound, with latency dominated by the cost of loading unique expert weights, analogous to AR decoding’s bandwidth bottleneck but at a larger scale of expert demand ( 33 ) . Within this regime, the MoE FFN layer emerges as the dominant latency contributor (Figure 2 ). Following the simplified latency model from previous work ( 27 ) , the time f(c) to process c tokens through a single expert is f(c)=ac+b for c>0 , where b represents the weight fetching cost from HBM to on-chip SRAM and a is the marginal computation cost. The total MoE block latency for a sequence of N parallel tokens is thus: \displaystyle L_{MoE}=\sum_{i=1}^{N_{total}}(b\cdot\mathbbm{1}_{cnt_{i}>0}+a\cdot cnt_{i})= (2) \displaystyle b\cdot\left|\bigcup_{n=1}^{N}\mathcal{S}_{n}\right|+a\cdot(N\cdot K) where cnt_{i} is the number of tokens routed to expert E_{i} , and \mathcal{S}_{n} is the set of K experts activated by the n -th parallel token. In MoE architectures with uniform routing, the expected number of unique activated experts |\bigcup_{n=1}^{N}\mathcal{S}_{n}| grows as N_{total}(1-(1-K/N_{total})^{N}) . As the degree of parallelism N increases, this union set expands rapidly (Figure 1 ), leading to an increase in latency (Figure 2 ). This ” expert explosion ” underscores the limitations of existing expert skipping methods in parallel decoding: 1. Diminishing Returns from Compute Sparsity: As latency is dominated by the weight-fetching cost b\cdot|\bigcup_{n=1}^{N}\mathcal{S}_{n}| , reducing the compute term a yields minimal speedup if an expert remains active for any other concurrent token. This motivates ”re-activating” experts within a shared coreset to recover accuracy at near-zero marginal latency (Section 5 ). 2. Lack of Cross-Token Synergy: Token-centric skipping ignores the redundancy inherent in parallel decoding. Failing to enforce a collective consensus, they do not minimize the unique expert load |\bigcup_{n=1}^{N}\mathcal{S}_{n}| , leaving the memory bottleneck unresolved. Optimizing parallel MoE inference thus requires a shift toward Dynamic Expert Sharing , which explicitly minimizes the unique expert load |\bigcup_{n=1}^{N}\mathcal{S}_{n}| at the sequence level. 4 Methodology 4.1 Dynamic Expert Sharing To optimize the memory-bound bottleneck identified in Section 3 , we define a Coreset Selection Function , \Phi:\mathcal{I}\to\mathcal{C} , which maps available runtime information \mathcal{I} (e.g., aggregated router logits or hidden states) to a shared subset of experts \mathcal{C}\subset\{E_{1},\dots,E_{M}\} . The coreset selection function identifies the most salient experts for a parallel decoding block. Unlike traditional AR batching, where tokens often belong to disparate tasks, parallel decoding tokens share a common context. We hypothesize that rather than allowing independent routing, we can identify a compact, sequence-level coreset via cross-token consensus. By restricting each token’s Top- k selection to this shared subset, we maximize expert reuse and minimize HBM transfer. By substituting this coreset into our latency model, we reformulate the MoE layer latency from Equation 2 . In the vanilla setting, the unique expert load is defined by the union of independent selections |\bigcup_{n=1}^{N}\mathcal{S}_{n}| , which is implicitly upper-bounded by the total expert pool size M . With the introduction of the coreset, the latency is expressed as: L_{\text{MoE}}(\Phi)\leq b\cdot|\Phi(\mathcal{I})|+a\cdot(N\cdot k) (3) where |\Phi(\mathcal{I})| is the cardinality of the selected coreset. Crucially, whereas the vanilla expert load is constrained by a constant, this formulation transforms the upper bound into an explicit variable |\Phi(\mathcal{I})| that guides the optimization process. We therefore define the search for an optimal coreset selection strategy as a constrained optimization problem: \displaystyle\Phi^{*}=\arg\min_{\Phi} \displaystyle|\Phi(\mathcal{I})| (4) \displaystyle\text{s.t.} \displaystyle\mathcal{A}(\Phi(\mathcal{I}))\geq\mathcal{A}_{\text{base}}-\epsilon where \mathcal{A}(\Phi) denotes model accuracy and \epsilon is a predefined threshold for tolerable performance degradation. By shifting from independent, token-centric routing to collective coreset selection, we decouple the HBM weight-fetching cost from the number of parallel tokens N . The DES algorithm implements this by transitioning from token-level to sequence-level routing. As detailed in Algorithm 1 , the process consists of two stages: (1) Sequence-level Consensus , where the shared coreset \mathcal{C} is identified, and (2) Constrained Local Routing , where each token selects its Top- k experts exclusively from \mathcal{C} . By re-normalizing weights via an activation function \sigma from the model architecture, we preserve the original routing intent within the optimized memory budget. The algorithm overview is shown in Figure 3 . Algorithm 1 Dynamic Expert Sharing (DES) 0: Sequence information \mathcal{I} , Coreset selection function \Phi , Activation function \sigma , Target K . 0: Layer output Y . 1: // Stage 1: Sequence-level Consensus 2: \mathcal{C}\leftarrow\Phi(\mathcal{I}) {Identify high-utility expert coreset} 3: // Stage 2: Constrained Local Routing 4: for each token n\in\{1,\dots,N\} do 5: \mathcal{S}_{n}\leftarrow\text{TopK}(\mathcal{I}_{n}|_{i\in\mathcal{C}},K) {Route within coreset} 6: g_{n}\leftarrow\sigma(\mathcal{I}_{n}|_{i\in\mathcal{S}_{n}}) {Re-normalize gate weights} 7: y_{n}\leftarrow\sum_{i\in\mathcal{S}_{n}}g_{n,i}\cdot E_{i}(x_{n}) 8: end for 9: return Y=\{y_{1},\dots,y_{N}\} Figure 4 : Analysis of Dynamic Expert Sharing (DES) in LLaDA-MoE-7B (8 tokens, Layer 10). (Left) Expert Importance: Heatmap of reconstruction loss sensitivity to expert ablation. Darker regions indicate an increase in reconstruction loss when an expert is removed, revealing strong dependencies between specific token positions and experts. (Middle) Selection Overlays: DES-Seq (orange), DES-Vote (pink), and shared (yellow) selections overlaid on log-routing weights. DES-Vote captures high-importance experts (see index 42) that local ranking misses. (Right) Performance vs. Coreset Size ( M_{\text{core}} ): DES-Vote achieves higher Top- k recall (top) and lower residual reconstruction loss (bottom) than DES-Seq across M_{\text{core}} . Red dashed lines denote the specific M_{\text{core}} visualized in the heatmaps. 4.2 Coreset Selection Methods While the optimization problem in Equation 4 is difficult to solve directly, we approximate the solution by proposing two strategies for dynamic coreset selection: Intra-Sequence Sharing (DES-Seq) and Saliency-Aware Voting (DES-Vote) . These methods serve as the function \Phi(\mathcal{I}) in Algorithm 1 to identify a sequence-wide consensus. Intra-Sequence Sharing (DES-Seq). A straightforward approach to form a smaller coreset is to select a fixed number of the most salient experts from each token. While previously explored for batch-level optimization in AR models ( 27 ) , we adapt this to the intra-sequence level for dLLM parallel decoding. For each token n in the block, we select its top- k experts, where k is a hyperparameter satisfying k