Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs

Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan · 2026

Read Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs by Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00879.txt.

Description

Hao Mark Chen and colleagues propose Dynamic Expert Sharing, shifting tuning from token-centric pruning and expert skipping to sequence-level coreset selection. Diffusion language models combined with Mixture-of-Experts architectures suffer expert explosion: as parallel tokens grow, distinct experts used grow nearly linearly. The resulting memory traffic pushes inference into a memory-bound regime. To maximize reuse, DES picks a compact, high-utility expert set for an entire parallel decoding block. Two strategies follow: Intra-Sequence Sharing, fitting allocation to the sequence level, and Saliency-Aware Voting.

Catalog record

CREATORHao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34852
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00879

Browse the full library · About the Tower of Babel