Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan · 2026
Read Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs by Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00879.txt.
Description
Hao Mark Chen and colleagues propose Dynamic Expert Sharing, shifting tuning from token-centric pruning and expert skipping to sequence-level coreset selection. Diffusion language models combined with Mixture-of-Experts architectures suffer expert explosion: as parallel tokens grow, distinct experts used grow nearly linearly. The resulting memory traffic pushes inference into a memory-bound regime. To maximize reuse, DES picks a compact, high-utility expert set for an entire parallel decoding block. Two strategies follow: Intra-Sequence Sharing, fitting allocation to the sequence level, and Saliency-Aware Voting.
Catalog record
| CREATOR | Hao (Mark) Chen, Zhiwen Mo, Royson Lee, Qianzhou Wang, Da Li, Shell Xu Hu, Wayne Luk, Timothy Hospedales, Hongxiang Fan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34852 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00879