MoESD: Unveil Speculative Decoding’s Potential for Accelerating Sparse MoE

Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang · 2025

Read MoESD: Unveil Speculative Decoding’s Potential for Accelerating Sparse MoE by Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19645.txt.

Description

Zongle Huang and seven co-authors wrote a 2025 paper showing that speculative decoding speeds up sparse mixture-of-experts models more than dense ones. Speculative decoding drafts tokens with a small model and verifies them with the big one, saving time without losing accuracy. It was long assumed useful only for dense models.

Testing at medium batch sizes overturned that assumption: expert models gained more. The sparser the model, the wider the batch range where the trick pays. The authors built a theoretical model of the trade-offs. They also introduce a metric, target efficiency, because high draft acceptance alone does not guarantee speed when workloads or architectures shift. The analysis locates the true bottlenecks in accelerated serving. Private, single-tenant deployments are a highlighted use case.

Catalog record

CREATORZongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19645

Browse the full library · About the Tower of Babel