Scaling Laws for Speculative Decoding

Siyuan Yan†, Mo Zhu†, Guo-qing Jiang, Jianfei Wang, Jiaxing Chen, Wentai Zhang, Xiang Liao, Xiao Cui, Chen Zhang, Zhuoran Song, Ran Zhu · 2025

Read Scaling Laws for Speculative Decoding by Siyuan Yan†, Mo Zhu†, Guo-qing Jiang, Jianfei Wang, Jiaxing Chen, Wentai Zhang, Xiang Liao, Xiao Cui, Chen Zhang, Zhuoran Song, Ran Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07858.txt.

Description

Siyuan Yan and co-authors work out scaling laws for speculative decoding, a trick that speeds up large language models by letting a small draft model propose tokens that the big model checks. Speed matters most for reasoning models that think in long chains. Yet how decoding speed scales with training and size was not known.

They prove log-linear laws linking the draft's acceptance rate to three factors: pretraining tokens, draft size, and batch size. Guided by the laws, their system coordinates all three dimensions for popular model families. It accepted drafts far more often than leading rivals and doubled decoding throughput in a production inference engine. The work turns an ad hoc trick into an engineering discipline.

Catalog record

CREATORSiyuan Yan†, Mo Zhu†, Guo-qing Jiang, Jianfei Wang, Jiaxing Chen, Wentai Zhang, Xiang Liao, Xiao Cui, Chen Zhang, Zhuoran Song, Ran Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07858

Browse the full library · About the Tower of Babel