EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization

Yize Wu, Ke Gao, Ling Li, Yanjun Wu · 2025

Read EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization by Yize Wu, Ke Gao, Ling Li, Yanjun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02493.txt.

Description

Yize Wu, Ke Gao, Ling Li, and Yanjun Wu sped up language model replies on multi GPU rigs. Their paper, from twenty twenty-five, deals with speculative decoding. A small draft model writes a guess line of tokens, and the big model checks it. This trick cuts wait with no loss in quality. On many GPUs, the draft model needs fewer cards than the big one, so cards sit idle while the draft runs.

Their fix, EasySpec, runs the draft model's layers in parallel, not in line. That breaks the chain between layers so more cards stay busy. Readers who serve large models get a way to use their full rig. Less idle time means faster replies at the same cost.

Catalog record

CREATORYize Wu, Ke Gao, Ling Li, Yanjun Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02493

Browse the full library · About the Tower of Babel