Training Domain Draft Models for Speculative Decoding: Best Practices and Insights

Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker · 2025

Read Training Domain Draft Models for Speculative Decoding: Best Practices and Insights by Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07807.txt.

Description

Fenglu Hong and co-workers study how to train small draft models for faster language model output, in a 2025 paper. In speculative decoding, a small model guesses ahead and a large model checks the guesses. A generic draft model guesses poorly for specialized target models, which slows the process.

The team compares ways to teach draft models with knowledge distillation, using full access to the target model or only its outputs. Offline teaching beat online teaching across function calling, biology, and Chinese domains. Full-access teaching beat output-only teaching by a smaller margin. Synthetic data reached most of the value of real user queries. The paper gives practical training guidelines.

Catalog record

CREATORFenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07807

Browse the full library · About the Tower of Babel