Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker · 2025
Read Training Domain Draft Models for Speculative Decoding: Best Practices and Insights by Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07807.txt.
Description
Fenglu Hong and co-workers study how to train small draft models for faster language model output, in a 2025 paper. In speculative decoding, a small model guesses ahead and a large model checks the guesses. A generic draft model guesses poorly for specialized target models, which slows the process.
The team compares ways to teach draft models with knowledge distillation, using full access to the target model or only its outputs. Offline teaching beat online teaching across function calling, biology, and Chinese domains. Full-access teaching beat output-only teaching by a smaller margin. Synthetic data reached most of the value of real user queries. The paper gives practical training guidelines.
Catalog record
| CREATOR | Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07807