ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts

Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, Alexander Heinecke · 2025

Read ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts by Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, Alexander Heinecke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13565.txt.

Introduction

Evangelos Georganas and co-authors sped up language model replies in a 2025 paper. Speculative decoding lets a small fast model draft the next words while the big model checks them, which keeps quality while cutting wait. The catch is that the draft model must be trained to match the big one, a poor fit for plug-and-play use.

Their design stacks more than one draft level and shrinks the drafts with quantization, which stores weights in fewer bits. A shrunken draft runs faster and needs no fresh training pact with the target. The balance between draft speed and how many drafts pass the check sets the gain. The work targets cheaper serving of large models.

Catalog record

CREATOREvangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, Alexander Heinecke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13565

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, Alexander Heinecke Abstract Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is to use a full-precision, small, fast model as “draft” to generate the next few tokens and use the “target” large model to verify the draft-generated tokens. The efficacy of this method heavily relies on the acceptance ratio of the draft-generated tokens and the relative token throughput of the draft versus the target model. Nevertheless, an efficient SD pipeline requires pre-training and aligning the draft model to the target model, making it impractical for LLM inference in a plug-and-play fashion. In this work, we propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4. In practice, our plug-and-play solution gives speedups up to 2 × \times over the BF16 baseline. Then we pursue an opportunity for further acceleration: the MXFP4 draft token generation itself can be accelerated via speculative decoding by using yet another smaller draft. We call our method ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts since it recursively applies speculation for accelerating the draft-token generation. Combining Multi-Level Speculative Decoding with MXFP4 Quantized Drafts we outperform state-of-the-art speculative decoding, yielding speedups up to 2.72 × \times over the BF16 baseline. I Introduction Large Language Models (LLMs) such as Chat-GPT [ 1 ] and GPT-4 [ 2 ] , have revolutionized the field of natural language processing (NLP) and have found application in areas like question-answering, coding, and user-interactive scenarios akin to chatbots and virtual assistants. As such, nowadays LLMs are being increasingly deployed on edge devices and PCs, giving birth to the era of “AI PC” where the the models are deployed locally to assist the user in real time (e.g. as standalone AI assistants in coding [ 3 ] ) or as part of more complicated software/workload workflows (e.g. LLMs used internally in other application for documents and image processing [ 4 ] ). In all these cases, in order to improve user-experience the LLMs shall not sacrifice accuracy, and at the same time the latency should still stay within reasonable limits for real-time interaction. Unfortunately, the rising size of LLMs implies ever-growing latency due to their auto-regressive nature: they generate tokens one by one. A recent solution to mitigate this issue is called speculative decoding (SD) [ 5 , 6 , 7 ] . In a typical SD setup, the idea is to use a small, fast model as “draft” to generate the next few tokens and use the “target” large model to verify the draft-generated tokens. The verification step comprises merely of a single forward pass of the target model. After verification, the draft-generated tokens following the first mis-matched token are discarded and the process continues with the next batch of draft-tokens to be generated. By leveraging this methodology, one can leverage a smaller, much faster model to generate many of the desired output tokens whereas the output is virtually identical to the one from large “target” model. As a result, SD does not compromise accuracy and still it enjoys substantial latency reduction over the traditional greedy decoding strategy. Fig. 1 : Various flavors of Speculative Decoding: (a) Speculative decoding with a large target model (e.g. Llama 7B in BF16) and a custom, small draft model (e.g. Tiny Llama 68M), (b) Speculative decoding with a large target model and an MXFP4-direct-cast-quantized target model as draft model, (c) Multi-Level Speculative Decoding: a large target model uses an MXFP4-direct-cast-quantized target model as draft model, and subsequently the MXFP4-direct-cast-quantized model uses a smaller draft, potentially also MXFP4-quantized. The common practice in SD is to use a very small draft model, around two orders of magnitude smaller than the target model, in order to observe substantial speedups [ 5 ] , since the draft model itself suffers from the same auto-regressive nature of generation. The latency issue for the drafting model is de-emphasized if it is noticeably smaller than the target model. Nevertheless, this common practice exhibit two drawbacks. First, a small draft is not always readily available; for example, the smallest model in the Llama2 foundational family has 7 billion parameters [ 8 ] . The community has therefore developed methods (e.g. FastDraft [ 9 ] ) to obtain such small draft models, but these techniques require pre-training and aligning the draft model to the target model, making them impractical for LLM inference in a plug-and-play fashion. Second, even in case such a small model is readily available, usually its direct usage as draft results in poor acceptance of the draft-generated tokens by the target models given the capacity gap between the target and draft [ 10 ] . As a consequence of the poor acceptance ratio of the draft-generated tokens, the larger and much slower target model has to be called an increasing number of times for verification/validation, and eventually the performance benefits of SD are diminished. We make two key observations with respect to the limitations of SD as described in the previous paragraph. First, recent advances in Weight-Only-Quantization (WOQ) with 4-bit datatypes (and beyond) has shown to generate high quality results with small accuracy loss over the BF16 models [ 11 , 12 , 13 , 14 ] . Typically one can leverage such WOQ methods without any fine-tuning/extra processing overhead by merely direct-casting the weights to the target precision, making them a plug-and-play solution for inference acceleration especially in bandwidth-bounds scenarios [ 15 ] . Therefore, we propose the usage of 4-bit quantized models as drafts in a SD setup, since they illustrate substantial speedups over the target BF16 models, and additionally they can yield high acceptance ratios of the draft-generated tokens given their intrinsic alignment with the base/target model (see Figure 1 (b)). The second key observation is inspired by multi-grid methods in HPC [ 16 ] where one considers a problem on different refinement levels and uses solutions on coarser levels to improve upon solutions on finer levels. In the context of inference via conventional SD, a custom, tiny model is available as draft (i.e. “coarse” model), and we use it directly from the large target model (i.e. “fine” model) for token-drafting in one-step, and inadvertently suffer from the low acceptance ratios of the coarsely-drafted-tokens (see Figure 1 (a)). In this work, we propose the introduction of an intermediate model for token-drafting between the target and the tiny models that is merely a 4-bit direct-cast of the target model (thus its granularity is between the “fine” target model and the “coarse” draft model). In such a setup, the target model uses a 4-bit direct-cast-model as draft, and subsequently the intermediate 4-bit model uses the tiny model for its own speculative decoding (i.e. we apply multi-level, hierarchical decoding illustrated in Figure 1 (c)). The benefit of this multi-level approach is two-fold: First, it mitigates the issue of low acceptance ratios of the tiny-draft-tokens that result in excessive number of invocations of the target BF16 model, since now their verification is subsumed by the intermediate, much faster

Read the complete text