Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler · 2025
Read Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment by Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19309.txt.
Introduction
Gregor Bachmann and co-authors speed up language model generation in a 2025 paper. Speculative decoding lets a small model draft tokens ahead of the big one. The big model then checks many drafted tokens in a single pass.
Strict checking rejects many perfectly good draft tokens. Even strong drafters and human text fail that likeness test surprisingly often. The authors propose letting a judge accept valid alternatives instead. Alignment to the target model's habits is deliberately loosened in the design. Speed rises when fewer good tokens are thrown away during checking. Readers see a tradeoff between exact mimicry and fast, faithful output.
Catalog record
| CREATOR | Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler Abstract The performance of large language models (LLMs) is closely linked to their underlying size, leading to ever-growing networks and hence slower inference. Speculative decoding has been proposed as a technique to accelerate autoregressive generation, leveraging a fast draft model to propose candidate tokens, which are then verified in parallel based on their likelihood under the target model. While this approach guarantees to reproduce the target output, it incurs a substantial penalty: many high-quality draft tokens are rejected, even when they represent objectively valid continuations. Indeed, we show that even powerful draft models such as GPT-4o , as well as human text cannot achieve high acceptance rates under the standard verification scheme. This severely limits the speedup potential of current speculative decoding methods, as an early rejection becomes overwhelmingly likely when solely relying on alignment of draft and target. We thus ask the following question: Can we adapt verification to recognize correct, but non-aligned replies? To this end, we draw inspiration from the LLM-as-a-judge framework, which demonstrated that LLMs are able to rate answers in a versatile way. We carefully design a dataset to elicit the same capability in the target model by training a compact module on top of the embeddings to produce “judgements” of the current continuation. We showcase our strategy on the Llama-3.1 family, where our 8b/405B-Judge achieves a speedup of 9 × 9\times over Llama-405B , while maintaining its quality on a large range of benchmarks. These benefits remain present even in optimized inference frameworks, where our method reaches up to 141 141 tokens/s for 8B/70B-Judge and 129 129 tokens/s for 8B/405B on 2 2 and 8 8 H100s respectively. Figure 1: Standard speculative decoding versus our judge decoding strategy for Llama-3.1-8B as draft and Llama-3.1-405B as target. Accepted (rejected) tokens are highlighted in green (red). 1 Introduction Large language models have transformed the field of natural language processing in recent years, displaying astounding capabilities across various tasks ( Radford et al., 2019 ; OpenAI et al., 2024 ) . The performance of these models is closely tied to their underlying size, with bigger models often achieving significantly better results across benchmarks ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) . For example, Meta recently released their largest and best model to date with Llama-3.1-405B , boasting an enormous parameter count of 405 billion ( Dubey et al., 2024 ) . While offering great performance, such big models require a vast amount of resources to be deployed, and inference efficiency starts to pose a critical problem. Due to the autoregressive nature of decoding coupled with the large parameter count, token generation becomes a memory-bound process, especially at small batch sizes ( Shazeer, 2019 ; Ivanov et al., 2021 ; Pope et al., 2023 ) . To speed up inference in such a setting, speculative decoding (SD) has been proposed ( Stern et al., 2018 ; Xia et al., 2023 ; Chen et al., 2023 ; Leviathan et al., 2023 ) , a technique that leverages the fact that processing several tokens in parallel comes at no additional latency cost in the memory-bound regime. More concretely, a small but fast draft model produces a sequence of M candidate tokens, which are then verified in parallel by the model of interest, usually referred to as the target model. In standard SD, the target model accepts a candidate token if it assigns the same or higher probability given the context, otherwise a biased coin is flipped. If at least one candidate token is accepted, inference time is reduced as the large model needs to be called only once to produce multiple tokens. As shown in Chen et al. (2023) , such a strategy provably preserves the distribution of the target model, while achieving significant speedups. Relying on target probabilities guarantees the same output, but as a consequence, a token is solely judged based on its alignment with the target model, and not by its inherent contextual quality. As a consequence, current approaches set the number of candidate tokens to small numbers M\in\{5,7\} , as an early rejection becomes overwhelmingly likely when drafting more. On the other hand, the quality of “ small ” language models (and thus the quality of candidate tokens) has been rapidly improving recently. GPT-4o and the recently introduced GPT-4o-mini show strong performance across many benchmarks, with OpenAI actively recommending these models over the more expensive GPT-4 . Similarly, Llama-3.1-8B has achieved bigger gains in performance over its prior iterations, compared to the larger Llama-3.1-70B ( Dubey et al., 2024 ) , highlighting as well that small models are catching up. Phi-3-mini is another small model at “only” three billion parameters that despite its size manages to match the performance of GPT-3.5 ( Abdin et al., 2024 ) . While draft models are rapidly improving and providing increasingly high-quality answers, alignment-based verification fails to reflect this progress, still rejecting tokens that are not perfectly aligned with the target response (see Fig. 1 for an example). Motivated by this insight, we explore the following question in this paper: Can we adapt verification to assess token quality rather than alignment? We draw inspiration from the LLM-as-a-judge framework, where LLMs are used to judge the quality of other model responses to user questions ( Zheng et al., 2023 ) . These judgements exhibit very strong correlation with human ratings, making this a cheap and scalable approach for model quality evaluation. Interestingly, LLM-judges display the ability to rate answers in a versatile way, allowing them to appreciate correct but potentially unaligned responses. To equip the target model with similar capabilities, we design a small dataset consisting of correct and wrong replies to user questions. We create a diverse set of responses from several models and precisely annotate the location of the mistaken tokens. We then leverage the powerful target embeddings to train a small module with the objective of predicting the correctness of a given token, mimicking the LLM-judge mechanism. In summary, we make the following contributions: • We demonstrate through a series of experiments how the decision mechanism in speculative decoding rejects many high quality tokens, identifying a key limitation of the technique. • We adapt verification using ideas from LLM-as-a-judge , eliciting the same versatile rating capability in the target by adding a simple linear layer that can be trained in under 1.5 hours. • Using a Llama 8B/70B-Judge , our approach obtains speedups of 9\times over standard decoding, achieving an unprecedented 129 tokens/s, while maintaining the quality of Llama-405B on a range of benchmarks. 2 Related Works Speculative decoding has been used and extended in a range of works, leading to significant speedups across many model families and datasets. Several ideas for draft models have been explored in the literature. Early papers rely on specialized draft models ( Sun et al., 2021 ; Xia et al., 2023 ) or smaller versions of the target model ( Chen et al., 2023 ; Leviathan et al., 2023 ) , usually trained using the same data and learning protocol.