Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao · 2025
Read Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios by Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20340.txt.
Introduction
Shi, Li, Zhang, and colleagues make speculative decoding survive the batching era. Their 2025 paper presents SpecFormer for faster language model inference. Speculative decoding drafts tokens cheaply, then verifies them in bulk. Classic drafters build big autoregressive guess trees. Batched serving eats the idle compute those trees need.
Huge batches leave little slack for fancy drafting. SpecFormer drafts with a hybrid attention design instead. One attention runs one-way over the full history. Another runs both ways for parallel drafting. Draft tokens emerge in parallel, not one by one. No giant prefix tree is required anymore. Accuracy stays high under a tight token budget. Experiments spanned models of many sizes. Acceleration held as batch sizes grew large. Training demands and compute costs both dropped. Lossless decoding guarantees output quality is unchanged. Serving stacks gain speed without surrendering throughput.
Catalog record
| CREATOR | Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao Abstract Speculative decoding accelerates LLM inference by utilizing otherwise idle computational resources during memory-to-chip data transfer. Current speculative decoding methods typically assume a considerable amount of available computing power, then generate a complex and massive draft tree using a small autoregressive language model to improve overall prediction accuracy. However, methods like batching have been widely applied in mainstream model inference systems as a superior alternative to speculative decoding, as they compress the available idle computing power. Therefore, performing speculative decoding with low verification resources and low scheduling costs has become an important research problem. We believe that more capable models that allow for parallel generation on draft sequences are what we truly need. Recognizing the fundamental nature of draft models to only generate sequences of limited length, we propose SpecFormer, a novel architecture that integrates unidirectional and bidirectional attention mechanisms. SpecFormer combines the autoregressive model’s ability to extract information from the entire input sequence with the parallel generation benefits of non-autoregressive models. This design eliminates the reliance on large prefix trees and achieves consistent acceleration, even in large-batch scenarios. Through lossless speculative decoding experiments across models of various scales, we demonstrate that SpecFormer sets a new standard for scaling LLM inference with lower training demands and reduced computational costs. Code — hhttps://github.com/ShiLuohe/SpecFormer Introduction Large language models (LLMs) that utilizes Transformer Decoders have rapidly become the industry standard in recent years, owing to their favorable properties such as scalability in training and lossless handling of long-context dependencies ( 21 ) . Nevertheless, these models continue to follow the conventional sequence-to-sequence generation paradigm: autoregressive decoding. Autoregressive decoding refers to the process where tokens are generated one at a time; each newly generated token is fed back into the model as input for the next step, alongside the existing context, to perform another forward pass. This paradigm offers several notable advantages. With only a causal mask, the attention mechanism can be readily adapted for generation tasks, making training straightforward. It also allows for the generation of virtually unlimited-length outputs and enables acceleration through state caching for repeated input prefixes ( 23 ) , and some corresponding acceleration opportunities ( 12 ; 35 ; 40 ; 39 ; 25 ) . However, during inference, generating one token at a time results in low arithmetic intensity (AI, 30 ). Once a datum fetched from memory into the chip, it contributes to only a few operations. In contrast, chips can perform hundreds of operations in the time, leading to substantial underutilization of compute resources. On the infrastructure side, techniques such as prefill-decoding (PD) separation ( 42 ) and continuous batching ( 36 ; 17 ) have been introduced to improve overall compute utilization and user experience. Nonetheless, under constraints imposed by service-level objects (SLOs, 29 ), these techniques often fall short of leveraging the full computational potential. Increasing AI, the ratio of computation to data transfer, is the fundamental approach to enhancing hardware efficiency during generation. Speculative decoding (SD, 33 ) is one of the most effective approaches for improving AI. Its core idea is to generate multiple tokens per pass of the large model. The process consists of three main steps ( 24 ) : 1. Multi-token generation : Based on information from the previous forward pass, the model samples multiple draft tokens. 2. Multi-token verification : The model evaluates all draft tokens simultaneously to determine whether each one aligns with its own top prediction, while also extracting and storing information for the next round of multi-token generation. 3. Multi-token acceptance : The model decides whether to accept the draft tokens based on the verification results and accordingly updates the contextual information. Multi-token generation is the most critical component of SD, as the acceptance rate of the sampled drafts directly determines how effectively computational resources are utilized. In this work, we focus specifically on lossless SD , which adheres to two strict conditions: 1. Only draft tokens that exactly match the outputs of the large model are accepted. 2. The LLM itself must remain unmodified. These constraints make a purely acceleration-oriented SD, ensuring strict mathematical equivalence with the original model outputs. A key observation about SD is that it does not reduce (usually increases significantly) the total amount of computation. The acceleration arises from repurposing compute capacity that would otherwise be idle while waiting for data transfer. In other words, every SD-based method has a theoretical upper bound on speedup, corresponding to the full utilization of previously wasted compute. It is important to note that continuous batching is also a method for reducing idle compute. Consequently, in batched settings where unused compute capacity is already diminished, speculative decoding methods face a stricter efficiency requirement. Current SD methods can be broadly categorized into autoregressive and non-autoregressive approaches ( 13 ) . They all generate draft tokens in time that scales linearly with the number of draft tokens, whether through autoregression or by accessing different parameters. However, we’ve noticed a significant issue with current methods: their incompatibility with large batch sizes. A larger batch size means each parameter experiences higher computational intensity, as it’s read and reused multiple times. This, in turn, reduces the computational cycles available for draft tokens. In other words, under batch processing conditions, there are twice as many draft tokens to process in less available time. This point is illustrated in Figure 1 , where we show that when processing the same number of tokens while varying the batch size, the speed at which peak computational power is reached accelerates with increasing batch size. The number of draft tokens we can use is less than what’s needed to reach peak performance. This means that in a batched environment, the usable draft tree size is rapidly compressed. Given that online batching has been widely adopted by mainstream inference frameworks, current speculative decoding methods must adapt to scarcer resources. This implies they can no longer rely on traditional, massive draft trees, but instead need to focus on higher accuracy drafts themselves. Mainstream models face a significant challenge in this regard: they struggle to efficiently scale to larger sizes to displace capabilities. This is due to their excessive position-dependent parameters—whether these are parameters that need to be repeatedly accessed for each position in AR methods, or parameters separately allocated for each position in non-AR methods. The problem with position-dependent parameters is that scaling up exponentially increases additional costs because multiple tokens are required in a sequence. This means these methods not only inherently use more computational power but are also highly sensitive to changes in available compute. Therefore, we aim to improve the performance of SD under low draft token budgets, by directly enhancing the capability of the draft generation model. This enables SD to be effectively applied in batched inference settings. To avoid fine-tuning the original LLM, the draft model must receive sufficiently rich input information. To this end, we employ a context causal attention to extract contextual information from the hidden states of the input sequence. We observe that in traditional approaches, the parameters used for draft generation are position-dependent, i.e., generating each