SPIRe: Boosting LLM Inference Throughput with Speculative Decoding

Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Akshay Mishra, Reiner Pope · 2025

Read SPIRe: Boosting LLM Inference Throughput with Speculative Decoding by Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Akshay Mishra, Reiner Pope (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06419.txt.

Introduction

Large language models answer one token at a time, and that step by step pace sets the cost of running them. Sanjit Neelam and four co-authors present SPIRe in 2025, a draft model meant to speed up a shortcut called speculative decoding. In that shortcut a small model guesses the next words and the big model checks them, but gains shrink when many requests run at once.

SPIRe mixes sparse attention, a pruned starting point, and a feedback memory to keep its guesses sharp in big batches. The team reports throughput gains of more than double over a much smaller draft model, and over thirty-five percent above a strong sparse baseline. Gains were largest when request lengths varied widely.

Catalog record

CREATORSanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Akshay Mishra, Reiner Pope
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPIRe: Boosting LLM Inference Throughput with Speculative Decoding Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Akshay Mishra, Reiner Pope Abstract Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3× for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model’s KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests. 1 Introduction Speculative decoding uses a cheap draft model to generate candidate tokens that are verified in parallel by an expensive target model ( Leviathan et al. 2023 ; Chen et al. 2023 ) . Accepted draft tokens are guaranteed to be samples from the target model’s distribution over the next token by a modified rejection sampling scheme. If draft tokens are generated quickly enough and have a high enough acceptance rate, speculative decoding improves both the latency and throughput of decoding by reducing the number of memory accesses needed to generate each token 1 1 1 The cost is that speculative decoding always requires performing more FLOPs than autoregressive decoding; the draft model performs forward passes, and the target model performs forward passes on not only the draft tokens that are accepted but also those that are rejected. Thus, speculative decoding only makes sense if autoregressive decoding is sufficiently memory-bound. . Optimizing speculative decoding for higher throughput requires different strategies than optimizing for lower latency; the latter is well-documented in the literature ( Xia et al. 2024 ; Miao et al. 2024 ; Yan et al. 2025 ) . For example, low latency can often be achieved with small batch sizes ( Pope et al. 2022 ) and small draft models are preferred in this setting since fetching weights is the bottleneck. On the other hand, high throughput requires large batch sizes, which enables the use of larger draft models since in this regime loading the KV cache is the bottleneck ( Chen et al. 2024 ) . See Figure 1 and Figure 4 for examples of how the speedup due to speculative decoding varies with draft model architecture, batch size, and context length according to our performance model below . Figure 1: Speedup of generating tokens in a batch of size B=64 given a context of length L . A round of speculation is the generation and subsequent verification of each block of k=4 draft tokens, the average generation length \tau is the average number of tokens generated per round of speculation, and speedup is \tau divided by how much longer a round of speculation takes compared to autoregressively decoding one token. Prior work has shown that the acceptance rate of draft tokens can be increased using the target model’s intermediate activations and output logits ( Du et al. 2024 ; Zhou et al. 2024 ) . Since a target model’s state can be cached during training at negligible cost, and since inference rather than training accounts for most of the cost of serving an LLM, the investment in training a draft model aligned to a particular target model may be worthwhile. Indeed, we show in Cost Analysis that this is the case for SPIRe under conservative assumptions. Our work, along with Chen et al. 2024 , demonstrates that speculative decoding is an effective technique for reducing the cost of decoding, and our work enables more efficient exploration of the design space of draft models based on the batch sizes and context lengths expected in production. Our main contributions are the following: • An implementation-agnostic model for evaluating the throughput of speculative decoding with different draft models. • An efficient method for training a Feedback Transformer draft model. • A draft model, SPIRe, that increases the modeled throughput of speculative decoding by over 100% compared to vanilla speculative decoding and by over 35% compared to the strong baseline of MagicDec ( Chen et al. 2024 ) . 2 Evaluation Metric It is problematic to compare different methods of training the draft model using the throughput achieved by the resulting models in a production environment. Measured throughput depends on myriad implementation details such as use of specialized kernels ( Dao et al. 2022 ) , how models are partitioned across chips ( Pope et al. 2022 ) , how queries are batched ( Daniel et al. 2023 ) , and which queries are used for evaluation ( Zhou et al. 2024 ) . Any of these variables could be confounders when comparing different draft model architectures. To derive a proxy for measured throughput, we first note that the throughput of speculative decoding is equal to the throughput of vanilla decoding multiplied by \displaystyle\begin{array}[]{c}\text{Throughput}\\ \text{Multiplier}\end{array} \displaystyle=\frac{\mathbb{E}[\text{\# Tokens Generated per Round of Speculation}]}{\frac{k\times t_{\text{draft}}+t_{\text{verify}}}{t_{\text{target}}}} where k is the maximum speculation depth , a round of speculation is the generation and subsequent verification of each block of k draft tokens, t_{\text{draft}} and t_{\text{target}} are the latencies of a draft and target model forward pass on a single token, and t_{\text{verify}} is the latency of a target model forward pass on k+1 draft tokens in parallel. In the sequel, we refer to the average generation length \tau:=\mathbb{E}[\text{\# Tokens Generated per Round of Speculation}] and the iteration time multiplier \Delta t:=\frac{k\times t_{\text{draft}}+t_{\text{verify}}}{t_{\text{target}}} , so that the throughput multiplier is simply \tau/\Delta t . To obtain a more implementation-agnostic expression for \Delta t , we can break down the cost of a forward pass into the compute cost and the memory cost, which are the number of FLOPs and the number of memory accesses that need to be performed. The memory cost can be further broken down into the size of the LLM’s weights and the size of its KV cache. By multiplying the memory cost in bytes with the hardware operational intensity (HOI) in FLOPs per byte, we can obtain an estimate of the memory cost in FLOP-equivalents ( MatX 2025 ) and compare the compute and memory costs on equal footing. Assuming that computation and memory accesses are maximally overlapped 2 2 2 Optimized inference stacks with e.g. pipeline parallelism approach this ideal. Throughput multipliers are higher than if we assumed that computation and memory accesses could not be overlapped, in which case we would have \text{Decode Cost}=\text{Compute Cost}+\text{Memory Cost} . , we obtain the following expression for the cost of a forward pass. Forward Cost \displaystyle=\max(\underbrace{\text{Compute Cost}}_{\text{FLOPs}},\underbrace{\underbrace{(\text{Weight Cost}+\text{KV Cache Cost})}_{\text{Bytes}}\times\text{HOI}}_{\text{Memory Cost (FLOP-equivalents)}}) Finally, by substituting t_{\text{draft}} , t_{\text{verify}} , and t_{\text{target}} with their corresponding forward costs, we obtain the following implementation-agnostic 3 3 3 The inclusion of HOI in the throughput multiplier makes it hardware-dependent, but this only makes a difference in the compute-bound (bottom-left) region of Figure 4 . When decoding and verification are both memory-bound, HOI cancels out. approximation of the throughput multiplier and use it as our

Read the complete text