LongSpec : Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An · 2025
Read LongSpec : Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification by Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17421.txt.
Introduction
Long documents slow language models down, because each new word must be checked against all that came before. Penghui Yang and co-authors speed that step up with a method called LongSpec, in a 2025 paper. A small draft model guesses the next words, and the big model checks them in one pass, so no output quality is lost.
Long drafts pose three snags: the draft's memory store grows, short training hurts long work, and tree checks get slow. LongSpec caps the draft's store, shifts its position codes, and merges fast prefix math with tree checks. Tests on five long-text sets ran up to 3.26 times faster than strong base lines. A hard math task also saw run time cut by more than half.
Catalog record
| CREATOR | Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17421
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LongSpec : Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An Abstract As Large Language Models (LLMs) can now process extremely long contexts, efficient inference over these extended inputs has become increasingly important, especially for emerging applications like LLM agents that highly depend on this capability. Speculative decoding (SD) offers a promising lossless acceleration technique compared to lossy alternatives such as quantization and model cascades. However, most state-of-the-art SD methods are trained on short texts (typically fewer than 4k tokens), making them unsuitable for long-context scenarios. Specifically, adapting these methods to long contexts presents three key challenges: (1) the excessive memory demands posed by draft models due to large Key-Value (KV) cache; (2) performance degradation resulting from the mismatch between short-context training and long-context inference; and (3) inefficiencies in tree attention mechanisms when managing long token sequences. This work introduces LongSpec , a framework that addresses these challenges through three core innovations: a memory-efficient draft model with a constant-sized KV cache; novel position indices that mitigate the training–inference mismatch; and an attention aggregation strategy that combines fast prefix computation with standard tree attention to enable efficient decoding. Experimental results confirm the effectiveness of LongSpec , achieving up to a 3.26 × \times speedup over strong Flash Attention baselines across five long-context understanding datasets, as well as a 2.34 × \times reduction in wall-clock time on four math reasoning tasks with the QwQ model, demonstrating significant latency improvements for long-context applications. 1 Introduction Figure 1: The SoTA SD method, EAGLE, has a training context length of 2048, which is significantly shorter than the context lengths of modern LLMs. Large Language Models (LLMs) have demonstrated remarkable capabilities Achiam et al. (2023) , and their ability to handle extensive contexts is becoming crucial for emerging applications such as LLM agents and long reasoning tasks Tan et al. (2025) ; Guo et al. (2025) , which now operate over context windows extending to millions of tokens Team et al. (2024) . In these demanding long-context scenarios, the high inference latency of standard autoregressive decoding becomes a pronounced bottleneck. While various acceleration techniques such as quantization Lin et al. (2024) , sparse attention Li et al. (2024b) , and model cascades Gupta et al. (2024) have been proposed to mitigate this problem, they often compromise the output quality, rendering them lossy solutions. In contrast, speculative decoding (SD) Leviathan et al. (2023) offers a lossless acceleration strategy by using a smaller draft model to propose token sequences, which are then verified in parallel by the larger target model. However, state-of-the-art (SoTA) SD methods Li et al. (2024c) , which often rely on a small and standalone draft model, are mainly designed and evaluated on short-context data, typically with sequences shorter than 4k tokens (see Figure 1 ). Although some existing SD methods can be extended to longer contexts, they often use the full target model with a compressed Key-Value (KV) cache as the draft model Chen et al. (2025b) ; Tiwari et al. (2025) . These approaches avoid the overhead of training a dedicated draft model, but their reliance on full target models, which are not sufficiently lightweight, limits the speed of draft generation. As a result, these methods may underperform compared to SoTA short-context SD techniques. This divergence raises a critical question: Why cannot SoTA SD methods for short contexts be directly applied to long sequences? In response to this question, we attribute the difficulty of directly adapting effective short-context SoTA SD techniques to long-context settings to three emergent challenges: 1. Architecture: In SoTA SD methods ( e.g. , EAGLE Li et al. (2024c) ), the draft model’s KV cache still grows linearly with context length. This linear growth becomes a prohibitive memory bottleneck as the context length increases. 2. Training: Language model training typically relies on plentiful short-sequence data, while long-sequence data remains relatively scarce. The imbalance of training data makes it difficult for the model to generalize to longer contexts. To address this, conventional wisdom in training long-context LLMs employs length extrapolation, in particular by extending the Rotary Position Embedding (RoPE) Su et al. (2024) base to accommodate longer contexts Gao et al. (2024) ; Liu et al. (2024c) ; Peng et al. (2024) . However, this solution is not directly applicable to SoTA SD draft models, because their RoPE base must match that of the target model 1 1 1 SoTA SD techniques often require the draft model to utilize intermediate features ( e.g. , hidden states or KV cache) from the target model, which is crucial for providing richer information from the target model, enabling the draft model to better align with and predict the target model’s outputs. See more explanations in Appendix B . , which is fixed and already scaled for long-context scenarios. 3. Inference: The effectiveness of tree attention verification Miao et al. (2024) ; Cai et al. (2024) diminishes in long-context scenarios. In particular, common inference optimizations for long-context scenarios are primarily designed to handle regular, structured attention masks and are not optimized for arbitrary or unstructured attention masks. As a result, potential speedups from speculation may be lower than expected. To address these challenges, we introduce LongSpec , a comprehensive framework for efficient long-context lossless speculative decoding. LongSpec overcomes the aforementioned obstacles through three key innovations: 1. Memory-Efficient Architecture. We propose a draft model architecture with constant memory usage regardless of context length, effectively resolving the scalability limitations of prior SoTA autoregressive draft models. 2. Effective Training Regimes. We develop a novel training strategy involving Anchor-Offset Indices, enabling draft models trained on short sequences to robustly generalize to much longer contexts at inference time. 3. Fast Tree Attention. We introduce Hybrid Tree Attention, a new computation method that significantly speeds up tree verification by decomposing attention calculations and leveraging optimized Triton kernels. Experiments on five long-context understanding datasets using five LLMs as target models show that our LongSpec can significantly reduce the long-context inference latency, achieving up to a 3.26 \times speedup over strong baselines with Flash Attention 2 2 2 In this paper, Flash Attention refers to the inference optimization technique FlashDecoding Dao et al. (2023) , implemented via the flash_attn_with_kvcache function from the Flash Attention library Dao (2022) . , and up to a 7 \times speedup over common baselines using the HuggingFace implementation. Additional experiments on four math reasoning datasets with the long reasoning model QwQ Qwen (2024) further validate the effectiveness of LongSpec , yielding a 2.34 \times speedup in wall-clock time. Furthermore, our proposed Anchor-Offset Indices enable models to reach the same loss level 3.93 \times faster, and our Hybrid Tree Attention reduces attention computation latency by approximately 75% compared to the standard HuggingFace implementation. 2 Related Work Speculative decoding offers a promising approach to accelerating LLMs without compromising the quality of their outputs 3 3 3 While this paper focuses on original speculative decoding methods which are lossless, some recent works explore lossy speculative decoding (see Appendix A for a brief overview). . Early efforts Xia et al. (2023) ; Leviathan et al. (2023) ; Liu et al. (2024d) ; Bae et al. (2023) ; Liu