Cross-Attention Speculative Decoding

Wei Zhong, Manasa Bharadwaj, Yixiao Wang, Nikhil Verma, Yipeng Ji, Chul Lee, LG Electronics, Toronto AI Lab · 2025

Read Cross-Attention Speculative Decoding by Wei Zhong, Manasa Bharadwaj, Yixiao Wang, Nikhil Verma, Yipeng Ji, Chul Lee, LG Electronics, Toronto AI Lab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24544.txt.

Introduction

Wei Zhong and four co-authors present a study from twenty twenty-five on a trick that speeds up large language models. The trick uses a small draft model to guess the next words. A big model then checks those guesses in one pass. Most draft models copy the big model's own attention style, which adds bulk and fuss.

Their draft model, called Beagle, uses cross attention instead, with no extra pooling layers. It matches the speed of a top rival draft model. A two stage training plan keeps its learning stable and fast. Tests on many models and data sets show like speed gains with better training yield. Memory use stays flat in training runs, which helps at scale.

Catalog record

CREATORWei Zhong, Manasa Bharadwaj, Yixiao Wang, Nikhil Verma, Yipeng Ji, Chul Lee, LG Electronics, Toronto AI Lab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Attention Speculative Decoding Wei Zhong, Manasa Bharadwaj, Yixiao Wang, Nikhil Verma, Yipeng Ji, Chul Lee, LG Electronics, Toronto AI Lab Abstract Speculative decoding (SD) is a widely adopted approach for accelerating inference in large language models (LLMs), particularly when the draft and target models are well aligned. However, state-of-the-art SD methods typically rely on tightly coupled, self-attention-based Transformer decoders, often augmented with auxiliary pooling or fusion layers. This coupling makes them increasingly complex and harder to generalize across different models. We present Budget EAGLE (Beagle), the first, to our knowledge, cross-attention-based Transformer decoder SD model that achieves performance on par with leading self-attention SD models (EAGLE-v2) while eliminating the need for pooling or auxiliary components, simplifying the architecture, improving training efficiency, and maintaining stable memory usage during training-time simulation. To enable effective training of this novel architecture, we propose Two-Stage Block-Attention Training, a new method that achieves training stability and convergence efficiency in block-level attention scenarios. Extensive experiments across multiple LLMs and datasets show that Beagle achieves competitive inference speedups and higher training efficiency than EAGLE-v2, offering a strong alternative for architectures in speculative decoding. 1 Introduction Speculative decoding (SD) ( Stern et al. 2018 ; Sun et al. 2021 ; Xia et al. 2022 ; Leviathan et al. 2023 ; Chen et al. 2023 ; Xia et al. 2024 ) is an effective method for accelerating inference in large language models (LLMs), where a lightweight draft model proposes the next n tokens in advance, reducing the need for multiple target model invocations. The adoption of SD has been growing in industry due to its ability to deliver lossless latency improvements in both greedy and sampling-based decoding ( Leviathan et al. 2023 ; Chen et al. 2023 ) , while also improving utilization of otherwise idle compute during memory-bound decoding phases. Implementing SD efficiently typically requires replacing generic, often misaligned draft models with dedicated ones that are co-trained alongside the target model to match its output distribution better. As a result, the design and integration of SD models pose both research and engineering challenges. From a research standpoint, identifying an effective draft model remains an open problem, nearly as difficult as designing the target LLM itself. A draft model must closely approximate the target model’s token predictions while remaining much smaller for practical deployment. Toward this goal, a range of architectures has been explored: from lightweight MLPs or FFNs ( Stern et al. 2018 ; Cai et al. 2024 ; Ankner et al. 2024 ) , to RNN-based designs ( Cheng et al. 2024 ) , to more expressive Transformer-based decoding heads ( Li et al. 2025a ; Li et al. 2024 ) , which have demonstrated superior performance over simpler FFNs ( Stern et al. 2018 ; Cai et al. 2024 ; Gloeckle et al. 2024 ) . More recently, self-attention-based autoregressive draft heads have gained traction due to their strong performance in future token prediction. Most of the latest high-performing SD systems ( Li et al. 2024 ; Ankner et al. 2024 ; Li et al. 2025 ; Xiao et al. 2024 ; Zhang et al. 2025 ) adopt a similar architecture: a self-attention layer that pools token embeddings and prior target states, often using similar design patterns and even shared code bases. Moreover, speculative decoding support remains limited in current LLM inference frameworks. At the time of writing, SGLang ( Zheng et al. 2024 ) is the only publicly available framework that supports one of the leading SD methods, EAGLE-v2, but only within the EAGLE model family ( Li et al. 2024 ; Li et al. 2025 ) . Other frameworks ( Joao Gante 2023 ; LM Studio Team 2025 ) are either still under development or rely on disconnected (and often misaligned) draft and target models, resulting in marginal or even negative speedups. Much of the complexity of importing advanced SD models stems from non-standard architectural components and extensive customization for each new model integration. Inspired by the success of the deep-shallow configuration in translation ( Kasai et al. 2020 ) , which reinterprets decoder-only LLMs as an encoder followed by a single cross-attention decoder, we explore whether a similarly minimal cross-attention structure can serve as a viable alternative to self-attention-based draft models. Unlike most existing self-attention-based SD solutions, we reduce a standard Transformer decoder ( Vaswani et al. 2023 ) to a minimal cross-attention structure without auxiliary layers. It is shown to match the performance of state-of-the-art SD models for the same training data while maintaining architectural simplicity. On the other hand, self-attention-based draft models generate queries and keys from the same hidden state, which complicates integration with autoregressive inputs and target outputs. To address this, existing work often require auxiliary pooling layers (e.g., of size 2d\times d , or even 3d\times d in EAGLE-v3 ( Li et al. 2025 ) where d is the hidden state dimension) to handle heterogeneous features. In contrast, our cross-attention architecture naturally handles different autoregressive states without pooling or custom fusion layers. Moreover, our cross-attention design enables efficient multi-token prediction during training, akin to “condensing” ( Gao & Callan 2021 ; Gao & Callan 2021a ) future token information into the draft representation. To fully exploit this, we introduce a novel Two-Stage Block-Attention Training method that makes our architecture not only simple but effective. Unlike prior Training-Time Test (TTT) based on self-attention ( Li et al. 2025 ; Zhang et al. 2025 ) , our training scheme maintains constant memory usage and avoids the need to scale hidden states with the number of simulated steps, enabling full training of a 7B model on a single 24GiB GPU. We believe that our method offers a strong alternative to existing SD architectures, combining simplicity, familiarity, and practical efficiency. 2 Related Work Multi-token prediction training and SD: Initial speculative decoding work ( Stern et al. 2018 ; Sun et al. 2021 ; Xia et al. 2022 ) concentrated on tasks such as machine translation, where parallel mappings in word space make it easier to realize substantial speed gains. These task-specific advances laid the foundation for more general-purpose SD methods ( Cai et al. 2024 ; Ankner et al. 2024 ; Gloeckle et al. 2024 ) , which leverage parallel decoding heads for broader LLM applications. A recent study ( Lindsey et al. 2025 ) demonstrates that, in poetic text, hidden states—even from early positions—may already contain information about several upcoming tokens. This insight aligns with empirical findings showing that LLM performance can be enhanced by multi-token prediction (MTP) ( DeepSeek-AI et al. 2025 ) or by adopting more challenging objectives that inject additional contextual signals into model states ( Gao & Callan 2021 ) . While multi-token prediction has shown promise in prior work, state-of-the-art SD methods ( Zhang et al. 2025 ; Li et al. 2025 ) continue to use autoregressive next-token prediction, aligning next-k tokens via step-by-step simulation during training, i.e., Training-Time Test, or TTT ( Li et al. 2025 ) . Furthermore, SD adaptation to every new LLM often entails training the SD model from scratch, amplifying the cost of training. Although self-speculative methods ( Zhang et al. 2024 ) address the co-training overhead by reusing the target model

Read the complete text