On multi-token prediction for efficient LLM inference

Somesh Mehra, Javier Alonso Garcia, Lukas Mauch · 2025

Read On multi-token prediction for efficient LLM inference by Somesh Mehra, Javier Alonso Garcia, Lukas Mauch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09419.txt.

Introduction

Somesh Mehra, Javier Alonso Garcia, and Lukas Mauch ask whether language models already know how to predict several words ahead. Models are trained to guess the next token only, which makes generation one slow step at a time. Guessing several tokens per step could speed things up.

The team shows that next-token models secretly carry multi-token skill, recoverable by averaging over paths. Bigger models do it better. But bolting prediction heads onto a frozen model works poorly, since its inner layers are tuned hard for the single next token. Training everything together helps, yet the barrier remains. Their findings chart what speed tricks can and cannot do.

Catalog record

CREATORSomesh Mehra, Javier Alonso Garcia, Lukas Mauch
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On multi-token prediction for efficient LLM inference Somesh Mehra, Javier Alonso Garcia, Lukas Mauch Abstract We systematically investigate multi-token prediction (MTP) capabilities within LLMs pre-trained for next-token prediction (NTP). We first show that such models inherently possess MTP capabilities via numerical marginalization over intermediate token probabilities, though performance is data-dependent and improves with model scale. Furthermore, we explore the challenges of integrating MTP heads into frozen LLMs and find that their hidden layers are strongly specialized for NTP, making adaptation non-trivial. Finally, we show that while joint training of MTP heads with the backbone improves performance, it cannot fully overcome this barrier, prompting further research in this direction. Our findings provide a deeper understanding of MTP applied to pretrained LLMs, informing strategies for accelerating inference through parallel token prediction. 1 Introduction In recent years, decoder-only transformers have emerged as the state-of-the-art models for language modeling and are widely adopted for large language models (LLMs). However, as these models grow in size and complexity, so do the challenges associated with their inference, due to the inherent sequential and memory bandwidth bounded nature of this process ( Kim et al. 2023 ) . In this context, there is growing interest in multi-token prediction (MTP) methods that diminish the purely auto-regressive nature of LLMs, enabling them to generate multiple adjacent tokens in parallel for a given partial input sequence. This can be considered a form of temporal sparsity, whereby a single forward pass is reused to predict multiple tokens, thus reducing sequential dependencies. MTP has shown great promise in accelerating inference through self-speculative decoding, which in practice can yield speedups of up to 3.6\times ( Cai et al. 2024 ) . While previous works have shown the feasibility of MTP, there is still a lack of understanding about the pre-requisites and limitations of MTP when applied to modern LLMs that are pre-trained for next-token prediction (NTP). Works such as Gloeckle et al. 2024 and Basharin et al. 2024 train MTP models from scratch, primarily using it as a tool to improve NTP performance. Cai et al. 2024 introduce MEDUSA, a framework for attaching MTP heads to a pretrained LLM, however they use it only for accelerating inference, and the underlying MTP performance is hidden (only speedups are reported). This paper addresses this knowledge gap by investigating MTP adaptation of NTP models. Specificaly, we make the following contributions: MTP capabilities of NTP models : We demonstrate that LLMs trained only for NTP already possess MTP capabilities, i.e. MTP is possible through numeric marginalization over intermediate token probabilities. Although this is computationally complex, it is a strong performance baseline. Model size dependence : We observe that the MTP capabilities of LLMs are strongly data dependent and improve with increasing model size. Challenges in adding MTP heads : Attaching additional MTP heads to a frozen LLM backbone is not straightforward, as hidden layers tend to specialize for NTP tasks. Joint training strategies : We explore joint training of the LLM backbone and MTP heads, which can improve performance but still falls short of the baseline, warranting further study. 2 Multi-token prediction capabilities of transformers trained for next-token prediction We delve into the multi-token prediction (MTP) capabilities of transformers trained for next-token prediction (NTP), leveraging simple information-theoretic criteria to gain deeper insights. Specifically, let \mathcal{X}=\{x_{1},x_{2},...,x_{T}\}\sim p(\mathcal{X}) be a token sequence from the true data generating distribution p(\mathcal{X}) , where x_{t}\in\mathbb{V} with vocabulary \mathbb{V} . Further, let \mathcal{X}_{\leq t}=\{x_{1},x_{2},...,x_{t}\} and \mathcal{X}_{t:t+k}=\{x_{t+1},x_{t+2},...,x_{t+k}\} denote sub-sequences. A transformer trained for NTP learns the factorization \displaystyle p(\mathcal{X};\theta)=p(x_{1})\prod_{t=1}^{T-1}p(x_{t+1}|\mathcal{X}_{\leq t};\theta)\;, (1) where \theta are the model parameters. NTP samples from p(\mathcal{X};\theta) in an auto-regressive way, i.e. generating all x_{t+1}\sim p(x_{t+1}|\mathcal{X}_{\leq t};\theta),\>\>\forall t=1,...,T-1 sequentially. On the other hand, MTP means to sample a full sub-sequence of length K tokens given the context \mathcal{X}_{\leq t} in parallel \displaystyle\mathcal{X}_{t:t+K}\sim p(\mathcal{X}_{t:t+K}|\mathcal{X}_{\leq t};\theta)\;. (2) Of course, this is intractable for the given auto-regressive factorization given in Eq. 1 . However, assuming conditional independence of the tokens within \mathcal{X}_{t:t+K} given \mathcal{X}_{\leq t} , we can compute p(\mathcal{X}_{t:t+K}|\mathcal{X}_{\leq t};\theta) , using marginalization: \displaystyle p(\mathcal{X}_{t:t+K}|\mathcal{X}_{\leq t};\theta)=p(x_{t+1}|\mathcal{X}_{\leq t};\theta)\prod_{k=2}^{K}\sum_{s_{1:k-1}\in\mathbb{V}^{k-1}}p(x_{t+k},s_{1:k-1}|\mathcal{X}_{\leq t};\theta)\;. (3) Note that in practice, this is only feasible for small k because of the sum over all possible subsequences of length K-1 . We evaluate MTP using the marginalization in Eq. 3 across different model families 1 1 1 EleutherAI’s Pythia model suite ( Biderman et al. 2023 ) and Meta’s Llama-3 models ( AI@Meta 2024 ) . and sizes using K=2 . The details of all our experiments are summarized in Appendix A . Figure 1 shows the obtained top-5 token accuracy (higher is better) for open-ended generation and translation. We see, that: 1) LLMs trained for NTP can perform MTP. 2) MTP capabilities grow with model size. 3) Performance is strongly data dependent. The improvement with scale can be attributed to a sparsification of next-token probabilities in larger models, indicating that they become more selective in their possible completions, simplifying marginalization (see Appendix B for details). The data dependence of MTP performance is also not surprising: for tasks such as translation, the provided context (i.e. the sentence in the source language) reduces ambiguity in predicting \mathcal{X}_{t:t+K} , making the predictions more deterministic. Figure 1: Top-5 accuracy of MTP using marginalization, for open-ended generation and translation across model families and sizes. MTP capabilities grow with model size and are data dependent. 3 Finetuning transformers for multi-token prediction Previous works such as MEDUSA adapt LLMs from NTP to MTP by attaching MTP heads to a pretrained model to amortize the marginalization from Eq. 3 . We take a similar approach with some key modifications: rather than attaching additional heads on top of the pretrained LLM, we replicate its final transformer layer N times to create N independent heads within the model. Each head processes the final hidden state from the shared backbone and applies a shared unembedding layer to compute token probabilities, avoiding the inefficiency of training separate per-token unembeddings. More formally, let L be the number of transformer layers in the original LLM and h be the hidden size of the model. Let z^{L-1}_{1:t}\in\mathbb{R}^{h\times t} be the hidden states from the penultimate transformer layer of the original model given the input sequence \mathcal{X}_{\leq t} . Define f^{L}_{n}:\mathbb{R}^{h}\to\mathbb{R}^{h} as the final transformer layer for the n^{\text{th}} head, and f_{u}:\mathbb{R}^{h}\to\mathbb{R}^{|\mathbb{V}|} as the unembedding to map hidden representations to logits. The n^{\text{th}} token probabilities are thus given by \displaystyle p(x_{t+n}|\mathcal{X}_{\leq t};\theta)=\sigma\big(f_{u}(f_{n}^{L}(z^{L-1}_{1:t}))\big)\>, (4) where n\in\{1,2,...N\} and \sigma(\cdot) is the softmax function. Further details and a visualization of this architecture are available in Appendix C . To simplify analysis, we use N=2 . We train the heads, with the backbone and unembedding frozen, on a translation task due to its suitability for MTP. Full training details are outlined in Appendix A . Our experiments show that this approach is not straightforward due to an early specialization of the LLM for NTP. To visualize this specialization, we compute the Kullback-Leibler (KL) divergence between intermediate token probabilities and the final token probabilities. More specifically, let z^{\ell}_{t}\in\mathbb{R}^{h}\text{ with }\ell=1,2,...,L-1 be the output of layer \ell at sequence position t . We compute the intermediate token probabilities p_{\ell}(x_{t+1}|\mathcal{X}_{\leq t};\theta) by applying the final unembedding layer directly to z^{\ell}_{t} , i.e. p_{\ell}(x_{t+1}|\mathcal{X}_{\leq

Read the complete text