Efficient Joint Prediction of Multiple Future Tokens
Kwangjun Ahn, Alex Lamb, John Langford · 2025
Read Efficient Joint Prediction of Multiple Future Tokens by Kwangjun Ahn, Alex Lamb, John Langford (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21801.txt.
Introduction
Kwangjun Ahn, Alex Lamb, and John Langford introduce a light tweak to language model training that predicts several future tokens at once, in a short report from 2025. Standard training predicts only the next token. Richer targets can force the model's inner state to hold a fuller picture of what lies ahead.
Their method feeds true future tokens through a tight bottleneck, so the model must pack rich forecasts into its hidden state. That design yields a short-horizon belief state, a compact summary of likely continuations. Popular rival methods fail to form such a state. On a synthetic star graph navigation task, the new method beat those rivals by a wide margin. The authors present the result as an early, promising signal.
Catalog record
| CREATOR | Kwangjun Ahn, Alex Lamb, John Langford |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Joint Prediction of Multiple Future Tokens Kwangjun Ahn, Alex Lamb, John Langford Abstract In this short report, we introduce joint multi-token prediction (JTP), a lightweight modification of standard next-token prediction designed to enrich hidden state representations by jointly predicting multiple future tokens. Unlike previous multi-token prediction approaches, JTP strategically employs teacher forcing of future-tokens through a carefully designed representation bottleneck, allowing the model to encode rich predictive information with minimal computational overhead during training. We show that the JTP approach achieves a short-horizon belief state representation, while popular alternatives for multi-token prediction fail to do so. We demonstrate the effectiveness of our method on the synthetic star graph navigation task from Bachmann and Nagarajan 2024 , highlighting a significant performance improvement over existing methods. This manuscript presents promising preliminary results intended to stimulate further research. 1 Introduction Standard large language models are pretrained using the next-token prediction objective, which, by the chain rule, allows them to model any distribution over sequences: \Pr(x_{0},...,x_{T})=\Pi_{t=0}^{T}\Pr(x_{t}\mid x_{0},...,x_{t-1}) . However, in practice, this theoretical expressivity is not always realized due to limitations in the model’s representational capacity or the difficulty of reaching an optimal solution through gradient-based optimization ( Bachmann and Nagarajan 2024 ; Abbe et al. 2024 ; Hu et al. 2024 ) . In this short report, we propose joint multi-token prediction (JTP), an exceptionally efficient solution to the limitations of next-token prediction. JTP aims to enrich the hidden representation of the model by predicting the joint distribution of multiple future tokens over a sliding window of size D . Our proposed scheme carefully controls the information flow to effectively enrich the hidden representation—unlike existing multi-token prediction schemes ( Gloeckle et al. 2024 ; DeepSeek-AI 2024 ) . This design enables model representations to converge on solutions that are otherwise inaccessible. Notably, JTP introduces only a minimal additional component to the transformer architecture, resulting in negligible overhead while enabling the model to solve problems that conventional next-token prediction cannot. Importantly, this manuscript aims to share a promising work-in-progress idea with the research community to encourage further exploration. At this stage, we primarily evaluate our approach on the synthetic star graph navigation task from Bachmann and Nagarajan 2024 , a simple task that is known to fail standard next-token predictors. Our experimental results highlight a striking performance gap between our method and existing multi-token prediction approaches. In Section 2 , we discuss how JTP works in more depth. In Section 3 , we then experiment with the approach showing it can solve small problems which simple next token prediction, and alternative methods of multitoken prediction cannot. In Section 4 , we provide some theoretical discussion covering the computational cost and capabilities of JTP. We close with Section 5 showing that optimizing JTP and the next token objective are largely compatible on text datasets. 2 Efficiently Predicting Joint Distributions of Future Tokens Figure 1 : Illustration of multi-token prediction mechanisms at position t=3 . The method of Gloeckle et al. 2024 (left) independently predicts future tokens through a single representation bottleneck, neglecting dependencies between the future tokens. DeepSeek-V3 ( DeepSeek-AI 2024 ) (middle) processes each token prediction through multiple layers considering the entire historical context, bypassing the desired bottleneck and thus diminishing representation enrichment. Dotted arrows indicate teacher-forcing dependencies. In contrast, our proposed method (right) efficiently funnels predictive information through a single representation bottleneck while utilizing teacher-forced tokens (dotted arrows), thus preserving token dependencies without compromising representation richness or computational efficiency. In this section, we first introduce our main method in more generality in Section 2.1 , and compare our approach with existing multi-token prediction methods in Section 2.2 . We then present an instantiation of our method in Section 2.3 . Let \mathbf{h}_{0:T-1} denote the hidden states produced by the main Transformer model. Typically, these hidden states are passed through the output head for next-token prediction. The standard next-token prediction loss is given by: \displaystyle\text{Next-Token Loss}~=~\frac{1}{T}\sum_{t=1}^{T}\mathcal{L}_{\mathrm{NTP}}(x_{t}\mid\mathbf{h}_{t-1})\,, where each term is defined as: \displaystyle\mathcal{L}_{\mathrm{NTP}}(x_{t}\mid\mathbf{h}_{t-1})~:=~-\log\mathrm{head}\bigl(\mathbf{h}_{t-1}\bigr)[x_{t}]. Here, \mathrm{head} processes the hidden state \mathbf{h}_{t-1} and outputs the logits for predicting the token x_{t} . Our goal is to enrich the hidden states \mathbf{h}_{0:T-1} by predicting not only the next token x_{t} but also several future tokens x_{t+1},\ldots,x_{t+D} . To achieve this, we incorporate a multi-token prediction (MTP) component alongside the standard next-token prediction (NTP) objective: \displaystyle\text{Training Loss}~=~\text{Next-Token Loss}~+~\lambda\cdot\underbrace{\frac{1}{T}\sum_{t=1}^{T}\mathcal{L}_{\mathrm{MTP}}(x_{t+1:t+D}\mid\mathbf{h}_{t-1})}_{\text{Multi-Token Loss}}\,. Here, \lambda>0 balances immediate and future predictions. Next, we detail the design of the multi-token prediction loss. 2.1 Joint Multi-Token Prediction (JTP) Ideally, \mathcal{L}_{\mathrm{MTP}} models the joint distribution of the D future tokens (x_{t},x_{t+1},\ldots,x_{t+D}) given \mathbf{h}_{t-1} . This contrasts with the method of Gloeckle et al. Gloeckle et al. 2024 , where each future token’s marginal distribution is predicted independently (see Section 2.2 for details). However, naive joint modeling of (x_{t},x_{t+1},\ldots,x_{t+D}) is excessively expensive because the support grows on the order of \lvert V\rvert^{D} . To address this complexity, we apply a teacher-forcing strategy that breaks down the joint distribution via the chain rule. Specifically, we factor \mathcal{L}_{\mathrm{MTP}}(x_{t+1:t+D}\mid\mathbf{h}_{t-1}) into a sum of conditionals, each of which can leverage teacher-forced tokens: \displaystyle\mathcal{L}_{\mathrm{MTP}}(x_{t+1:t+D}\mid\mathbf{h}_{t-1})\;=\;\frac{1}{D}\sum_{i=1}^{D}\mathcal{L}^{i}_{\mathrm{MTP}}\bigl(x_{t+i}\,\mid\,\mathbf{h}_{t-1},\,x_{t:t+i-1}\bigr). (1) A naive implementation of teacher-forcing, however, risks placing too much emphasis on the teacher-forced tokens x_{t:t+i-1} . This undermines our goal of enriching the hidden state \mathbf{h}_{t-1} with multi-step “planning” information, since the model could simply rely on those forced tokens and fall back to the next-token prediction mode. To encourage \mathbf{h}_{t-1} to capture future information, we introduce a lightweight processing module. Instead of passing teacher-forced tokens directly into the main Transformer, we process them through a light component, denoted \mathsf{Fetch} . This module refines the hidden state \mathbf{h}_{t-1} using the teacher-forced tokens x_{t:t+i-1} to extract relevant information for predicting x_{t+i} : \displaystyle\mathsf{Fetch}_{x_{t:t+i-1}}(\mathbf{h}_{t-1})\quad\text{extracts information about }x_{t+i}. The output of \mathsf{Fetch} is then fed into an MTP head, \mathrm{head}_{\mathrm{MTP}} , which performs the prediction. Formally, \displaystyle\mathcal{L}^{i}_{\mathrm{MTP}}\bigl(x_{t+i}\,\mid\,\mathbf{h}_{t-1},\,x_{t:t+i-1}\bigr)\;:=\;-\log\mathrm{head}\Bigl(\mathsf{Fetch}_{x_{t:t+i-1}}\bigl(\mathbf{h}_{t-1}\bigr)\Bigr)[x_{t+i}]. By structuring the model this way, the hidden state \mathbf{h}_{t-1} remains the primary carrier of future-planning information, leading to richer representations. We refer to this approach as joint multi-token prediction ( JTP ). We discuss a concrete instantiation of our proposal in Section 2.3 after first highlighting how our proposal differs from existing multi-token prediction schemes next. 2.2 Comparison with Existing MTPs Notable existing approaches for multi-token prediction include the frameworks proposed in Gloeckle et al. 2024 and DeepSeek-AI 2024 . Each of these exhibits shortcomings for the purpose of representation enrichment. (They of course have other virtues.) Gloeckle et al. 2024 . Their method independently predicts each of the next D tokens from the same hidden state, capturing only the marginal distribution of each token rather than a coherent joint distribution. Because modeling marginals can require strictly less information than modeling the full joint distribution, the hidden state need not encode all the multi-token dependencies. This undermines the goal of forcing a richer hidden representation. For instance, consider a short sequence (x_{1},x_{2},x_{3},x_{4},x_{5},x_{6},x_{7}) where x_{1} , x_{2} , x_{3} , x_{4} are sampled uniformly between 0 and 1 independently. Then define: (x_{5},x_{6},x_{7})\;=\;\begin{cases}(\,x_{1},\,x_{2},\,x_{3})&\text{if }x_{4}=1,\\ (\,1-x_{1},\;1-x_{2},\;1-x_{3})&\text{if }x_{4}=0.\end{cases} While the marginal distribution of each future token remains uniform, correctly predicting them jointly requires encoding (x_{1},x_{2},x_{3}) in the hidden state. The marginal MTP approach can ignore these dependencies and still match marginals, thus failing to