Alternatives To Next Token Prediction In Text Generation - A Survey
Charlie Wyatt, Aditya Joshi, Flora Salim · 2025
Read Alternatives To Next Token Prediction In Text Generation - A Survey by Charlie Wyatt, Aditya Joshi, Flora Salim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24435.txt.
Introduction
Predicting the next token has powered large language models, and it also causes their worst habits: weak long-term planning, compounding errors, and slow generation. Charlie Wyatt and 2 co-authors survey the growing family of alternatives. They sort the field into five families, from predicting blocks of tokens at once to planning first and generating after.
Other families move reasoning into a continuous latent space, replace step-by-step output with parallel refinement such as diffusion, or drop the transformer shape entirely. The survey knits these threads into one taxonomy. Its aim is to guide work on models that keep the strengths of token generation while shedding its known limits.
Catalog record
| CREATOR | Charlie Wyatt, Aditya Joshi, Flora Salim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24435
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Alternatives To Next Token Prediction In Text Generation - A Survey Charlie Wyatt, Aditya Joshi, Flora Salim Abstract The paradigm of Next Token Prediction (NTP) has driven the unprecedented success of Large Language Models (LLMs), but is also the source of their most persistent weaknesses such as poor long-term planning, error accumulation, and computational inefficiency. Acknowledging the growing interest in exploring alternatives to NTP, the survey describes the emerging ecosystem of alternatives to NTP. We categorise these approaches into five main families: (1) Multi-Token Prediction , which targets a block of future tokens instead of a single one; (2) Plan-then-Generate , where a global, high-level plan is created upfront to guide token-level decoding; (3) Latent Reasoning , which shifts the autoregressive process itself into a continuous latent space; (4) Continuous Generation Approaches , which replace sequential generation with iterative, parallel refinement through diffusion, flow matching, or energy-based methods; and (5) Non-Transformer Architectures , which sidestep NTP through their inherent model structure. By synthesizing insights across these methods, this survey offers a taxonomy to guide research into models that address the known limitations of token-level generation to develop new transformative models for natural language processing. 1 Introduction Large Language Models (LLMs) have achieved remarkable performance across a wide range of natural language processing (NLP) tasks, from question answering and summarization to creative writing and coding Hendrycks et al. 2021 ; Srivastava et al. 2023 ; Jiang et al. 2024 . At the heart of this success lies a simple yet powerful objective: Next Token Prediction (NTP) Vaswani et al. 2023 . Defined as the task of predicting the next subword token given the previous tokens in a sequence, NTP has become the dominant training and decoding paradigm for modern Transformer models. Yet, despite their fluency, LLMs often exhibit brittle behavior: they hallucinate facts, lose coherence in long outputs ( Maharana et al. 2024 ) , and struggle with tasks that require long-term planning or structured reasoning ( Hao et al. 2023 ) . These limitations persist even as model scale, context length, and training data continue to grow. A growing body of work suggests that these failures are not incidental but symptomatic of deeper structural problems with the next-token prediction framework itself. As Bachmann and Nagarajan 2024 argue, the token-level chain rule that NTP relies on can break down, as the probability of the next most likely token does not guarantee the start of the most probable overall sequence . This survey examines the emerging ecosystem of alternatives to next-token prediction . We argue that many core weaknesses of LLMs - greedy decoding, error accumulation, misaligned supervision, granularity mismatches, and computational inefficiencies - are downstream of the NTP objective. As Figure 1 illustrates, a diverse set of methods now reimagine the prediction target, shifting away from token-by-token generation (as in the case of NTP) toward richer and more global semantic representations. While we organize these into five main families for clarity, emerging variants like flow matching and energy-based models represent important extensions within these categories, particularly in continuous generation approaches. 2 Motivation This section outlines the core limitations of NTP and motivates the need for alternative decoding and training paradigms. As is evident from the discussion, the use of NTP poses challenges in several aspects of language modeling itself. Figure 1: An abstracted visualization of Next Token Prediction and its alternatives 2.1 Inference-Time Failures: Greedy Generation and Error Accumulation NTP is an autoregressive generation strategy, i.e. , tokens are generated one at a time where the probability of the next token depends only on the existing context. Formally, given a sequence of tokens x=(x_{1},x_{2},\dots,x_{T}) , the model is trained to maximize the likelihood: P(x)=\prod_{t=1}^{T}P(x_{t}\mid x_{<t}), (1) where x_{<t} denotes all tokens preceding the position t and x_{t} denotes the token at the position t . While this enables fluent local continuations, it also encourages greedy generation —a process that myopically optimizes for short-term token likelihood rather than long-range coherence. This greediness can lead to error accumulation Rohatgi et al. 2025 , where minor inaccuracies early in generation compound into significant incoherence or factual drift later in the sequence. These issues are especially pronounced in long-form generation tasks such as summarization and storytelling. As observed in surveys on abstractive summarization ( Nnadi and Bertini 2024 ) , even strong transformer models often struggle with maintaining global consistency and factual grounding across lengthy generations. Although retrieval-augmented generation and extended context windows mitigate some of these issues, they do not fundamentally address the underlying problem: generation proceeds locally , without a representation of, or commitment to, a global plan. Indeed, despite near-perfect performance on “needle-in-a-haystack” retrieval tasks (e.g., Gemini 1.5 ’s >99\% accuracy ( Gemini Team et al. 2024 ) ), LLMs still underperform at generating coherent long-document summaries. As shown by Maharana et al. 2024 , LLMs struggle with long-context understanding and temporal grounding, even when equipped with large context windows. A similar phenomenon has been documented outside pure text generation, where the model’s step-wise choices reveal a preference for short-term gains. Recent studies in decision-making have further characterized this failure mode as greediness in action selection , where LLMs prematurely commit to locally optimal continuations rather than exploring alternate trajectories ( Schmied et al. 2025 ) . This behaviour reflects a broader limitation in NTP-based generation: the inability to revise earlier decisions in light of later ones, or to reason over long-term structure during decoding. 2.2 Granularity Mismatch: Tokens vs. Ideas Another potential shortcoming of NTP is its mismatch in granularity: NTP operates over subword tokens, but human reasoning unfolds over ideas, sentences, and discourse structures . Modern LLMs typically rely on subword tokenization, most commonly the Byte-Pair Encoding (BPE) algorithm ( Sennrich et al. 2016 ) . While effective for handling large vocabularies, BPE fragments text into units that are neither linguistically natural nor semantically meaningful. In contrast, human communication is organized around higher-level structures such as phrases, sentences, and discourse segments. As a result, tasks like summarization or dialogue understanding—which require reasoning over ideas and maintaining global coherence—are forced to operate on token sequences that fail to reflect these natural semantic boundaries. This granularity misalignment also manifests in unexpected and undesirable behaviors at the token level, such as the well-documented handling of ‘glitch tokens" like solidgoldmagikarp ( Land and Bartolo 2024 ) , which arise from under-training or limited exposure in the corpus. Although recent work has proposed methods to detect and mitigate these under-trained tokens, the fact that such ad hoc fixes are needed suggests that sub-word tokens lack the right level of abstraction for language understanding. Beyond these outlier cases, it also leads to surprisingly poor performance on trivial character-level tasks. Cosma et al. 2025 and Fu et al. 2024 demonstrate that tokenized language models can stumble on rudimentary character-counting tasks such as answering the question “How many r’s are in strawberry?" . Similarly, autoregressive models fail to plan ahead on even the simplest task: ask an LLM “ How many words are in your response? " and it will almost always miscount. Since the model generates each word step-by-step, the model cannot see ahead and adjust its plan accordingly. These failures collectively indicate a structural mismatch: subword