How Transformers Learn to Plan via Multi-Token Prediction
Jianhao Huang Zhanpeng Zhou Renqiu Xia Baharan Mirzasoleiman, Weijie Su Wei Huang, [0.8em] University of California, Los Angeles Shanghai Jiao Tong University · 2026
Read How Transformers Learn to Plan via Multi-Token Prediction by Jianhao Huang Zhanpeng Zhou Renqiu Xia Baharan Mirzasoleiman, Weijie Su Wei Huang, [0.8em] University of California, Los Angeles Shanghai Jiao Tong University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11912.txt.
Introduction
Huang and five colleagues study how multi-token prediction helps language models plan, versus next-token prediction. On synthetic graph path-finding and realistic tests like Countdown and boolean satisfiability, multi-token prediction consistently beats next-token prediction. Theory on a simplified two-layer Transformer and a star graph task shows why: multi-token prediction drives a two-stage reverse process, first attending to the end node, then tracing nodes backward. That pattern stems from gradient decoupling, which gives a cleaner training signal, biasing optimisation toward robust, interpretable reasoning circuits.
Catalog record
| CREATOR | Jianhao Huang Zhanpeng Zhou Renqiu Xia Baharan Mirzasoleiman, Weijie Su Wei Huang, [0.8em] University of California, Los Angeles Shanghai Jiao Tong University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82393 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11912
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Transformers Learn to Plan via Multi-Token Prediction Jianhao Huang Zhanpeng Zhou Renqiu Xia Baharan Mirzasoleiman, Weijie Su Wei Huang, [0.8em] University of California, Los Angeles Shanghai Jiao Tong University Abstract While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks. Multi-token prediction (MTP) has recently emerged as a promising alternative, yet its underlying mechanisms remain poorly understood. In this paper, we study how MTP facilitates reasoning, with a focus on planning. Empirically, we show that MTP consistently outperforms NTP on both synthetic graph path-finding tasks and more realistic reasoning benchmarks, such as Countdown and boolean satisfiability problems. Theoretically, we analyze a simplified two-layer Transformer on a star graph task. We prove that MTP induces a two-stage reverse reasoning process: the model first attends to the end node and then reconstructs the path by tracing intermediate nodes backward. This behavior arises from a gradient decoupling property of MTP, which provides a cleaner training signal compared to NTP. Ultimately, our results highlight how multi-token objectives inherently bias optimization toward robust and interpretable reasoning circuits. 1 Introduction Recently, large language models (LLMs) have demonstrated a range of emerging abilities as they scale, with reasoning becoming one of the most critical capabilities ( 18 ) . Reasoning enables language models to go beyond simple knowledge retrieval, allowing them to actively process and infer solutions to new problems. Despite these advances, how such reasoning abilities emerge in language models remain poorly understood. Recent theoretical advances focused on the role of chain-of-thought ( 7 ; 19 ; 15 ) and architecture design ( 10 ; 30 ) in enabling reasoning. However, a more fundamental factor may lie in the modeling paradigm itself, particularly the training objective that underlies language model training. Next-token prediction (NTP) with teacher forcing has long been the standard objective for training language models, where models are optimized to produce the next token given all preceding ground-truth context tokens. Yet, it is limited in capturing long-term dependencies ( 27 ) and prone to overfitting to local patterns ( 2 ) . To address these limitations, multi-token prediction (MTP) ( 12 ) has recently been proposed. Instead of predicting only the next token, it predicts multiple future tokens in parallel. MTP substantially improves performance on math and code benchmarks that require complex reasoning, and has been widely adopted in leading models such as DeepSeek-V3 ( 21 ) . Understanding how MTP leads to such improvements is crucial for uncovering the mechanisms underlying the reasoning capabilities of modern language models. (a) (b) Figure 1: Overview of the MTP architecture and its advantage in reasoning tasks. (a) The MTP architecture. Building upon 12 , our MTP employs a shared backbone with multiple independent output heads to predict several future tokens simultaneously. (b) Illustration of Star Graph and Binary Tree. In standard forward planning, early steps can be difficult due to a large search space. MTP facilitates “reverse reasoning” by looking ahead to the goal ( G ), transforming complex forward searches into easy backward steps. In this work, we study how MTP facilitates reasoning in language models, with a focus on planning 1 1 1 Reasoning refers to the ability to perform logical or compositional inference, whereas planning is a specific form of reasoning that requires anticipating future steps and computing a global solution before generating outputs. , which requires considering future steps before generating the current output. Among MTP variants, we adopt the formulation of 12 , which predicts multiple future tokens using parallel heads. Note that our primary focus is on the reasoning performance rather than inference efficiency. Though the model is trained to predict multiple tokens, it generates only a single token at each step using the first head during inference. We begin with empirical observations on synthetic graph path-finding tasks, which are widely regarded as a standard abstraction for studying reasoning ( 20 ; 37 ) . Given the graph structure and start/end nodes, the model is required to generate the full path. We find that MTP consistently outperforms NTP on these tasks. We first study the star graph (see Figure 1(b) ), a directed graph in which multiple paths originate from the start node, with one leading to the target. In a star graph, only the first step requires a non-trivial decision. Thus, under NTP with teacher forcing, the model tends to follow edges based on previously revealed nodes in the prefix, a phenomenon known as the Clever Hans cheat ( 2 ) . To remove this shortcut, we introduce a more complex task, the binary tree , in which decisions must be made at every step. Notably, MTP continues to outperform NTP, indicating that its advantage cannot be explained solely by disabling the Clever Hans cheat. We evaluate on more realistic tasks, including Countdown ( 9 ) and boolean satisfiability problems, where MTP still succeeds. We next delve deeper into the underlying mechanisms behind the success of MTP on planning tasks, with a focus on the star graph. For simplicity, we study a two-layer disentangled Transformer ( 8 ) on the 2-path 3-node star graph. Our theory reveals a two-stage reverse reasoning process in models trained with MTP: the model first attends to the end node, then retrieves intermediate nodes by following edges that point to it. Intuitively, solving the star graph is easier when starting from the end node, which extends to the binary tree task. Importantly, MTP modifies only the training objective, while the inference process remains the same as NTP, indicating that reverse reasoning arises from optimization differences. Under NTP, learning signals from different layers are entangled, making it difficult for the model to discover the patterns needed for planning. In contrast, MTP provides an isolated training signal, allowing the first layer to attend to the end node and the second layer to recover the intermediate node through simple edge matching. Technical novelty. We are the first to formally analyze the convergence dynamics of MTP and establish its difference from NTP, extending beyond prior results ( 39 ) . The key is the gradient decoupling property of the MTP head, which provides an isolated training signal to the first layer. This property enables the emergence of reverse reasoning mechanism. Our technique might have independent interest to learning theory. 2 Related Work Understanding of MTP. Despite the empirical success of MTP, a rigorous theoretical understanding of its underlying mechanisms remains unclear. Recent work ( 39 ) studied how MTP enhances planning on synthetic tasks. However, its analysis is largely qualitative and does not provide a formal comparison with NTP. Moreover, its theoretical setup considers a simplified shared-head variant of MTP, rather than the standard independent-head architecture introduced by 12 . In contrast , our work provides a theoretical comparison between NTP and MTP under the more general independent-head architecture. We mathematically unpack how the gradient decoupling property of MTP naturally induces a two-stage reverse reasoning process on the star graph, providing a theoretical explanation for its empirical advantage over NTP. For an extended discussion of related work on MTP, we refer to Appendix A . Theories of Transformers. Extensive theoretical studies have been conducted