Pre-Training Curriculum for Multi-Token Prediction in Language Models

Ansar Aynetdinov, Alan Akbik · 2025

Read Pre-Training Curriculum for Multi-Token Prediction in Language Models by Ansar Aynetdinov, Alan Akbik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22757.txt.

Introduction

Ansar Aynetdinov and Alan Akbik propose a training order for language models that learn to predict many words ahead, in a 2025 paper. Most models predict just the next word. Predicting several words at once can speed output and lift quality, above all in large models. Small models, though, struggle with that harder goal from the start.

A stepwise plan eases small models in. One path ramps up from next-word to many-word goals. The other path runs in reverse. The ramp-up path helped small models use the hard goal, lifting later scores and output quality while keeping a fast self-draft trick. The reverse path scored higher still but lost that speed trick.

Catalog record

CREATORAnsar Aynetdinov, Alan Akbik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22757

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pre-Training Curriculum for Multi-Token Prediction in Language Models Ansar Aynetdinov, Alan Akbik Abstract Multi-token prediction (MTP) is a recently proposed pre-training objective for language models. Rather than predicting only the next token (NTP), MTP predicts the next k tokens at each prediction step, using multiple prediction heads. MTP has shown promise in improving downstream performance, inference speed, and training efficiency, particularly for large models. However, prior work has shown that smaller language models (SLMs) struggle with the MTP objective. To address this, we propose a curriculum learning strategy for MTP training, exploring two variants: a forward curriculum, which gradually increases the complexity of the pre-training objective from NTP to MTP, and a reverse curriculum, which does the opposite. Our experiments show that the forward curriculum enables SLMs to better leverage the MTP objective during pre-training, improving downstream NTP performance and generative output quality, while retaining the benefits of self-speculative decoding. The reverse curriculum achieves stronger NTP performance and output quality, but fails to provide any self-speculative decoding benefits. 1 Introduction In recent years, large language models (LLMs) have demonstrated remarkable capabilities in understanding and generating complex text, code, and other modalities. These advances have been driven primarily by improvements in model architectures, training data quality and scale, and optimization strategies, and yet the underlying objective for most widely-adopted LLMs remains the next-token prediction loss. Models such as GPT Brown et al. 2020 ; OpenAI et al. 2024 , LLaMA Touvron et al. 2023a ; Touvron et al. 2023b ; Grattafiori et al. 2024 , and Phi Li et al. 2023 ; Abdin et al. 2024 are conventionally trained to predict a single token at each generation step, implicitly treating language modeling as a sequence of one-step-ahead predictions. While this formulation is simple and effective, it may not fully leverage the predictive capabilities of LLMs or reflect the underlying structure of natural language. Gloeckle et al. 2024 investigated the potential of a multi-token prediction (MTP) objective for LLMs. Instead of predicting just the next token (NTP), their approach aims to produce multiple subsequent tokens at each prediction step, with multiple output heads that are independent from each other but share a common model backbone. They found this approach to improve model’s downstream performance, inference speed, and training sample efficiency without significantly increasing training time. Recently, DeepSeek-AI adopted the MTP training objective for their V3 model Liu et al. 2024a that serves as a base model for the reasoning R1 model Guo et al. 2025 . So far, the previous work prioritized mid- and large-sized models with at least 7B parameters, since Gloeckle et al. 2024 observed that MTP leads to more performance gains as the model size increases. We hypothesize that smaller language models (SLMs) enjoy less benefits from the MTP objective during pre-training due to the fact that they struggle to deal with morphological and semantic dependencies between multiple tokens at once from the get-go. A curriculum-based approach to MTP. Inspired by the work of Bengio et al. 2009 , who were the first to propose curriculum learning strategies in the context of machine learning, we explore using a pre-training curriculum to enable SLMs to better leverage the benefits of the MTP objective. Since the complexity of the MTP objective increases with the amount of tokens considered, it is easy to construct a curriculum by gradually changing the number of tokens considered in the objective. Figure 1: Visualization of the forward and reverse MTP curricula. When training a language model on a 3-token-prediction task for n steps, the forward curriculum starts with a vanilla NTP task, adding an additional token to the task every \frac{n}{3} steps. The reverse curriculum does the opposite, starting with a full 3-token-prediction task, and dropping a token from the task every \frac{n}{3} steps. Figure 1 illustrates our proposed curriculum variants. Under a forward curriculum, models are given a simpler NTP task in the beginning of pre-training, and as the training progresses, they are guided towards a more complex MTP task. The task is expanded by incrementally including one additional token at uniform intervals throughout pre-training. The reverse curriculum, on the other hand, starts with a full MTP task, and incrementally removes one token from the task also at uniform intervals. The two opposing curricula are designed to clearly determine whether MTP SLMs perform better with an easy-to-hard or a hard-to-easy learning strategy. Contributions. This work makes the following contributions: • We provide a comprehensive exploration of the MTP objective’s potential when used to train SLMs predominantly on natural language. • We propose and evaluate the validity of novel curriculum-based MTP training strategies. • We showcase that curriculum-based (dynamic) MTP pre-training strategies allow SLMs to learn the MTP task better than static MTP baselines. 2 Preliminaries In this section, we formalize the multi-token prediction objective and describe the design of the proposed curricula. 2.1 Multi-Token Prediction Objective Traditional large language models are trained using the next-token prediction (NTP) objective, where the model, given a context sequence \mathbf{x}=(x_{1},x_{2},\dots,x_{t}) , is tasked with predicting the next token x_{t+1} . This is accomplished by maximizing the log-likelihood of the target token: \mathcal{L}_{\text{NTP}}=-\sum_{t=1}^{T}\log P(x_{t+1}|x_{1},\dots,x_{t};\theta), where \theta represents the model parameters. In contrast, the multi-token prediction (MTP) objective extends this task by requiring the model to predict a sequence of k tokens \mathbf{y}=(x_{t+1},x_{t+2},\dots,x_{t+k}) simultaneously. The MTP loss is defined as: \mathcal{L}_{\text{MTP}}=-\sum_{t=1}^{T}\sum_{i=1}^{k}\log P(x_{t+i}|x_{1},\dots,x_{t};\theta), where P(x_{t+i}|x_{1},\dots,x_{t};\theta) is computed using multiple output, or language modeling, heads conditioned on the shared model backbone. Predicting multiple tokens at each prediction step allows for self-speculative decoding Stern et al. 2018 during inference without the need for an auxiliary model. This can speed up the inference by up to k times by reducing the number of forward passes needed to generate a sequence. In addition, the MTP objective promotes the learning of richer contextual dependencies, as the model must optimize for multiple interrelated predictions at each step. 2.2 Forward and Reverse Multi-Token Curricula The design of the curricula dynamically adjusts the number of output, or language modeling, heads k\in\{1,2,\dots,k_{\max}\},\quad k\in\mathbb{Z} , which corresponds to the number of tokens predicted by the model at each prediction step. These adjustments are based on the training epoch e , ensuring systematic changes every E/k_{\max} epochs, where E is the total number of training epochs. While it is certainly possible to come up with a more adaptive curriculum that relies on the training dynamics, we aim to isolate and demonstrate the core benefit of using a curriculum-based MTP objective over a static one. Therefore in this paper we decide to focus on a simple pre-defined curriculum design. 2.2.1 Forward Multi-Token Curriculum The forward multi-token curriculum starts with the simplest prediction task ( k=1 ) and progressively increases the number of prediction heads. Formally, the number of active output heads at training epoch e can be defined as: k_{\text{current}}(e)=\min\left(k_{\max},\left\lfloor\frac{e}{E/k_{\max}}\right\rfloor+1\right). This gradual increase in complexity ensures that the model first learns fundamental token-by-token predictions before transitioning to more complex multi-token prediction tasks. 2.2.2 Reverse Multi-Token Curriculum In contrast, the reverse multi-token curriculum starts with the maximum number of prediction heads (k=k_{\max}) and gradually reduces the number of active heads over time. The number of active prediction heads at training epoch e is defined as:

Read the complete text