Scaling LLM Pre-training with Vocabulary Curriculum

Fangyuan Yu · 2025

Read Scaling LLM Pre-training with Vocabulary Curriculum by Fangyuan Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17910.txt.

Introduction

Fangyuan Yu proposes growing a language model's vocabulary during pretraining, the way children grow theirs, instead of fixing it on day one. Static vocabularies are the industry habit. Writing in twenty twenty-five, the method alternates two moves. Entropy-guided expansion adds tokens where the model needs finer grain. Optimization rounds then train with the enlarged vocabulary. Representations transfer across token sizes, so earlier learning is not wasted. A natural allocation emerges: long tokens absorb predictable stretches, while short tokens handle the hard, surprising contexts.

Experiments on small GPT-style models showed better scaling efficiency, with gains following a log-linear pattern in vocabulary size. Code is released for follow-up work. The author plans larger models and more domains. The paper reframes tokenization as a curriculum decision rather than a preprocessing afterthought.

Catalog record

CREATORFangyuan Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling LLM Pre-training with Vocabulary Curriculum Fangyuan Yu Abstract Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains. 1 Introduction Figure 1: Scaling better with vocabulary curriculum Modern language model pre-training relies on static vocabularies, fixed before training and detached from the model’s learning dynamics—unlike human language acquisition. This fixed approach limits models’ ability to adapt to different levels of linguistic granularity, potentially hindering efficiency and performance. While humans acquire language hierarchically, starting with basic units before building more complex representations, language models typically operate with predetermined tokenization schemes. Our approach dynamically merges predictable tokens, enabling the model to allocate computational resources more efficiently and shift focus toward harder-to-predict patterns. This results in an adaptive curriculum that evolves alongside the model’s capabilities. The vocabulary curriculum learning strategy begins with basic units (characters) and progressively expands to more complex representations, allocating more capacity to regions of high modeling entropy and refining the model’s understanding of difficult linguistic structures. A digram of our approach is provided in 2 Empirical results from pre-training GPT models [ 10 ] on the enwiki8 dataset [ 11 ] highlight two key advantages of our vocabulary curriculum learning approach: 1. It improves model performance across various vocabulary sizes, consistently achieving lower bits-per-character (BPC) compared to traditional fixed-vocabulary training. 2. It enhances scaling efficiency—models trained with a vocabulary curriculum exhibit a shallower slope (0.109 vs. 0.147) in log-scale vocabulary size vs. BPC plots, indicating more effective utilization of larger vocabularies. As shown in Figure 1 , models trained with incremental vocabulary curriculum learning (red) exhibit a steeper improvement curve compared to compute-matching baselines (blue). The log-scale vocabulary size vs. bits-per-character (BPC) plot reveals that vocabulary curriculum learning achieves a slope of -0.147, meaning it leverages larger vocabularies more effectively than compute-matching learning, which only reaches -0.109. Additionally, we observe that the curated vocabulary naturally forms a hierarchical structure, where longer tokens become increasingly predictable (lower BPC), while shorter tokens remain harder to predict (higher BPC). This structural organization emerges organically from our training process, reinforcing the effectiveness of our dynamic tokenization strategy. Our key contributions are: • A dynamic vocabulary creation system that adapts based on model entropy • A curriculum learning approach for tokenization that improves scaling efficiency • Evidence that hierarchical token organization emerges naturally from our approach While our focus is on language modeling, we believe this scaling effect can generalize to other modalities and domains, as byte sequences serve as the fundamental building blocks of digital data. Figure 2: Scaling better with vocabulary curriculum 2 Relevant Work 2.1 Tokenization Methods and Limitations Standard tokenization approaches like Byte Pair Encoding (BPE) [ 8 ] , [ 9 ] rely on static co-occurrence statistics detached from model learning. This creates representational limitations, particularly evident in early language models’ struggles with mathematical operations [ 10 ] . Naive BPE tokenization produces inconsistent representations of numbers—for example, "711" might be encoded as a single token while "703" requires multiple tokens. This inconsistency makes it harder for models to learn arithmetic operations compared to specialized approaches that assign unique tokens to all 1-3 digit integers [ 7 ] . Even with a fixed vocabulary, different encoding strategies can produce varying segmentations of the same text. BPE-dropout [ 2 ] leverages this property by introducing stochasticity during training, showing improvements in neural machine translation. More recent work exploits segmentation equivariance during inference to enhance reasoning through self-consistency [ 3 ] . Additionally, research [ 5 ] has established the existence of optimal vocabulary sizes for BPE-style tokenization, which correlate with model size, a log-linear relationship is observed between perplexity and vocabulary size. 2.2 Curriculum Learning Curriculum learning [ 13 ] progressively increases task difficulty during training to improve model performance. While successful in LLM post-training [ 15 , 21 ] , effective curriculum strategies for pre-training remain challenging [ 16 ] . Previous attempts at vocabulary-based curricula for decoder-only models found no improvements [ 19 ] , highlighting the difficulty of designing effective curricula for language model pre-training. Our work addresses these limitations with a novel adaptive approach to vocabulary curriculum. 2.3 Entropy Aware Tokenization Recent work has begun exploring entropy-aware tokenization. The Byte Latent Transformer [ 4 ] builds tokenization vocabularies using separately trained small language models. However, this approach creates vocabularies that are detached from the actual model’s entropy patterns and cannot be dynamically updated during training. Our work differs by integrating vocabulary building directly into the training process, allowing the tokenization scheme to evolve with the model’s developing understanding of the text. This creates a true curriculum that adapts to the specific learning trajectory of each model, rather than imposing a static or pre-computed vocabulary structure. 3 Approach Given a text corpus \mathcal{D} consisting of numerous character sequences, where each sequence x_{1:m}\in\mathcal{D} consists of characters x_{i} (or bytes). A vocabulary \mathcal{V} and an encoding function e(x_{1:m}|\mathcal{V}) together define a tokenization scheme that converts character sequences to token sequences s_{1:n} . Language modeling then focuses on predicting the next token: p(s_{n}|s_{1:n-1}) through minimizing entropy H(s_{t}|s_{1:t-1}) . We propose a dynamic tokenization framework that jointly learns the vocabulary and encoding strategy alongside the language model. Our approach consists of two key components: (1) entropy-guided vocabulary update and (2) vocabulary curriculum learning. 3.1 Entropy-Guided Vocabulary Update Given a trained language model f , we identify mergeable token sequences based on their predictability. For a sequence (s_{1},s_{2},\dots,s_{n}) , we compute the entropy H(s_{t}|s_{1:t-1}) for each token. A sequence is considered mergeable if all tokens after the first position exhibit monotonically decreasing entropy below threshold \epsilon : \text{mergeable}(s_{1:n})\iff\forall t>1:H(s_{t}|s_{1:t-1})<H(s_{t-1}|s_{1:t-2})\land H(s_{t}|s_{1:t-1})<\epsilon The vocabulary update process can either increase or decrease the vocabulary size: \mathcal{V}_{k+1}=\begin{cases}\text{add}(\mathcal{V}_{k},f,\mathcal{D})&\text{for vocabulary expansion}\\ \text{reduce}(\mathcal{V}_{k},n_{target})&\text{for vocabulary reduction}\end{cases} where n_{target} is the target vocabulary size. For each new token added to \mathcal{V}_{k+1} , we expand the model’s embedding layer W_{E}\in\mathbb{R}^{|\mathcal{V}|\times d} and language modeling head W_{L}\in\mathbb{R}^{|\mathcal{V}|\times d} : W_{E}[v_{new}]=h_{t}^{(L)},\quad W_{L}[v_{new}]=W_{L}[v_{t}] (1) where h_{t}^{(L)} represents the final hidden state for the merged sequence. Figure 3: Token grouping process based on entropy patterns from a trained character-level language model Unlike BPE which prohibits merges across space characters, our encoding function e(x_{1:m}|\mathcal{V}) allows unrestricted merging. The encoding process identifies longest valid token sequences using a sliding window approach, optimized through a trie structure for efficient prefix matching. To speed up encoding speed for long sequences, we employ batch encoding with additional tokenization at batch boundaries, considering connection sequences of length up to \max_{v\in\mathcal{V}}|v| . The vocabulary management preserves several invariants: (1) non-leaf tokens are preserved during removal to maintain

Read the complete text