How Much is Enough? The Diminishing Returns of Tokenization Training Data
Varshini Reddy, Craig W. Schmidt, Yuval Pinter, Chris Tanner · 2025
Read How Much is Enough? The Diminishing Returns of Tokenization Training Data by Varshini Reddy, Craig W. Schmidt, Yuval Pinter, Chris Tanner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20273.txt.
Introduction
Varshini Reddy and three co-authors ask how much training data a tokenizer needs in this two thousand twenty-five paper. Tokenizers split text into pieces before a language model reads it. Their quality depends on settings like vocabulary size and the algorithm used. One setting is rarely studied: how much text the tokenizer itself trains on. The team trained three kinds of tokenizers across many vocabulary sizes.
Their training corpora ranged from one gigabyte to nine hundred gigabytes of English text. Gains in tokenizer quality faded once data passed roughly one hundred fifty gigabytes. Past that point, more data bought little. The authors trace why the returns flatten. The finding gives model builders a sane budget for tokenizer training.
Catalog record
| CREATOR | Varshini Reddy, Craig W. Schmidt, Yuval Pinter, Chris Tanner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Much is Enough? The Diminishing Returns of Tokenization Training Data Varshini Reddy, Craig W. Schmidt, Yuval Pinter, Chris Tanner Abstract Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization strategy, inference strategy, and training data corpus. This paper investigates the impact of an often-overlooked hyperparameter, tokenizer training data size. We train BPE, UnigramLM, and WordPiece tokenizers across various vocabulary sizes using English training data ranging from 1GB to 900GB. Our findings reveal diminishing returns as training data size increases beyond roughly 150GB, suggesting a practical limit to the improvements in tokenization quality achievable through additional data. We analyze this phenomenon and attribute the saturation effect to constraints introduced by the pre-tokenization stage. We then demonstrate the extent to which these findings can generalize by experimenting on data in Russian, a language typologically distant from English. For Russian text, we observe diminishing returns after training a tokenizer from 200GB of data, which is approximately 33% more than when training on English. These results provide valuable insights for optimizing the tokenization process by reducing the compute required for training on large corpora and suggest promising directions for future research in tokenization algorithms. 1 Introduction Tokenizers are a foundational component of any NLP pipeline, as they are responsible for converting raw text into useful sequences of indexed tokens. Practitioners often default to standard tokenization algorithms such as Byte-Pair Encoding ( Sennrich et al., 2016 , BPE;) , UnigramLM ( Kudo, 2018 ) or WordPiece ( Schuster & Nakajima, 2012 ; Devlin et al., 2019 ) , sourced directly from libraries such as Hugging Face. 1 1 1 https://github.com/huggingface/tokenizers The training process of a tokenizer involves using a corpus of training data and a specific tokenization algorithm to generate a fixed-size vocabulary, usually containing between 32,000 and 128,000 tokens in the monolingual case. While extensive research explores the influence of a model’s training data on LLM performance ( Pearce et al., 2024 ; Zhang et al., 2024 ; Hoffmann et al., 2022 ; Kaplan et al., 2020 ) , the impact of a tokenizer’s training data remains relatively unexplored. Recent work has begun to address the importance of tokenizers’ vocabulary sizes and the training data domains ( Dagan et al., 2024 ) , tying into existing exploration of other aspects of tokenization, including the influence of different tokenization algorithms ( Schmidt et al., 2024 ; Ali et al., 2024 ; Mielke et al., 2021 ; Rai & Borah, 2021 ) , vocabulary size optimization ( Gowda & May, 2020 ) , and the interplay between data type and tokenization strategy, especially in multi-lingual applications ( Limisiewicz et al., 2023 ; Rust et al., 2021 ) . Yet, to the best of our knowledge, we are the first to investigate how much training data is needed for a tokenizer, and how this affects performance. We address this by examining the impact of scaling tokenizer training data with sizes ranging from 1GB to 900GB. We train English BPE, UnigramLM, and WordPiece tokenizers with vocabulary sizes of 40,960, 64,000, 128,000, and 256,000. For each variant, we examine the proportion of vocabulary tokens that are shared with the 900GB reference case, and use intrinsic metrics to measure the quality of tokenization on a held-out evaluation corpus of 150GB. Our findings indicate that increasing the amount of tokenizer training data leads to diminishing returns in the compute-versus-gain tradeoff, suggesting a saturation point at roughly 150GB of training data beyond which further data provides minimal to no improvements in tokenization quality. We then examine the proportion of pre-tokenization chunks that exactly match a single token in the tokenizer vocabulary, and suggest that this very high proportion is a possible explanation for these diminishing returns. Following up on the English experiments, we turn to Russian as a second test case to see whether the main findings hold up on a language that is distinct from English both by using a different alphabet and by various typological properties such as freedom in word-order and morphological richness. We focus on BPE vocabularies of size 40,960 ranging from 30GB to 600GB, identifying a later saturation point than in English at around 200GB, suggesting that complexity in word formation may impact the data hunger of tokenizers, requiring them to take up more samples from a larger and more idiosyncratic word-level vocabulary. 2 Effect of Training Corpus Size on Tokenization Our training corpus for the main experiments in English combines the de-duplicated PILE ( Gao et al., 2020 ) and RedPajama ( Weber et al., 2024 ) datasets, totaling 900GB of text. 2 2 2 As our work is the first of its kind in the tokenization space, there was no predefined baseline for corpus size to reference. However, a recent study ( Schmidt et al., 2024 ) used 200 billion tokens (approximately 850GB of the PILE dataset) for tokenizer comparisons, which we adopted as our upper limit. This choice serves as a practical reference point rather than an assumed optimal baseline. In addition, we held out a de-duplicated set of size 150GB from the PILE and RedPajama corpus for evaluation. We build token vocabularies using three algorithms: BPE, 3 3 3 We use a custom BPE tokenizer based on https://github.com/karpathy/minbpe as a starting point. UnigramLM, 4 4 4 https://github.com/huggingface/tokenizers/blob/main/bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py and WordPiece. 5 5 5 https://github.com/huggingface/tokenizers/blob/main/bindings/python/py_src/tokenizers/implementations/bert_wordpiece.py For each algorithm, we train vocabularies of sizes 40K, 64K, 128K, and 256K, on progressively larger subsets of the randomly shuffled 900GB corpus. In order to evaluate the effects of training data size on both smaller and larger scales, we start with finer increments of 1GB, 5GB, and 15GB subsets, followed by all 30GB increments between 30GB and the full 900GB corpus. The size increase is augmentative, meaning that each corpus subset properly includes all those with smaller size. Overall, we trained 33 distinct vocabularies for each of the four vocabulary sizes for each of the three tokenizer algorithms, leading to a total of 396 trained vocabularies for English. Pre-processing To facilitate training token vocabularies on datasets ranging to hundreds of gigabytes, we implemented a parallel pre-tokenization step, which used a regular expression to break the documents into chunks. We then aggregated the counts of each pre-tokenized chunk over all documents. This pre-tokenized corpus of chunks and counts served as the input for all subsequent vocabulary building, significantly reducing its computational overhead by avoiding tokenization of common chunks more than once. All tokenizer code was modified to support this method of aggregate chunking and incremental counting. shows the regular expression used by GPT-4 for pre-tokenization, which we also used for our parallel pre-tokenizer. A breakdown of this regex, with an explanation of each branch, is presented in Appendix A . ⬇ r "(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ␣ ?[^\s\p{L}\p{N}]++[\r\n]|\s[\r\n]|\s+(?!\S)|\s+" Listing 1: GPT-4 pre-tokenizer regular expression. 2.1 Vocabulary analysis (a) Vocabulary size 40,960 (b) Vocabulary size 64,000 (c) Vocabulary size 128,000 (d) Vocabulary size 256,000 Figure 1 : Proportion of common vocabulary for BPE, Unigram, and WordPiece tokenizers, trained with cumulatively increasing data, relative to the vocabulary of the corresponding tokenizer trained with 900GB of data. Figure 1 illustrates the fraction of vocabulary shared between tokenizers trained on varying amounts