Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer · 2026
Read Compute Optimal Tokenization by Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01188.txt.
Introduction
Tomasz Limisiewicz and eight co-authors study compute-optimal tokenization, asking how the size of the data unit, the token, affects language model scaling laws. Scaling laws guide choices of data amount and model size, but the token's information granularity, controlled by compression rate in bytes per token, has been underexplored. The team trains nearly a thousand latent tokenized models across a wide size range, with flexible compression control.
In compute-optimal configurations, model size scales proportionally with data measured in bytes, not tokens as commonly assumed. The optimal compression rate differs from that of a popular byte-pair tokenizer and decreases with compute. Findings generalize across latent and subword tokenization and to languages beyond English, guiding developers toward tokenization choices with maximal compute efficiency.
Catalog record
| CREATOR | Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01188
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Compute Optimal Tokenization Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer, Mike Lewis, Sachin Mehta, Alisa Liu, Margaret Li, Gargi Ghosh, Luke Zettlemoyer Abstract Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token , on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained with a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes , not in tokens as commonly perceived ( 10 ; 8 ) . Furthermore, we discover that the optimal compression rate differs from the one obtained with BPE and decreases with compute. These findings generalize to both latent and subword tokenization, as well as to languages other than English, guiding language model developers on tokenization scheme selection for maximal compute efficiency. † † affiliation: FAIR at Meta † † affiliation: University of Washington † † date: May 4, 2026 † † correspondence: [email protected] † † Code: https://github.com/facebookresearch/compute-optimal-tokenization † † Blogpost: https://co-tok.github.io (a) Optimal bytes per parameter ratio across compression rates. Fixed training budget 10^{20} FLOPs. (b) Optimal compression rate based on a scaling law fit. Training budget marked by color. (c) Optimal compression rate differs from compression of subword tokenizers Figure 1 : Key findings of this work: 1(a) in compute optimal scaling: bytes (not tokens) of data increase proportionally to parameter count; 1(b) for each training budget, we find optimal compression rate, its value decreases with scale; 1(c) the optimal compression rate varies across languages and differs from compression of popular BPE tokenizers. 1 Introduction Scaling laws have informed the efficient design of language models, prescribing the optimal balance between model size and training data ( 10 ; 8 ) . Standard approaches estimate the optimal amount of data in tokens for a given compute budget (and model size). However, expressing data volume in tokens overlooks a critical aspect: the information density that each token represents. Consequently, scaling findings inherently depend on specific tokenizers and their key property: the compression rate. To fill the research gap, we introduce laws that are aware of the compression rate T , defined as the average number of bytes per token in a given dataset. For that purpose, we need to vary the compression rate without changing the vocabulary size (and thus the number of parameters). Therefore, in our experiments we rely on Byte Latent Transformer (BLT, 26 ), a recent architecture that segments byte-level input in a latent space. BLT’s latent tokenization is a robust tool for this purpose, as it allows us to precisely adjust the compression rate by setting an average segment size. 1 1 1 Recent latent tokenized models allow achieving a wide range of compression rate, similarly to BLT. However, in other approaches compression rate cannot be precisely controlled due to reliance on a segment boundary predictor ( 9 ; 23 ) or whitespace supervision ( 24 ; 36 ; 42 ) . Additionally, compression plays a significant role in subword tokenization. We can order popular subword methods by their compression rate: from pure byte or character-level segmentation ( T\approx 1 ) ( 44 ; 43 ) , through widely used BPE ( T\approx 4.57 ) ( 35 ) , to SuperBPE ( T\approx 6.16 ) ( 16 ) , which achieves high compression by allowing multi-word tokens. 2 2 2 Estimates of compression rate are computed for DCLM corpus ( 12 ) consisting of “plain English” texts. In the context of scaling, compression rate impacts model efficiency in both training and inference. Increasing compression allows the same data to be represented with fewer tokens, directly reducing the computational cost of processing. The unlocked savings in FLOPs (unit of computation) can be used to increase training data, model size, or both, without increasing the total computation budget. To the best of our knowledge, this is the first thorough study of the effect of compression rate on the compute efficiency of language models. We pose the following research questions: [R1] : How does compression rate impact the compute-optimal ratio between parameters and data? This question concerns the unit of data we should use in model scaling. We investigate whether the compute-optimal ratio is best expressed in tokens or bytes (which are the underlying unit of text encoding). For example, given the Chinchilla rule of thumb of training on \approx 20 tokens per parameter ( 8 ) , does this ratio hold as we increase compression, or should the ratio of bytes to parameters remain constant given a dataset of English texts? [R2] : Is there an optimal compression rate for specific datasets? We investigate whether there exists a compression rate that yields the lowest loss for a fixed compute budget, assuming the optimal data to parameter ratio. Furthermore, we examine whether this optimal compression rate shifts with the compute budget or dataset domain. [R3] : Is the impact of compression rate on scaling trends similar for latent and subword tokenized models? Does the answer to the previous questions depend on the tokenization method? We conduct experiments on subword-tokenized models to validate if the scaling trends match those observed for BLT. [R4] : Is optimal compression rate language specific? We extend our experiments to languages other than English to test whether optimal data to parameter ratio and compression rate change depending on language. We hypothesize that both will grow proportionally to parity , defined as the ratio of byte length of parallel sentences expressed in two languages ( 29 ; 2 ) . The structure of the paper is as follows. In Section 2 , we describe our experimental setting, including details of the datasets, models, and methods for deriving power laws. In Section 3 , we present experiments scaling BLT across a wide range of compression rates to answer [R1] and [R2] . In Section 4 , we examine subword-tokenized models to compare with the findings from the previous section and address [R3] . Finally, in Section 5 , we extend our scaling experiments to languages other than English to answer [R4] . 2 Methodology In this section, we provide details on the language models used and experimental setup. We also describe the evaluation and the procedure for fitting power laws to estimate the optimal data-to-parameter ratio and loss. 2.1 Model Architectures Figure 2 : The grid of experiments for the budget of C=10^{20} FLOPs. For each compression rate T (x-axis) and model size N (y-axis) we can read amount of training data B (color) and the corresponding bytes per parameter ratio \rho (values in squares). For all experiments, we train Transformer models ( 41 ) of varying parameter sizes, adhering to Llama 3 architectural choices ( 18 ) . We follow a standard scaling recipe: increasing models’ width and depth in a 1:1 ratio, meaning the number of heads equals