Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, Xun Zhou · 2025
Read Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling by Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, Xun Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16975.txt.
Introduction
Hongzhi Huang and co-authors argue that vocabulary size is an underpriced scaling lever for language models in a 2025 paper. Their Over-Tokenized Transformers split the input and output vocabularies, letting the input side grow into multi word chunk tokens while the output stays lean.
Across experiments, training loss fell in a clean log linear line as input vocabulary grew, whatever the model size. A fat input vocabulary let a model match rivals twice its size at no extra run cost. The authors read this as a gap in standard scaling laws, which mostly count parameters and data. Tokenizer design, they conclude, deserves first class status in planning the next model generation.
Catalog record
| CREATOR | Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, Xun Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, Xun Zhou Abstract Tokenization is a fundamental component of large language models (LLMs), yet its influence on model scaling and performance is not fully explored. In this paper, we introduce Over-Tokenized Transformers, a novel framework that decouples input and output vocabularies to improve language modeling performance. Specifically, our approach scales up input vocabularies to leverage multi-gram tokens. Through extensive experiments, we uncover a log-linear relationship between input vocabulary size and training loss, demonstrating that larger input vocabularies consistently enhance model performance, regardless of model size. Using a large input vocabulary, we achieve performance comparable to double-sized baselines with no additional cost. Our findings highlight the importance of tokenization in scaling laws and provide practical insight for tokenizer design, paving the way for more efficient and powerful LLMs. 1 Introduction The rapid advancements in large language models (LLMs) have been driven by innovations in model architectures ( Vaswani et al., 2017 ) and training paradigms ( Radford et al., 2019 ; Brown et al., 2020 ) . Moreover, with the guidance of scaling laws ( Kaplan et al., 2020 ) , models prove to become stronger with increasing number of parameters or training data. Tokenization, the process of converting raw text into discrete tokens as the input and output of the model, is recently found relevant to scaling laws, where larger models deserve larger vocabulary and achieve better performance under the same training cost ( Tao et al., 2024 ) . In fact, expanding the input vocabulary incurs almost no additional computational cost, whereas expanding the output vocabulary significantly increases the training overhead for smaller models. Thus, it is natural to consider decoupling the input and output vocabularies for separate investigations. Our research is fundamentally based on this idea. Figure 1 : Scaling trend for Over-Encoded models and baselines on OLMo2. We plot the loss with 400B tokens’ training. For over-encoding, input vocabulary size is extended from 0.1 to 1.2 and 12.8 million ( 12\times and 128\times larger than baseline), referred to as OE-1.2M and OE-12.8M. We observe OE-12.8M with 400M parameters matches the baseline with 1B parameters. Starting with synthetic experiments on context-free grammar modeling, we systematically analyze the effects of token granularity and vocabulary size on models of varying scales. Firstly, it is revealed that larger tokenizers improve the performance of larger models while introducing challenges for smaller ones, which is consistent with previous studies. Furthermore, once the input and output vocabulary is decoupled, we find that scaling up input vocabulary solely keeps improving model, while larger output vocabulary may be harmful to smaller models. This insight motivates the development of Over-Tokenized Transformers, which decouple the encoding and decoding vocabularies to achieve greater flexibility and performance gains. Specifically, we introduce Over-Encoding(OE), which utilizes large hierarchical n -gram input vocabulary. As shown in Figure 1 , our method improves the model scalability by a significant margin. Increasing the input vocabulary size by 128 \times , our 400M model matches the training loss of a 1B baseline with no additional training cost (see left panel). More interestingly, we observe a strong log-linear relationship between the input vocabulary size and model performance, i.e ., exponentially increasing the input vocabulary size consistently results in a linear decrease in loss (see right panel). These findings represent a new dimension in scaling laws and also indicate embedding parameters as a new scalable sparse dimension. Moreover, we propose the concept of Over-Decoding(OD), which leverages larger output vocabulary to provide more fine-grained supervision. We typically treat multi-token prediction methods ( Gloeckle et al., 2024 ; DeepSeek-AI et al., 2024 ) as approximations of OD. Combining OE and OD together, we build Over-Tokenized Transformer, which shows greater potential than apply either OE or OD solely. Although introducing a large amount of embedding parameters for OE, the training and inference cost barely increases, as embedding parameters are used in an extremely sparse manner. In practice, we propose efficient engineering solutions to mitigate the computational and memory challenges introduced by large vocabularies, resulting an additional training overhead less than 5%. Through this work, we aim to bridge the gap between tokenizer design and model scaling, positioning tokenization as a critical factor in the continued evolution of large language models. 2 Related Work 2.1 Tokenization Design Tokenization is a critical component in the development of large language models (LLMs). Established methods such as Byte-Pair Encoding (BPE) ( Sennrich, 2015 ) and Unigram Language Models ( Kudo, 2018 ) have been widely adopted to create subword vocabularies that balance sequence length and vocabulary size. Recently, alternative paradigms have been proposed to address the computational inefficiencies of byte-level models ( Xue et al., 2022 ; Clark et al., 2022 ; Tay et al., 2022 ; Yu et al., 2023 ) . For instance, MegaByte ( Yu et al., 2023 ) combines byte-level modeling with patching by first predicting patches and then predicting bytes within each patch, improving processing efficiency. However, its performance at scale remains inferior to that of tokenizer-based approaches. Building on this concept, In this work, our findings suggest that applying n -gram embeddings on top of BPE expands the vocabulary size, improving model capability, while BLT ( Pagnoni et al., 2024 ) also has the same finding on byte-level models. 2.2 Scaling Vocabulary Huang et al. (2021) firstly explores recurrent neural networks with scalable n -gram embedding table, where modular hashing is used to control vocabulary size. Roy et al. (2022) further augments the Transformer architecture with n -grams that are constructed from a discrete latent representation of the text sequence. Recent empirical studies have systematically investigated the relationship between vocabulary size and model performance. Tao et al. (2024) demonstrates that expanded vocabularies enhance both training efficiency and model performance, particularly in larger architectures. Building on these findings, we argue that vocabulary size research should separately consider embedding (input) and unembedding (output). While embedding incurs only lookup costs, unembedding introduces computational costs that scale with vocabulary size. More importantly, we find that input and output vocabularies exhibit distinct scaling behaviors, underscoring the need for independent scaling strategies to optimize model design. Moreover, Liu et al. (2024) revisits n -gram language modeling at massive scale, proposing an \infty -gram model trained on 5 trillion tokens, which achieves competitive performance and can effectively complement neural LMs to reduce perplexity. 2.3 Multi-Token Prediction Multi-Token Prediction (MTP) ( Stern et al., 2018 ; Qi et al., 2020 ; Gloeckle et al., 2024 ) has advanced the field of next-token prediction through the introduction of auxiliary objectives for simultaneous multi-token prediction. This methodology shares fundamental principles with our n -gram modeling framework, where multi-token prediction can be theoretically formulated as an approximation of n -gram unembedding utilization (over-decoded models). In our work, we further compare the effectiveness of MTP and over-encoded models, demonstrating that the performance gains from MTP and over-encoded models are complementary and can be combined to achieve even greater improvements. 3 Method 3.1 Insights from Synthetic Experiments Figure 2 : Performance comparison for models trained on CFG data. The left panel compares 1-gram and 3-gram tokenizers, showing that 3-gram improves larger (85M parameters)