Reducing Tokenization Premiums for Low-Resource Languages

Geoffrey Churchill, Steven Skiena · 2026

Read Reducing Tokenization Premiums for Low-Resource Languages by Geoffrey Churchill, Steven Skiena (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13328.txt.

Introduction

Geoffrey Churchill and Steven Skiena cut the extra cost that low resource tongues pay in AI tools. Next to English, such tongues need far more tokens to state the same thought. A token is the chunk a model reads at a step. More tokens mean more fee, more power, and less room left for the task.

That gap is called a tokenization premium. It stems from how token cuts were fit to rich tongues first. The team gives ways to shrink the premium for thin tongues. For users of such tongues, cuts here mean AI that costs less and can hold more of a chat at once.

Catalog record

CREATORGeoffrey Churchill, Steven Skiena
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reducing Tokenization Premiums for Low-Resource Languages Geoffrey Churchill, Steven Skiena Abstract Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model. 1 Introduction The very first step in the creation of any language model is the definition of its vocabulary, the set of atomic text blocks (tokens) into which input strings are segmented. Most modern LMs have vocabularies of hundreds of thousands of tokens, typically including every distinct base character or byte pattern to ensure that every string can be encoded. They also include longer strings representing common words and subwords, to compactly represent typical input strings. Early tokenization strategies included breaking text at whitespace and punctuation, and more sophisticated handcrafted rules for splitting words into subwords. Today, popular tokenizers derive their token vocabularies through data-driven approaches, typically byte-pair encoding (BPE) or a variant thereof. BPE learns a tokenizer from a training corpus by starting with an initial set of possible characters, and then repeatedly merging the most common pair of tokens into a new token that is added to its growing vocabulary ( 6 ) . This procedure continues for a given number of rounds until a desired vocabulary size is reached or, for smaller corpora, when the most common pair falls below some frequency threshold. As a data-driven procedure, BPE learns a vocabulary that reflects the training corpus. High-resource languages, especially English, typically dominate available training data, so the vocabularies of BPE-trained tokenizers largely comprise words and subwords from these languages. This has several undesirable consequences, as the tokenization premium (penalty) for important but lower-resource languages such as Bangla, Hindi, and Urdu can easily be three to five times that of English. This penalty manifests itself in several hard costs and limitations: content in these languages will be processed slower, cost three to five times more in API calls/energy usage, and be limited to context windows of only a fraction of that available to English ( 15 ) . In this paper, we study the landscape of current tokenizers to better understand their similarities and differences, and the tokenization premiums they impose on various languages. We additionally explore an approach to mitigating tokenization premiums, by retrofitting a frozen model with additional tokens and derived (as opposed to trained) embeddings for these new tokens. Our approach presupposes that the model’s training corpus included content in the target language, so that our challenge is to identify a single embedding for a common substring in the target language that the frozen model treats similarly to the substring’s original multi-embedding encoding. This could yield more efficient tokenizations with hopefully only minor loss in quality. Our main contributions in this paper are: • Comparison of Vocabularies Across Models – Our analysis of the vocabularies of ten prominent LMs demonstrates that they cluster into several distinct groups according to Jaccard similarity. • Language-Specific Tokenization Penalties by Model – We show that models differ substantially in how efficiently they tokenize text between Latin and non-Latin low-resource languages. Important languages in non-Latin scripts (e.g. Bangla, Hindi, and Urdu) have tokenization penalties between three and four times that of English, while much smaller languages in Latin scripts (e.g. Danish, Swedish, and Norwegian) are efficiently encoded. • Mitigation Strategies for Language-Specific Tokenization Penalties – We propose and evaluate different approaches to retrofitting a frozen model with new tokens to reduce tokenization premiums while minimizing the reduction in inference quality of the generated sequences. We focus our efforts on adding tokens to represent individual missing Unicode characters from non-Latin scripts. Our paper is organized as follows. We review previous work on tokenization strategies in Section 2 . We perform a comparative vocabulary analysis of ten popular tokenizers in Section 3 , and report on language-specific tokenization premiums. We propose and evaluate mitigation strategies for these languages in Section 4 , before concluding with possible future research directions in Section 5 . 2 Related Work We consider related work in (i) tokenization strategies and (ii) post-training adaptations of frozen models. Tokenization: The most popular tokenization strategies used by LMs are based on byte-pair encoding (BPE). This strategy was first proposed by 6 in the context of dictionary-based compression. The idea is simple: initialize the vocabulary of tokens to the set of all characters, and in each iteration merge a most frequent adjacent pair of tokens in the corpus into a single new token which is added to the growing vocabulary. Running BPE on "she_shakes_shoes" until all pairs are singletons would (up to tie-breaking) look like: Tokenized string Vocabulary Merge s h e _ s h a k e s _ s h o e s _ a e h k o s s h sh e _ sh a k e s _ sh o e s _ a e h k o s sh _ sh sh e _sh a k e s _sh o e s _ a e h k o s sh _sh e s sh e _sh a k es _sh o es _ a e h k o s sh _sh es 20 applied BPE to improving the machine translation of rare words. BPE improves upon naive methods like whitespace splitting (which yields a huge set of opaque tokens), or handcrafting a more morpheme-aware tokenizer (which is brittle and does not scale). Whitespace splitting would yield unrelated tokens for "unrelated" and "relatable" whereas BPE could tokenize them as "un|relat|ed" and "relat|able" , which share the "relat" subword and contain the informative morphemes "un" , "ed" , and "able" . 15 and 1 explore and quantify some of the downstream impacts of imbalanced BPE training corpora, showing that applying LMs to content in low-resource languages is often much slower and many times more costly than the analogous English content. WordPiece ( 19 ) is a related strategy that, like BPE, iteratively merges pairs of tokens. WordPiece, however, is explicitly geared towards language modeling by merging the pair that would most increase the likelihood of the training corpus, with respect to some language model. For example, according to a unigram model, the log likelihood of the training corpus C is \sum_{x\in C}\log p(x)=\sum_{x\in C}\log\text{\#}x-|C|\log|C| where x\in C ranges over every token in C , and \text{\#}x denotes the frequency of x in C . Merging every a b pair into an ab gives a net reduction of # ab tokens, where # ab is computed “sequentially”, i.e. without overlaps, since otherwise if a and b were the same token the bigram frequency would be

Read the complete text