IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
Souvik Rana , Arul Menezes , Ashish Kulkarni , Chandra Khatri , Shubham Agarwal [6pt] Krutrim AI, Bangalore, India, [12pt] · 2025
Read IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs by Souvik Rana , Arul Menezes , Ashish Kulkarni , Chandra Khatri , Shubham Agarwal [6pt] Krutrim AI, Bangalore, India, [12pt] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03237.txt.
Introduction
Souvik Rana and co-authors offer MUTANT, a recipe for building tokenizers that serve many languages at once. Tokenizers split text into pieces before a language model reads it, and poor splits raise cost and hurt quality. The task is hardest where scripts differ widely and words carry rich endings, as across India.
MUTANT pays close care to word lists, training mix, and language-aware pre-splitting. Its India-focused version keeps pieces that respect word structure and scores best among rivals tested. Across English, code, and twenty-two Indian languages, it cut average piece counts sharply versus strong baselines. Faster reading followed, with quality on tests held steady. Detailed side tests show which design choices carried the gains.
Catalog record
| CREATOR | Souvik Rana , Arul Menezes , Ashish Kulkarni , Chandra Khatri , Shubham Agarwal [6pt] Krutrim AI, Bangalore, India, [12pt] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-03237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs Souvik Rana , Arul Menezes , Ashish Kulkarni , Chandra Khatri , Shubham Agarwal [6pt] Krutrim AI, Bangalore, India, [12pt] Abstract Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods such as Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored. We present IndicSuperTokenizer, a tokenizer for Indic multilingual LLMs, that combines both subword and multi-word tokenization, along with language-specific pre-tokenization, leading to more linguistically aligned tokens and achieving a new state-of-the-art in fertility score. Evaluated across English, 22 22 Indian languages and code data, our tokenizer improves the average fertility score by 39.5 % 39.5\% over LLaMA4 and by 18 % 18\% over Sutra (the current best). This translates to 44 % 44\% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We also present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices. 1 Introduction Large Language Models (LLMs) ( 54 ; 18 ; 1 ; 20 ; 57 ; 51 ) rely on the crucial step of tokenization, the process of converting raw text into discrete units called tokens . A key metric for evaluating tokenizers is the “fertility score” (or token-to-word ratio) ( 7 ) where, a lower fertility score is desirable due to more efficient (and hence cheaper) LLM training and inference. Among the many proposed approaches, subword tokenization schemes such as BPE ( 46 ) , Unigram ( 28 ) , WordPiece ( 49 ) , and their byte-level extensions have become the de facto choice. However, tokenization remains an understudied topic within the LLM literature ( 11 ; 35 ) , especially in multilingual settings ( 38 ) , where, skewed fertility scores across languages, often lead to concerns around fairness, high inference latency, cost and context size. With 22 constitutionally recognized languages 1 1 1 https://en.wikipedia.org/wiki/Languages_with_official_recognition_in_India , these issues are especially pronounced for Indic languages comprising multiple scripts and a rich morphology. Our analysis suggests that tokenizers of popular multilingual tokenizers, largely designed for English, could produce fertility scores as high as 10.5 (LLaMA-4 tokenizer for Oriya; Table 22 ) for Indic languages, far worse than the near-ideal scores achieved for English. This leads to longer token sequences, higher compute overheads, and poor alignment with linguistic units like morphemes and compounds. Designing an efficient tokenizer involves making careful choices around the size of the vocabulary (of tokens), tokenizer training data, the tokenization approach, and, doing this across languages is nontrivial. In this work, we present IndicSuperTokenizer, an efficient tokenizer for Indic LLMs, that achieves state-of-the-art fertility scores across 22 Indic languages, English, and code. Our design choices are grounded in detailed ablations and our tokenizer combines linguistically grounded pre-tokenization with a two-stage subword–superword learning process (inspired from SuperBPE ( 33 ) ), yielding a more compact and semantically faithful vocabulary. Figure 1 illustrates some examples where our approach avoids fragmenting common words or idiomatic phrases into unnatural subunits across different languages. We make the following contributions: Figure 1: IndicSuperTokenizer (IST) captures superwords (e.g. “wake up”, “in the morning”) and avoids fragmenting Indic words (see for e.g. Bengali, Tamil). • We present IndicSuperTokenizer, a state-of-the-art tokenizer for Indic LLMs, systematically benchmarking it against popular multilingual baselines. • We study the impact of vocabulary size, training data, and language-specific pre-tokenization choices on fertility score, showing that careful pre-tokenization outweighs naive vocabulary scaling. • To the best of our knowledge, we are the first to carry out a comprehensive benchmarking of a tokenizer across multiple intrinsic quality measures, as well as to study its downstream impact on task performance and LLM inference efficiency in both pretraining from scratch as well as continual pretraining settings. 2 Related Work Tokenization Algorithms. Tokenization strategies differ in both theory and practice. While alternate sub-word tokenization algorithms have been explored in the past such as WordPiece ( 49 ) , Unigram LM ( 27 ) , Byte Pair Encoding (BPE) remains the most widely adopted. Originally developed for compression ( 15 ) and later adapted for neural MT ( 47 ) , BPE merges frequent character pairs to balance coverage with efficiency. Its variants aim to address inefficiencies: PickyBPE ( 10 ) discards uninformative merges to improve vocabulary utility, while Scaffold-BPE ( 31 ) iteratively prunes low-frequency scaffold tokens to reduce imbalance and enhance downstream performance. Recent extensions like SuperBPE ( 32 ) expand beyond word boundaries, jointly learning subwords and multi-word “superwords” yielding improved compression and inference efficiency in a 2-stage curriculum. BoundlessBPE ( 45 ) , another contemporary work, relaxes the Pre-tokenization word boundary constraint in an unified learning step. Our work compares these two recent approaches and builds upon SuperBPE that preserves subword coverage while capturing larger semantic units in morphologically rich Indian languages. Multilingual Tokenizers. Multilingual tokenization faces challenges from script diversity, morphology, and structural variation. Comparative studies show that vocabulary size and construction strategies strongly affect performance for morphologically rich languages ( 23 ) , while inefficiencies in underrepresented ones, such as Ukrainian, translate to higher fertility and computational costs ( 34 ) . Tokenization also influences how multilingual models encode morphology, as demonstrated in mT5 vs. ByT5 ( 12 ) . For Indic languages, tailored resources ( 22 ) and IndicBERT ( 6 ) highlight the value of domain-specific tokenization. Recent benchmarks further reveal economic implications, with BLOOM’s tokenizer achieving the best cost efficiency among popular multilingual LLMs ( 2 ) . Together, these studies show that current multilingual tokenizers fragment low-resource and morphologically rich languages, motivating approaches like ours that combine normalization, language-tailored pre-tokenization, and multi-word learning to achieve better efficiency and fairness in Indic languages. Tokenization for Indic languages presents unique challenges due to their linguistic diversity, rich morphology, and script multiplicity. Pre-tokenization. Pre-tokenization plays a pivotal role in shaping token boundaries, directly influencing both compression efficiency and reasoning performance ( 56 ) . SentencePiece ( 27 ) introduced a language-agnostic approach by treating input as raw streams, effective for languages without whitespace boundaries. More recent approaches like BoundlessBPE ( 45 ) relax pre-token constraints to improve frequency distributions, while regex-based designs continue to prove crucial for capturing script-specific structures. 3 IndicSuperTokenizer (IST) Language modeling involves estimating the probability distribution over text sequences, P(S) , where S may represent a sentence, paragraph, or document. To achieve this, the text is first converted into a sequence of discrete tokens through a tokenization function g(S)=X=(X_{1},X_{2},\ldots,X_{n})\in V^{n} , where V denotes the vocabulary and n the sequence length. Tokenizers can be open-vocabulary, ensuring any string can be represented (e.g., byte-level), or closed-vocabulary, where unseen text maps to an out-of-vocabulary symbol (e.g., word lists) ( 40 ) . In our work, we adopt an open-vocabulary approach that combines byte-pair encoding (BPE) with a UTF-8 byte fallback, following 39 . In this section, we describe our tokenizer training and evaluation approach. 3.1 Tokenizer Training With the aim of improving fertility in Indic languages and scripts,