Tokenization Matters: Improving Zero-Shot NER for Indic Languages
1 st Priyaranjan Pattnayak, 2 nd Hitesh Patel, 3 rd Amit Agarwal · 2025
Read Tokenization Matters: Improving Zero-Shot NER for Indic Languages by 1 st Priyaranjan Pattnayak, 2 nd Hitesh Patel, 3 rd Amit Agarwal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16977.txt.
Introduction
Priyaranjan Pattnayak, Hitesh Patel, and Amit Agarwal show that how a model chops words decides whether it can find names in languages it never trained on. Named entity recognition in low-resource Indic languages lives or dies by tokenization, the splitting of text into model-sized pieces. Rich word forms punish clumsy splitters.
Three strategies faced off on a regional language model: byte pair encoding, SentencePiece, and raw characters. Tests spanned seven languages, from Bengali down to the extremely low-resource Santali and Manipuri. SentencePiece won overall, holding entities together across languages in zero-shot transfer. The standard encoding was most compact but stumbled worst on unseen tongues, sometimes losing the entity label entirely.
Catalog record
| CREATOR | 1 st Priyaranjan Pattnayak, 2 nd Hitesh Patel, 3 rd Amit Agarwal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tokenization Matters: Improving Zero-Shot NER for Indic Languages 1 st Priyaranjan Pattnayak, 2 nd Hitesh Patel, 3 rd Amit Agarwal Abstract Tokenization is a critical component of Natural Language Processing (NLP), especially for low-resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte-Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low-resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character-Level tokenization strategies using IndicBERT for NER tasks in low-resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low-resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties—tokenization efficiency, out-of-vocabulary (OOV) rates, and morphological preservation—as well as extrinsic downstream performance, including fine-tuning and zero-shot cross-lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low-resource Indic Languages, particularly in zero-shot cross-lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low-resource and morpholically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low-resource Indic NLP applications. I Introduction Multilingual pre-trained models have transformed NLP tasks, but low-resource Indic languages are under-represented as they possess limited corpora, complex morphology, and diverse scripts. Indic languages have a large speaker base, including Assamese (25M), Bengali (270M+), Marathi (83M), and Odia (38M) [ 1 ] . However, extremely low-resource languages like Santali (7.6M), Manipuri (1.8M), and Sindhi (25M) face significant NLP challenges due to limited digital resources and linguistic complexities [ 2 ] . Named Entity Recognition (NER), being a very critical task for information extraction [ 3 ] , relies heavily on proper tokenization to segment entities appropriately [ 4 , 5 ] . In contrast to classification or translation, NER models are also marred with erroneous tokenization, splitting or merging entities, and thus hampering performance. While it is crucial, the impact of tokenization methods on NER in Indic languages remains unexplored. While Byte-Pair Encoding (BPE) [ 6 ] is common in multilingual models like mBERT [ 7 ] and IndicBERT [ 8 ] , how it can handle Indic morphology and zero-shot generalization is not yet studied. Earlier studies [ 9 , 7 , 10 ] examined tokenization for multilingual models [ 11 ] , but cross-lingual entity recognition remains a topic yet to be covered. Tokenization affects word representation in transformer models. Subword methods like BPE and WordPiece [ 7 ] use frequency-based merging but they don’t do well with low-resource languages due to out-of-vocabulary items and complex inflections [ 12 ] . SentencePiece [ 13 ] provides more flexible segmentation, holding linguistic structure more than frequency-based methods. Character-Level tokenization preserves morphological structure fully but increases sequence lengths [ 14 ] . This paper evaluates three tokenization strategies—BPE, SentencePiece, and Character-Level—using intrinsic and extrinsic analyses. Intrinsic evaluation covers tokenization efficiency, OOV rates, and morphological preservation across Indic languages, leading to the exclusion of Character-Level tokenization due to inefficiency and long sequences. Extrinsic evaluation fine-tunes IndicBERT [ 8 ] on Bengali and Hindi, with zero-shot transfer to Assamese, Oriya, Marathi, Manipuri, Sindhi, and Santali. Our results show that SentencePiece outperforms BPE in cross-lingual NER. Although BPE is able to get compact tokenization, it does not generalize as well and tends to use ”O” labels aggressively in zero-shot settings. SentencePiece retains more linguistic information, improving entity recognition for morphologically complex languages like Santali and Manipuri. It also handles script variation better, particularly in the case of Arabic-script Sindhi, where BPE fails. Intrinsic analysis shows that although BPE reduces OOV rates, it is poor in morphological preservation, while SentencePiece strikes a balance between flexibility and generalization. Key contributions: • Comparison of BPE, SentencePiece, and Character-Level tokenization for low-resource Indic languages. • Evaluation of intrinsic linguistic properties and extrinsic downstream performance. • Exclusion of Character-Level tokenization from extrinsic evaluation based on intrinsic findings. • Demonstration of SentencePiece’s superior zero-shot cross-lingual transferability. • Insights into tokenization’s impact on multilingual NLP, with broader implications for low-resource languages. Our study underscores the importance of tokenization in improving NLP for underrepresented languages. By demonstrating SentencePiece’s superiority in cross-lingual NER and Character-Level tokenization’s impracticality, we provide guidance for future multilingual NLP research and real-world applications [ 15 ] , contributing to more effective and inclusive language models. II Related Work II-A Tokenization Strategies for Low-Resource NLP Tokenization is at the heart of NLP, influencing vocabulary size, model generalization [ 16 ] , and cross-lingual transfer by specifying how words and subwords are represented in transformer-based models, particularly for low-resource languages [ 17 ] . Word-level tokenization fails with rare words, leading to unacceptably high out-of-vocabulary (OOV) rates in low-resource languages. Subword techniques like Byte-Pair Encoding (BPE) [ 6 ] and WordPiece [ 7 ] address this by splitting words into frequent subword units. BPE, which is widely used in multilingual models like mBERT and XLM-R [ 9 ] , combines character sequences frequency-wise, and WordPiece optimizes this further with probability-driven subword selection [ 18 ] . SentencePiece [ 13 ] removes the whitespace dependency so that character-aware segmentation is feasible. Character-Level Tokenization, although preserving morphology, is extremely long and computationally costly [ 19 , 20 , 14 ] . Despite advances, tokenization still remains optimized for high-resource languages. BPE struggles with rich morphology [ 21 ] and unseen entities, affecting Indic languages with inflectional variation, vocabulary fragmentation and diverse scripts [ 22 , 23 ] . This study systematically evaluates tokenization strategies for Indic NER in fine-tuning and zero-shot settings to address these challenges. II-B Tokenization and Multilingual Transfer Learning Multilingual models like XLM-R [ 9 ] and IndicBERT [ 24 ] rely on subword tokenization to balance vocabulary size and representation efficiency. While BPE supports compact vocabularies, it underperforms in morphologically rich low-resource languages [ 25 , 26 , 27 ] . Tokenization impacts cross-lingual performance [ 14 ] . BPE excels in similar languages but struggles in zero-shot settings, often fragmenting named entities into unrecognizable subwords. SentencePiece’s character-aware segmentation is a promising alternative, as prior studies suggest it preserves morphological structure better than frequency-based approaches [ 28 ] , though its efficacy in zero-shot NER remains underexplored. Our study addresses this gap by comparing BPE and SentencePiece in Indic zero-shot entity recognition. Wang et al. (2022) [ 29 ] further supports this by showing that tokenization-aware adaptation enhances cross-lingual transfer. II-C Named Entity Recognition in Indic Languages NER relies on accurate tokenization to preserve entity boundaries. Unlike classification, which uses broad text representations, NER models depend on precise segmentation—misalignment leads to misclassification. Prior Indic NER research has focused on dataset creation and model architectures [ 26 ] , often overlooking tokenization’s role. While transfer learning has been explored for low-resource NER [ 25