Can maiBERT Speak for Maithili?

Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Md Azizul Hoque, Ganesh Gautam · 2025

Read Can maiBERT Speak for Maithili? by Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Md Azizul Hoque, Ganesh Gautam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15048.txt.

Introduction

Sumit Yadav and five co-authors build maiBERT, a language model for Maithili, in a 2025 natural language processing paper. Maithili is spoken by millions yet lacks the data and models that digital tools require. Low-resource languages like it are routinely left out of AI-driven services.

The team assembled a new Maithili text corpus and pre-trained a BERT-style model on it with masked language modeling, where the model learns by filling in hidden words. Evaluation used a news classification task. The model reached accuracy above that of existing regional models for Nepali and Hindi, with the clearest gains on several individual news classes. Releasing maiBERT openly on Hugging Face lets others fine-tune it for sentiment analysis, named entity recognition, and further Maithili applications.

Catalog record

CREATORSumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Md Azizul Hoque, Ganesh Gautam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can maiBERT Speak for Maithili? Sumit Yadav, Raju Kumar Yadav, Utsav Maskey, Gautam Siddharth Kashyap, Md Azizul Hoque, Ganesh Gautam Abstract Natural Language Understanding (NLU) for low-resource languages remains a major challenge in NLP due to the scarcity of high-quality data and language-specific models. Maithili, despite being spoken by millions, lacks adequate computational resources, limiting its inclusion in digital and AI-driven applications. To address this gap, we introduce maiBERT , a BERT-based language model pre-trained specifically for Maithili using the Masked Language Modeling (MLM) technique. Our model is trained on a newly constructed Maithili corpus and evaluated through a news classification task. In our experiments, maiBERT achieved an accuracy of 87.02%, outperforming existing regional models like NepBERTa and HindiBERT, with a 0.13% overall accuracy gain and 5–7% improvement across various classes. We have open-sourced maiBERT on Hugging Face 1 1 1 https://huggingface.co/rockerritesh/maiBERT˙TF , enabling further fine-tuning for downstream tasks such as sentiment analysis and Named Entity Recognition (NER). 1 Introduction Natural Language Processing (NLP) has seen rapid advancements in recent years, with the development of sophisticated models like BERT ( Devlin et al. 2019 ) , GPT ( Radford et al. 2018 ) , and T5 ( Raffel et al. 2020 ) that enable machines to understand, interpret, and generate human language. However, most of this progress has primarily focused on high-resource languages such as English, Mandarin, and Hindi. Low-resource languages—languages that lack sufficient digital content, annotated corpora, and computational resources—have remained largely underserved. One such language is Maithili 2 2 2 In terms of linguistic structure, Maithili follows a syntax and morphology that are comparable to other Devanagari-script languages such as Hindi and Nepali. It comprises 47 segmental phonemes: 8 vowel phonemes characterized by duration and quality, 39 consonant phonemes, and 10 numerals represented as letters. as shown in Fig. 1 , an Indo-Aryan language spoken by millions of people in Nepal and India, particularly in the Terai region of Nepal (mainly Madhesh Province and parts of Koshi Province) and neighboring districts in the Indian state of Bihar. Figure 1: Density-Based Map of Maithili Language Maithili is constitutionally recognized in Nepal and India and is widely spoken in both formal and informal contexts. Despite its rich linguistic and cultural heritage, Maithili has received minimal attention in the computational linguistics community ( Mundotiya et al. 2021 ) . Unlike related languages such as Hindi and Nepali, which have been the subject of substantial research ( Joshi 2022 ; Pudasaini et al. 2023 ; Maskey et al. 2022 ) and have several publicly available resources ( Bal et al. 2024 ; Bafna and Žabokrtskỳ 2022 ) , Maithili lacks a comprehensive digital footprint. There are no large-scale pre-trained language models, standardized datasets, or open-source tools dedicated to this language, which significantly impedes the development of NLP applications for Maithili speakers. While existing models developed for related languages such as ( Hartmann et al. 2017 ; Alian and Awajan 2020 ; Pandit et al. 2019 ; Patil and Kolhe 2022 ; Nie et al. 2022 ; Ramchandra et al. 2019 ; Lalrempuii et al. 2021 ; Priyadarshi and Saha 2020 ; Mundotiya et al. 2022 ) have shown some promise, they fall short when applied directly to Maithili due to dialectal variations, unique lexical items, and morphological structures that distinguish it from other Indo-Aryan languages. To bridge this critical gap, we propose a novel approach that focuses on the development of a Maithili-specific language model using the BERT architecture. Our work begins with the construction of a high-quality Maithili corpus collected from diverse sources, including online articles, books, and conversational data. This raw corpus forms the basis for pre-training a Masked Language Model (MLM) tailored specifically for Maithili. Following pre-training, we fine-tune the model for a downstream NLP task—text classification. Our model, referred to as maiBERT , is then benchmarked against State-of-the-Art (SOTA) to evaluate performance. Moreover, to support the wider research and developer community, we have open-sourced the pre-trained maiBERT model on Hugging Face. 2 Dataset Category Count Avg Words Avg Sent Culture 2,575 147.4 8.5 Economy 549 176.8 8.5 EduTech 619 57.0 6.7 Entertainment 844 156.1 8.1 Health 506 58.5 4.3 Interview 424 95.4 6.8 Literature 2,212 155.0 10.3 Opinion 805 123.1 9.9 Politics 2,075 52.1 4.8 Sports 1,774 135.7 8.1 Total 12,383 130.7 7.6 Table 1: Dataset overview: Distribution and statistics across text categories for Count, Average number of words and Sentences. 2.1 Pre-training Corpus: The raw corpus 3 3 3 https://huggingface.co/datasets/rockerritesh/maithiliNewsData for MLM pre-training consists of 1,028,017 sentences, with an average sentence length of 16.88 words and an average word length of 4.44 characters. Figure 2 provides an overview of the Maithili text corpus, illustrating how characters, words, and key linguistic features are distributed across the dataset. Figure 2: Comprehensive text statistics of the Maithili corpus showing character distribution, word counts, and linguistic features. The corpus has a rich vocabulary of 1,357,081 unique tokens, highlighting its lexical diversity. A linguistic analysis revealed frequent occurrences of the Maithili-specific punctuation marker, with a total of 1,044,037 appearances across all text files—providing insights into syntactic structure and sentence boundaries. Figure 3 displays the overall word distribution patterns in the Maithili text corpus. Figure 3: Word distribution patterns in Maithili corpus. For enhanced language modeling, the raw data underwent extensive preprocessing, including normalization, lemmatization, and tokenization using a customized BertTokenizer 4 4 4 https://www.tensorflow.org/text/api˙docs/python/text/BertTokenizer , which preserved linguistic features specific to Devanagari script while filtering out non-Devanagari characters. Classification Dataset: For downstream evaluation, we constructed a separate labeled dataset from multiple credible Maithili news sources, such as Esamaad 5 5 5 https://esamaad.com/ , Maithilijindabad 6 6 6 https://maithilijindabaad.com/ , and Maithili Purnajagran Prakash 7 7 7 https://mppdainik.com/ . Figure 4: Distribution of news categories in the Maithili classification dataset. The annotation process involved 3 native speakers and 2 linguistic experts who labeled the corpus into ten predefined categories, totaling 11,959 labeled instances, as shown in Figure 4 . This distribution reflects the natural occurrence of topics in Maithili news media, with cultural and literary content being predominant. To illustrate the varying levels of content complexity across categories, Figure 5 presents the distribution of word counts for different news sections. Figure 5: Word count distribution across different news categories in the classification dataset. Figure 6 shows the distribution of text lengths across the classification dataset, illustrating the variety in article lengths. Figure 6: Distribution of text lengths in the Maithili classification dataset. To validate the quality and consistency of annotations, inter-annotator agreement was computed using Cohen’s Kappa coefficient, yielding a strong agreement score of 0.82, indicating reliable annotation practices. The labeled dataset was then split into training (70%), validation (15%), and test (15%) subsets to ensure robust performance evaluation. Table 1 provides a comprehensive overview of the classification dataset characteristics across different text complexity levels. 3 Related Works NLP has shown remarkable advancements in foundational tasks such as semantic similarity and text categorization, significantly benefiting high-resource languages but posing persistent challenges for low-resource languages like Maithili ( Chandrasekaran and Mago 2021 ; Zhang et al. 2015 ) . While semantic similarity tasks often depend on pretrained models or lexical resources, distributional semantic models such as Latent Semantic Analysis (LSA) ( Landauer et al.

Read the complete text