DunbaaBERT: From Sacrifice to Semantics
Iffat Maab Waleed Jamil, Raphael Schmitt · 2026
Read DunbaaBERT: From Sacrifice to Semantics by Iffat Maab Waleed Jamil, Raphael Schmitt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26935.txt.
Introduction
Maab, Jamil, and Schmitt present DunbaaBERT, a set of Urdu RoBERTa models trained from scratch on a deduplicated 17GB Urdu corpus with three vocab sizes. Tests cover acceptability, news sorting, offensive language spotting, and sentiment analysis. The models rival strong multilingual baselines while staying efficient. Bigger vocabularies do not always help, and the 32k model offers the strongest overall efficiency profile.
Catalog record
| CREATOR | Iffat Maab Waleed Jamil, Raphael Schmitt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58229 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26935
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DunbaaBERT: From Sacrifice to Semantics Iffat Maab Waleed Jamil, Raphael Schmitt Abstract Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k tokens on a deduplicated 17 GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP benchmarks covering linguistic acceptability, news classification, offensive language detection, and sentiment analysis while analyzing vocabulary-size effects on performance and efficiency trade-offs. Across benchmarks, the DunbaaBERT variants achieve competitive performance against strong multilingual baselines while consistently maintaining favorable efficiency trade-offs. Interestingly, larger vocabularies do not consistently improve downstream effectiveness, with DunbaaBERT 32k repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license. 1 Introduction Large language models have demonstrated strong performance across many NLP tasks through zero-shot, few-shot, and in-context learning approaches, often reducing the need for task-specific fine-tuning 21 ; 35 ; 40 ; 1 ; 54 . However, the benefits of these methods are not distributed equally across languages. Their effectiveness is most evident in high-resource languages such as English, where large-scale pretraining data and evaluation resources are widely available. In contrast, under-resourced languages such as Urdu remain comparatively underexplored and continue to face challenges in both language understanding and generation due to limited resources 37 and rich morphological structure 18 ; 11 . As a result, task-specific fine-tuning continues to play a central role in Urdu NLP, with prior work relying on supervised models for sentiment and emotion classification 44 ; 33 , hate-speech detection 12 ; 17 ; 37 , named entity recognition 59 , AI-generated text detection 4 , fake news detection 25 , and more recently fact-checking 3 . Nevertheless, compared to higher-resource languages, Urdu still lacks a broad ecosystem of openly available and systematically evaluated encoder-only language models, depsite being spoken by over 100 million people 7 . In Pakistan, Urdu remains a key language of daily communication and cultural expression, even though English continues to hold official status 43 . Standard Urdu is written in an abjad script historically linked to the Persian and Arabic writing traditions 5 ; 55 . However, online and informal communication frequently appears in Roman Urdu, where Latin characters are used to represent Urdu text 51 . In addition, bilingual speakers commonly alternate between Urdu and English, producing code-switched text across multiple linguistic units ( 9 ; 15 ; 27 ) . From an NLP perspective, progress in Urdu is further constrained by the scarcity and fragmentation of available resources. Existing datasets often vary in script, format, annotation scheme, and domain, spanning standard Urdu, Roman Urdu, and code-switched text 45 . Moreover, many studies use different training and evaluation setups, which make fair comparison across methods difficult. The absence of widely adopted benchmark datasets therefore remains a major obstacle to reliable and reproducible progress in Urdu NLP 37 . Recent efforts have partially improved Urdu language coverage through large-scale language-specific encoder models released within multilingual initiatives such as HPLT-BERT 46 ; 23 . However, systematic analyses of Urdu-specific encoder models remain limited, particularly with respect to tokenizer design and vocabulary size. Prior work has shown that tokenization choices can substantially influence model quality in morphologically rich languages 53 , yet such analyses remain largely unexplored for Urdu. In addition, RoBERTa-based architectures have recently demonstrated strong performance-efficiency trade-offs in Portuguese NLP 47 , motivating their investigation in the context of Urdu encoder models. To address these limitations, we introduce DunbaaBERT, a set of Urdu RoBERTa-base models trained with Byte-BPE vocabularies of 32k, 52k, and 96k tokens. We further provide intrinsic and downstream evaluations across multiple Urdu NLP benchmarks and linguistic probing tasks, while analyzing the impact of vocabulary size on model performance, training dynamics, and efficiency trade-offs. All models are released under the MIT license to support future research in Urdu NLP. 2 Related Works Multilingual pretrained encoders have played a central role in extending Transformer-based language modeling to lower-resource languages such as Urdu. mBERT 24 established a strong early multilingual baseline by training a single masked language model on Wikipedia corpora from multiple languages. XLM-R 22 subsequently improved multilingual encoder pre-training by replacing Wikipedia-only supervision with substantially larger Common Crawl-derived multilingual data. More recently, mmBERT 39 has further increased multilingual scale, with the model family reported to cover over 1,800 languages and more than 3T training tokens, thereby including Urdu in a massively multilingual pre-training regime. By contrast, EuroBERT 19 is explicitly designed for a restricted set of 15 European languages and is therefore not directly relevant as an Urdu baseline. While such multilingual encoders are highly important comparison points, they necessarily distribute modeling capacity across many languages and thus differ substantially from monolingual Urdu-focused pre-training. Recent work on Urdu representation learning remains comparatively limited. One potentially relevant Hugging Face model wajidhussain/bert-base-urdu , is difficult to interpret reliably as an Urdu BERT baseline, since its public model card is inconsistent: it is tagged for text generation and includes GPT-2-style descriptive content rather than a standard encoder-style masked-language-model. We therefore treat it with caution and do not rely on it as a reference point. Additional Urdu-language models hosted on Hugging Face likewise suffer from limited documentation and unclear training characteristics. The mahwizzzz/UrduBert model is presented as a RoBERTa-style encoder, but its model card does not provide sufficient details regarding training data, optimization setup, or pre-training duration. Preliminary experiments further suggested that the model may be substantially under-trained. Similar limitations apply to mahwizzzz/Urdu-Bert , whose architecture and training configuration are insufficiently documented and whose downstream behavior likewise indicates possible under-training. Another publicly available checkpoint, eshaaftab900/urdu-bert-base , is described as a BERT-base model, but its model card provides only minimal technical information and lacks reproducibility-relevant details concerning corpus composition, tokenizer construction, and pre-training methodology. Due to these documentation limitations and the resulting difficulty for reliable scientific interpretation, we exclude these models from our primary comparative evaluation. A clearer comparison point is provided by the HPLT Urdu model HPLT-BERT ur 1 1 1 https://huggingface.co/HPLT/hplt_bert_base_ur , which is explicitly described as an encoder-only monolingual masked language model based on LTG-BERT, i.e., a BERT-family architecture adapted for large-scale monolingual pre-training 46 . Its underlying data originate from the HPLT corpus collection 23 , which was constructed from large web-derived resources including Common Crawl and Internet Archive data, with document-level deduplication applied during corpus preparation. This makes HPLT-BERT ur a relevant recent baseline for Urdu encoder pre-training on large-scale crawl-based text. Another noteworthy line of work is RUBERT, introduced by 32 , which targets Roman Urdu rather than standard Urdu written in Perso-Arabic script. RUBERT was obtained by additional pre-training of English BERT on a large Roman Urdu Twitter corpus comprising 54 million tokens and 3 million sentences. While this work demonstrates the feasibility of Urdu-adjacent pre-training in noisy social media settings, its script, domain, and initialization strategy differ substantially from monolingual encoder pre-training for standard Urdu text. For this reason, we