COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing
Rajvee Sheth, Himanshu Beniwal, Mayank Singh · 2025
Read COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing by Rajvee Sheth, Himanshu Beniwal, Mayank Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21670.txt.
Introduction
Rajvee Sheth, Himanshu Beniwal, and Mayank Singh introduce COMI-LINGUA in a 2025 paper, a large expert-annotated data set for Hindi-English code-mixing. Code-mixing blends two tongues in one text, a norm in much of India. Tools for such text lag tools for pure text, in part for want of good training data.
Their set holds well over a hundred thousand cases across five core tasks, from naming the main language to translation. Three bilingual experts marked each case, with strong agreement in the scores. Both scripts of the mix are in the set. Closed models beat open ones in first tests.
Catalog record
| CREATOR | Rajvee Sheth, Himanshu Beniwal, Mayank Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21670
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing Rajvee Sheth, Himanshu Beniwal, Mayank Singh Abstract We introduce COMI-LINGUA, the largest manually annotated Hindi-English code-mixed dataset, comprising 125K+ high-quality instances across five core NLP tasks: Token-level Language Identification, Matrix Language Identification, Named Entity Recognition, Part-Of-Speech Tagging and Machine Translation. Each instance is annotated by three bilingual annotators, yielding over 376K expert annotations with strong inter-annotator agreement (Fleiss’ Kappa ≥ \geq 0.81). The rigorously preprocessed and filtered dataset covers both Devanagari and Roman scripts and spans diverse domains, ensuring real-world linguistic coverage. Evaluation reveals that closed-weight LLMs significantly outperform traditional tools and open-weight models in zero-shot settings. Notably, one-shot prompting consistently boosts performance across tasks, especially in structure-sensitive predictions like POS and NER. Fine-tuning open-weight LLMs on COMI-LINGUA demonstrates substantial improvements, achieving up to 95.25 F1 in NER, 98.77 F1 in MLI, and competitive MT performance, setting new benchmarks for Hinglish code-mixed text. COMI-LINGUA is publicly available at this URL 1 1 1 https://huggingface.co/datasets/LingoIITGN/COMI-LINGUA Version 1.0, Updated till 15 nd September 2025. . 1 Introduction Code-mixing is the blending of multiple languages within a single utterance—a pervasive phenomenon in multilingual societies, especially on social media platforms ( Jamatia et al., 2020 ; Srivastava and Singh, 2020 ) . Over half of the world’s population is bilingual or multilingual and frequently uses mixed-language expressions in digital communication ( Grosjean, 2021 ) . In the Indian context, Hindi-English (Hinglish) code-mixed text is particularly widespread and presents significant computational challenges due to orthographic complexity, frequent language switches, and script variation between Devanagari and Roman forms ( Bali et al., 2014 ; Takawane et al., 2023 ; Thara and Poornachandran, 2018 ) . A characteristic example is: Kal mujhe hai , but will be an issue , where Hindi and English tokens co-occur and certain English words like “ office ” and “ traffic ” may appear in Devanagari script. (English Translation: “Tomorrow I have to go to the office, but traffic will be an issue.”) Despite growing interest, current Hinglish datasets have critical limitations: (1) a predominant focus on Roman script, ignoring natural script variation ( Begum et al., 2016 ; Bali et al., 2014 ; Srivastava et al., 2020 ) , (2) limited scale and coverage ( Srivastava and Singh, 2021a ; Kumar et al., 2018 ; Tiwari et al., 2024 ; Kartik et al., 2024 ) , (3) insufficient task diversity ( Aguilar et al., 2020 ; Khanujaa et al., 2020 ; Bohra et al., 2018 ; Khanuja et al., 2020 ) , and (4) reliance on synthetic data generation and labeling rather than human annotation ( Chatterjee et al., 2022 ; Srivastava and Singh, 2021c ; Kartik et al., 2024 ; Sravani and Mamidi, 2023 ) . Figure 1: Sample Annotations Across COMI-LINGUA Tasks: Shown here are annotated instances for each of the five tasks defined in the COMI-LINGUA task set, emphasizing the annotation strategy and linguistic diversity. To address these limitations, we present a novel comprehensive dataset COMI-LINGUA ( CO de- MI xing and LING uistic Insights on Natural Hinglish U sage and A nnotation) that advances Hinglish code-mixing research. The key contributions include: • Curation of the largest publicly available Hinglish dataset (376K manually annotated instances), released under a CC-BY-4.0 license, capturing real-world code-mixing behavior across both Roman and Devanagari scripts. Each instance is annotated by one annotator across one of the key NLP tasks: token-level language identification, matrix language identification, named entity recognition, part-of-speech tagging and machine translation. • Robust benchmarking of state-of-the-art multilingual LLMs (mLLMs), including both open-weight and closed-weight models—alongside traditional NLP tools, under two inference paradigms: zero-shot and one-shot in-context learning. • In-depth error analysis of mLLMs on code-mixed tasks, uncovering critical limitations such as misclassification of English borrowings in Devanagari script, context truncation, overfitting in one-shot settings, prompt mimicry, repetitive or hallucinated outputs, and practical deployment barriers like API usage constraints—highlighting persistent challenges in script-aware and context-sensitive language modeling. 2 Related Work Code-mixing—the blending of multiple languages in a single utterance—poses major challenges for NLP due to its structural variability ( Srivastava and Singh, 2021a ) . This is especially true for Hinglish, given their distinct scripts and syntax ( Bali et al., 2014 ) . Progress is hindered by the lack of large, annotated datasets, as collecting and labeling such data remains costly and labor-intensive ( Srivastava and Singh, 2021a ) . Language Identification is a foundational task in code-mixed NLP. Multiple approaches have been developed to detect language boundaries within mixed-language sequences, including statistical models, CRFs, and deep learning-based techniques ( Shekhar et al., 2020 ; Singh et al., 2018a ; Gundapu and Mamidi, 2018 ; Molina et al., 2016 ) . These efforts have paved the way for improved preprocessing and downstream modeling of code-mixed data. Named Entity Recognition in code-mixed text has seen significant progress through both resource development and model improvements. Dowlagar and Mamidi (2022) showed that leveraging multilingual data enhances NER accuracy, while Ansari et al. (2019) created cross-script datasets using Wikipedia. Transformer-based approaches and meta-embeddings have also been effective in improving NER for Indian code-mixed data ( Priyadharshini et al., 2020 ) . Part-of-Speech Tagging A variety of annotated datasets have been introduced for POS tagging in code-mixed contexts. Singh et al. (2018b) and Vyas et al. (2014) developed corpora from Twitter and Facebook, respectively, while Pratapa et al. (2018) generated synthetic datasets for evaluating bilingual word embeddings. Sequiera et al. (2015) experimented with various machine learning algorithms, and Chatterjee et al. (2022) introduced PACMAN, a large-scale synthetic POS-tagged dataset that achieved state-of-the-art performance in code-mixed POS tagging tasks. Machine Translation for code-mixed content remains a growing research area. Dhar et al. (2018) and Srivastava and Singh (2020) developed parallel corpora for Hinglish code-mixed sentences, while Hegde and Lakshmaiah (2022) proposed translation models using transliteration and pseudo-translation, achieving competitive results in the MixMT shared task at WMT 2022. Benchmarking and Evaluation Frameworks Several benchmark datasets have been introduced to evaluate NLP systems on code-mixed tasks. LinCE ( Aguilar et al., 2020 ) provides a comprehensive benchmark covering 11 corpora and 4 language pairs. GLUECoS ( Khanuja et al., 2020 ) demonstrated the benefits of fine-tuning multilingual models on code-switched datasets across multiple tasks. Emotion and sentiment annotation efforts, such as the Hinglish Twitter corpus by Vijay et al. (2018) , the L3Cube-HingCorpus ( Nayak and Joshi, 2022 ) , and the emotion-annotated SentiMix dataset by Ghosh et al. (2023) further support affective computing in code-mixed settings. Despite ongoing efforts, standardized benchmarks for evaluating LLMs on diverse Hinglish code-mixed tasks—such as acceptability judgments, syntactic fluency, and translation fidelity—remain limited. Existing benchmarks are often narrow in scope and rely on synthetic or small-scale data. To address this, we curate the largest high-quality, human-annotated dataset for training and evaluating LLMs on a broad range of Hinglish code-mixed phenomena. It serves as both an evaluation suite and a diagnostic tool to advance multilingual and code-mixed language understanding research. 3 The COMI-LINGUA dataset 3.1 Raw Dataset Curation We curated raw data from publicly accessible and licensed platforms spanning diverse categories such as news, politics, entertainment, social events, sports, and informational content, with a focus on the Indian subcontinent. Sources included