CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems

Soham Bhattacharjee, Mukund K Roy, Yathish Poojary, Bhargav Dave, Mihir Raj, Vandan Mujadia, Baban Gain, Pruthwik Mishra, Arafat Ahsan, Parameswari Krishnamurthy, Ashwath Rao, Gurpreet Singh Josan, Preeti Dubey, Aadil Amin Kak, Anna Rao Kulkarni, Narendra VG, Sunita Arora, Rakesh Balbantray, Prasenj · 2025

Read CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems by Soham Bhattacharjee, Mukund K Roy, Yathish Poojary, Bhargav Dave, Mihir Raj, Vandan Mujadia, Baban Gain, Pruthwik Mishra, Arafat Ahsan, Parameswari Krishnamurthy, Ashwath Rao, Gurpreet Singh Josan, Preeti Dubey, Aadil Amin Kak, Anna Rao Kulkarni, Narendra VG, Sunita Arora, Rakesh Balbantray, Prasenj (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19941.txt.

Introduction

India's languages number in the hundreds, and parallel text between them remains scarce for machine translation. Soham Bhattacharjee and twenty-one co-authors present CorIL, a corpus of seven hundred seventy-two thousand sentence pairs across eleven languages. English anchors the set alongside Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati. Pairs are sorted into government, health, and general domains.

Domain labels support translation systems that adapt to context. The team also benchmarks translation quality on the corpus to show its use. Indian-language-to-Indian-language translation is the neglected direction; most data routes through English. Government services and health information reach citizens only in their own tongues. A curated corpus of this scale is infrastructure for that mission.

Catalog record

CREATORSoham Bhattacharjee, Mukund K Roy, Yathish Poojary, Bhargav Dave, Mihir Raj, Vandan Mujadia, Baban Gain, Pruthwik Mishra, Arafat Ahsan, Parameswari Krishnamurthy, Ashwath Rao, Gurpreet Singh Josan, Preeti Dubey, Aadil Amin Kak, Anna Rao Kulkarni, Narendra VG, Sunita Arora, Rakesh Balbantray, Prasenj
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems Soham Bhattacharjee, Mukund K Roy, Yathish Poojary, Bhargav Dave, Mihir Raj, Vandan Mujadia, Baban Gain, Pruthwik Mishra, Arafat Ahsan, Parameswari Krishnamurthy, Ashwath Rao, Gurpreet Singh Josan, Preeti Dubey, Aadil Amin Kak, Anna Rao Kulkarni, Narendra VG, Sunita Arora, Rakesh Balbantray, Prasenjit Majumdar, Karunesh K Arora, Asif Ekbal, Dipti Mishra Sharma Abstract India’s linguistic landscape is one of the most diverse in the world, comprising over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized as scheduled languages in the Indian Constitution 1 1 1 https://en.wikipedia.org/wiki/Languages_of_India . Despite recent progress in multilingual neural machine translation (NMT), high-quality parallel corpora for Indian languages remain scarce, especially across varied domains. In this paper, we introduce a large-scale, high-quality annotated parallel corpus covering 11 of these languages—English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati—comprising a total of 772,000 bi-text sentence pairs. The dataset is carefully curated and systematically categorized into three key domains: Government, Health, and General, to enable domain-aware machine translation research and facilitate effective domain adaptation. To demonstrate the utility of CorIL and establish strong benchmarks for future research, we fine-tune and evaluate several state-of-the-art NMT models, including IndicTrans2, NLLB, and BhashaVerse. Our analysis reveals important performance trends and highlights the corpus’s value in probing model capabilities. For instance, the results show distinct performance patterns based on language script, with massively multilingual models showing an advantage on Perso-Arabic scripts (Urdu, Sindhi) while other models excel on Indic scripts. This paper provides a detailed domain-wise performance analysis, offering insights into domain sensitivity and cross-script transfer learning. By publicly releasing CorIL, we aim to significantly improve the availability of high-quality training data for Indian languages and provide a valuable resource for the machine translation research community. 1 Introduction India is one of the most linguistically diverse countries in the world. While English and Hindi functions as a common medium for formal communication, particularly in sectors such as business, education, governance, and the judiciary, the multilingual nature of the country underscores the critical need for effective language translation systems. Language translation in India is not merely a matter of convenience but a necessity for fostering social inclusion, ensuring equitable access to information, and maintaining national cohesion. For instance, the successful dissemination of government policies, public welfare schemes, and legal rights relies heavily on the availability of content in regional languages. Furthermore, the country grapples with a significant shortage of medical professionals, particularly in rural areas and community health centers. In such contexts, machine translation can serve as a vital tool to bridge the communication gap between healthcare providers and patients across linguistic boundaries, thereby enhancing the reach of medical services. Moreover, translation facilitates national integration by enabling seamless communication for individuals migrating or traveling across linguistically distinct regions. In this light, the development of robust machine translation systems tailored to India’s linguistic landscape holds substantial promise in addressing both practical and socio-cultural challenges. Indian languages present a unique set of challenges for machine translation due to their morphological richness and linguistic diversity. Recent efforts, particularly the development of corpora and models such as Samanantar [ 1 ] and IndicTrans2 [ 2 ] , have marked significant progress in this space. These models were the first to provide translation support for all 22 scheduled Indian languages recognized by the Government of India, collectively spoken by approximately 97% of the population. However, a persistent limitation in this domain has been the scarcity of high-quality parallel corpora for low-resource languages. The BPCC corpus released alongside IndicTrans2 was constructed through human translation from English to various Indic languages. While valuable, this approach introduces a well-documented issue in low-resource machine translation known as translationese [ 3 ] , wherein language-specific subtleties and authentic expressions are lost due to the non-native origin of the source text. To address this, our work, though it caters to only 11 Indian languages, enhances the existing landscape by sourcing texts originally written in low-resource Indian languages and translating them into both Hindi and English. By anchoring our dataset in source-language authenticity, we aim to better preserve linguistic nuances and cultural context. Furthermore, by employing both Hindi and English as pivot languages, we introduce a level of robustness to the data that facilitates the development of future pivot-based machine translation systems. In this work, we release a new parallel corpus spanning 11 Indian languages and three critical domains, comprising a total of 772,000 sentence pairs. To evaluate the utility of this dataset, we conduct extensive experiments using state-of-the-art models, NLLB-200 [ 4 ] , IndicTrans2 [ 2 ] and BhashaVerse [ 5 ] , analyzing their performance trends after fine-tuning on our released corpus. Additionally, we perform domain-specific fine-tuning to assess the effectiveness of domain adaptation. Beyond parallel data, we also provide a linguistically annotated subset of 133,000 sentences, covering part-of-speech (POS) tagging, named entity recognition (NER), morphological features, and chunking. Notably, approximately 40% of the collected sentences originate in native low-resource Indian languages, helping to preserve linguistic authenticity. Detailed language-wise and domain-wise data statistics are presented in Table 1 . By releasing this corpus to the research community, we aim to democratize access to high-quality translation resources and foster further exploration in the field of Indian machine translation. The corpus is freely available at the following link https://huggingface.co/datasets/HimangY/CoRil-Parallel 2 Languages and Domains India’s linguistic landscape is a testament to its rich cultural heritage. Our dataset encompasses 11 of these languages including, Hindi, Punjabi, Urdu, Telugu, Kannada amongst others. These languages are written in diverse scripts, many of which trace their origins to the ancient Brahmi script. For instance, Devanagari is used for Hindi, Sanskrit, and Marathi; Bengali script serves both Bengali and Assamese; Gurmukhi is employed for Punjabi; and the Odia script is utilized for the Odia language. Each script is an abugida, wherein consonant-vowel sequences are written as a unit, with inherent vowels modified by diacritics. This structure accommodates the phonetic richness of Indian languages. This section delves into the languages and domains covered by our dataset in greater detail. 2.1 Language and Linguistic Properties In this work, we started with Hindi and ten Indian languages along with English. The languages involved are Dogri, Hindi, English, Gujarati, Kashmiri, Kannada, Oriya, Punjabi, Telugu, and Urdu. These languages represent three major language families: the Indo-Aryan family includes Dogri, Hindi, Gujarati, Kashmiri, Oriya, Punjabi, and Urdu; the Dravidian family includes Telugu and Kannada; and English, from the Indo-European family, serves as a point of comparison. All the Indian languages considered here follow the Subject-Object-Verb (SOV) word order, whereas English follows the Subject-Verb-Object (SVO) order and uses the Latin script, unlike the Indian languages, which use a variety of Brahmi-derived scripts or the Perso-Arabic script (in the case of Urdu). Sindhi Sindhi is an Indo-Aryan language spoken by approximately 30 million people in Pakistan and around 2.5 million in India. It is written in both Perso-Arabic and Devanagari script, with each script adapted to capture Sindhi’s unique phonology. The coexistence of two scripts poses orthographic and standardization challenges. Morphologically, Sindhi exhibits both simple and complex stems formed via derivation, compounding, and reduplication. Verbs are richly inflected for

Read the complete text