SPRING Lab IITM’s submission to Low Resource Indic Language Translation Shared Task

Hamees Sayed, Advait Joglekar, Srinivasan Umesh · 2024

Read SPRING Lab IITM’s submission to Low Resource Indic Language Translation Shared Task by Hamees Sayed, Advait Joglekar, Srinivasan Umesh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00727.txt.

Description

Abstract We develop a robust translation model for four low-resource Indic languages: Khasi, Mizo, Manipuri, and Assamese. Our approach includes a comprehensive pipeline from data collection and preprocessing to training and evaluation, leveraging data from WMT task datasets, BPCC, PMIndia, and OpenLanguageData.

Catalog record

CREATORHamees Sayed, Advait Joglekar, Srinivasan Umesh
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00727

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPRING Lab IITM’s submission to Low Resource Indic Language Translation Shared Task Hamees Sayed, Advait Joglekar, Srinivasan Umesh Abstract We develop a robust translation model for four low-resource Indic languages: Khasi, Mizo, Manipuri, and Assamese. Our approach includes a comprehensive pipeline from data collection and preprocessing to training and evaluation, leveraging data from WMT task datasets, BPCC, PMIndia, and OpenLanguageData. To address the scarcity of bilingual data, we use back-translation techniques on monolingual datasets for Mizo and Khasi, significantly expanding our training corpus. We fine-tune the pre-trained NLLB 3.3B model for Assamese, Mizo, and Manipuri, achieving improved performance over the baseline. For Khasi, which is not supported by the NLLB model, we introduce special tokens and train the model on our Khasi corpus. Our training involves masked language modelling, followed by fine-tuning for English-to-Indic and Indic-to-English translations. 1 Introduction Translation of low-resource languages poses significant challenges in natural language processing. While substantial progress has been made in developing machine translation models for high-resource languages, low-resource languages often suffer from a lack of parallel corpora and digital resources Haddow et al. 2022 . Languages like Khasi, Mizo, Manipuri, and Assamese are representative of this challenge, where limited data and unique linguistic complexities hinder the development of robust translation systems. In recent years, efforts to bridge this gap have gained momentum, driven by initiatives such as the Bharat Parallel Corpus Collection 1 1 1 https://ai4bharat.iitm.ac.in/bpcc/ (BPCC) Gala et al. 2023 and government-supported projects like PMIndia Haddow and Kirefu 2020 , which aim to provide bilingual data for Indic languages. Despite these efforts, translation models for low-resource Indic languages have yet to achieve performance levels comparable to their high-resource counterparts Suman et al. 2023 , necessitating innovative approaches to model training and data utilization. In this work, we develop a robust translation model for four low-resource Indic languages: Khasi, Mizo, Manipuri, and Assamese. Our approach involves data collection, preprocessing, training, and evaluation. We utilize datasets from WMT, BPCC, PMIndia, and OpenLanguageData 2 2 2 https://github.com/openlanguagedata/seed Maillard et al. 2023 , and enhance bilingual data through back-translation Edunov et al. 2018 techniques, especially for Mizo and Khasi, significantly expanding our training corpus. We follow Meta’s data preprocessing standards and use LoRA (Low-Rank Adaptation) Hu et al. 2021 fine-tuning on the NLLB et al. 2022 3.3B model to improve efficiency and performance with fewer parameters. Our model initially focuses on one-way translation from English to the Indic languages, then on reverse translations Dabre et al. 2019 . The results show improved performance over the baseline, particularly for Khasi, where we address gaps in pre-trained model support. 2 Dataset In this study, we focus on four low-resource Indic languages covered in the Low Resource Indic Languages Shared Task: Khasi, Mizo, Manipuri, and Assamese. This section highlights the significance of each language, including their role in their respective regions, their linguistic and cultural importance, and the details of the datasets used. Statistics regarding language speakers are according to the 2011 Indian Census 3 3 3 https://censusindia.gov.in/ . Language ISO-693-3 WMT Parallel BPCC PMIndia OLD Back-Translated Total Assamese asm 50,000 35,354 9,732 0 0 95,086 Manipuri mni 21,687 0 7,419 6,193 0 35,036 Khasi kha 24,000 0 0 0 102,070 126,070 Mizo lus 50,000 0 0 0 30,164 80,164 Table 1: Breakdown of data sources and volumes for each language. ‘‘OLD’’ refers to OpenLanguageData. The ‘‘Back-Translated’’ data was initially generated using Google Translate 5 5 5 https://google.translate.com/ for the first 500k characters from the monolingual WMT task data, with subsequent iterations increasing the data size using the trained model 2.1 Languages Assamese (Asamiya) is an Indo-Aryan language spoken primarily in the northeastern Indian state of Assam, where it serves as an official language and a regional lingua franca. With over 15 million native speakers, it is one of the most widely spoken languages in the region. Historically, Assamese was the court language of the Ahom kingdom. It is written in the Assamese script, an abugida system, known for its unique typographic ligatures. Manipuri (Meiteilon) is a key Tibeto-Burman language spoken mainly in Manipur, India, where it is an official language and it is one of the constitutionally scheduled official languages of the Indian Republic. With 1.76 million speakers, it is the most widely spoken Tibeto-Burman language in India and holds the third place among the fastest-growing languages of India, following Hindi and Kashmiri. It is written in its own Meitei script as well as the Bengali script. Khasi (Ka Ktien Khasi) is an Austroasiatic language primarily spoken by the Khasi people in Meghalaya, India, with approximately 1 million native speakers as of the 2011 census. The language holds an associate official status in certain districts of Meghalaya. Khasi is written in the Latin script. The closest relatives of Khasi are other languages in the Khasic group, such as Pnar and War. Mizo (Mizo ṭawng) belonging to the Sino-Tibetan language family, is primarily spoken in the state of Mizoram, India, with around 800 thousand speakers. The Mizo language, also known as Lushai, has a rich oral history and was first written using the Latin script in the late 19th century. Mizo is recognized as the official language of Mizoram and is used in education, government, and media. 3 Methodology This section covers the preprocessing steps and training methods used, including dataset preparation and the fine-tuning of Meta’s multilingual NLLB 3.3B base pre-trained model. Detailed statistics on data distribution are presented in Table 5 . 3.1 Preprocessing In the preprocessing phase, we followed a series of steps to ensure the text data was clean and consistent before model training. We began by normalizing punctuation using Moses Koehn et al. 2007 , an open-source toolkit designed for preprocessing, training, and testing translation models. This step helps maintain consistency in text data, which is crucial for training robust models. Non-printable characters, which often interfere with text processing, were replaced with a space. This choice ensures that any invisible or non-standard characters do not disrupt the tokenization process and ensures that the text is composed of standard printable characters. We also applied Unicode normalization (NFKC) to transform characters into their canonical forms, making the text more uniform across different Unicode representations. These preprocessing steps are aligned with those outlined by Meta for their multilingual models, and further details can be found on their GitHub 6 6 6 https://github.com/facebookresearch/stopes/blob/main/stopes/pipelines/monolingual/monolingual_line_processor.py . This approach ensures that the text data used for training is clean, consistent, and compatible with the modelling requirements. 3.2 Training For model training, we employed Meta’s NLLB (No Language Left Behind) 3.3B parameter model, a state-of-the-art multilingual machine translation model built to support over 200 languages, making it ideal for tasks involving low-resource languages Tran et al. 2021 ; Yang et al. 2021 . The NLLB 3.3B model is based on a Transformer Vaswani et al. 2023 architecture with 3.3 billion parameters, featuring a dense encoder-decoder design. It includes the following hyperparameters: Hyperparams embed size 2048 ffn size 8192 attn heads 16 enc/dec layers 48 Table 2: Hyperparameters for the baseline pre-trained model. 24 Encoder and 24 Decoder Layers. To fine-tune the model, we employed LoRA, a technique that significantly reduces computational demands and training time by adapting only a small subset of

Read the complete text