Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO
Umer Butt, Stalin Varanasi, Günter Neumann · 2024
Read Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO by Umer Butt, Stalin Varanasi, Günter Neumann (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12997.txt.
Introduction
Umer Butt, Stalin Varanasi, and Günter Neumann build the first large search benchmark for Urdu in a 2024 paper. Urdu is spoken by millions but lacks the datasets that power modern search research. The team machine translated a major English retrieval collection into Urdu and used it to set baseline results.
Systems were first tested with no Urdu training, then fine tuned with a multilingual method adapted to the new data. The tuned model's scores rose clearly over the zero shot baseline on both ranking and recall measures. The authors stress the social stakes: search access in low resource languages is an inclusion issue. Their approach should transfer to other South Asian languages.
Catalog record
| CREATOR | Umer Butt, Stalin Varanasi, Günter Neumann |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO Umer Butt, Stalin Varanasi, Günter Neumann Abstract As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. This paper introduces the first large-scale Urdu IR dataset, created by translating the MS MARCO dataset through machine translation. We establish baseline results through zero-shot learning for IR in Urdu and subsequently apply the mMARCO multilingual IR methodology to this newly translated dataset. Our findings demonstrate that the fine-tuned model (Urdu-mT5-mMARCO) achieves a Mean Reciprocal Rank (MRR@10) of 0.247 and a Recall@10 of 0.439, representing significant improvements over zero-shot results and showing the potential for expanding IR access for Urdu speakers. By bridging access gaps for speakers of low-resource languages, this work not only advances multilingual IR research but also emphasizes the ethical and societal importance of inclusive IR technologies. This work provides valuable insights into the challenges and solutions for improving language representation and lays the groundwork for future research, especially in South Asian languages, which can benefit from the adaptable methods used in this study. 1 Introduction Advancements in Natural Language Processing (NLP), particularly transformer architectures, have revolutionized Information Retrieval (IR), enhancing information access worldwide. However, this progress predominantly benefits high-resource languages, leaving low-resource languages like Urdu underrepresented. Urdu is spoken by over 70 million people primarily in South Asia, and it features unique linguistic characteristics, such as its Perso-Arabic script with right-to-left writing direction and rich morphology [ 2 ] , which pose significant challenges for IR systems. These include complex tokenization, script handling, and morphological analysis, which are also present in some high-resource languages such as Arabic, Chinese, and Russian but have benefited from extensive research and resource availability, making these challenges less of a barrier in those languages. One of the primary challenges in expanding IR capabilities for low-resource languages like Urdu is the scarcity of large-scale, high-quality datasets necessary for training and evaluating models. Creating such datasets manually is resource-intensive and often impractical for languages with limited digital presence. Consequently, to close this gap, researchers have explored alternative methods, such as machine translation of existing datasets, but it introduces other challenges like translation errors and context loss, adversely affecting IR performance. To our knowledge, no prior work has specifically addressed these issues for IR in Urdu, leaving a gap that our work aims to fill. In this paper, we address this issue by translating a large-scale MS MARCO passage ranking dataset [ 3 ] into Urdu using the IndicTrans2 model [ 5 ] . This translation provides a valuable resource for Urdu IR development. We first established baseline performance on our newly created dataset using BM25 and mMarco (a multilingual mT5 model trained on the MS MARCO dataset) [ 4 ] in a zero-shot setting, as it was not originally trained on Urdu. We then fine-tune the model on our Urdu-translated dataset. Our experiments demonstrate significant improvements in Mean Reciprocal Rank (MRR) and Recall metrics compared to the zero-shot results, highlighting the potential of our approach. Our key contributions are: • Creation of the Urdu MS MARCO Dataset: We provide the first large-scale IR dataset for Urdu by translating the MS MARCO dataset using the IndicTrans2 model, addressing the data scarcity issue. • Establishment of Baseline IR Performance: We conduct experiments using BM25 and a fine-tuned mT5 model to establish baseline retrieval performance on the Urdu dataset. • Insights into Low-Resource Language IR: We analyze the challenges and potential solutions for improving IR in Urdu, providing a foundation for future research in similar languages. • Publicly Available Resources: To facilitate future research in Urdu IR, we make our translated dataset, fine-tuned mT5 model, and code publicly available on Hugging Face 1 1 1 https://huggingface.co/datasets/Mavkif/urdu-msmarco-dataset 2 2 2 https://huggingface.co/Mavkif/urdu-mt5-mmarco and GitHub 3 3 3 https://github.com/UmerTariq1/Urdu_MsMarco_Translation_Retrieval . 2 Related Work The lack of large-scale, high-quality datasets for low-resource languages has been a significant barrier to advancing IR capabilities. To address this data scarcity, several efforts have created multilingual datasets through machine translation. For example, mMarco [ 4 ] extended the MS MARCO dataset [ 3 ] to 13 languages using Helsinki NLP models [ 10 ] and Google Translate. However, Urdu was not included, partly due to the lack of high-quality translation models for the language at that time. Recent developments in machine translation, such as the IndicTrans2 model [ 5 ] , have improved translation quality for South Asian languages, including Urdu. IndicTrans2 achieves a chrF++ 4 4 4 Character-level F-score, used for capturing partial matches and morphological variations across languages score of 68.2 for English-to-Urdu translation, surpassing models like Google Translate. This improvement opens new possibilities for creating large-scale Urdu datasets suitable for training IR models. In the domain of multilingual NLP, models like mT5 [ 11 ] have demonstrated strong performance across multiple languages by pre-training on a massive multilingual corpus. Similarly, several works have focused on multilingual IR. ColBERT-X [ 8 ] extends the ColBERT [ 6 ] architecture to multilingual settings, showing effectiveness in languages included in its training data. Benchmarks like MIRACL [ 12 ] and initiatives like CIRAL [ 1 ] aim to standardize evaluation and improve cross-lingual retrieval, particularly for low-resource scenarios. MIRACL addresses this by providing a diverse set of 18 languages, including both high and low-resource languages from 10 different language families, with a focus on monolingual retrieval. It features a variety of tasks, including question answering and document retrieval, and incorporates challenges like code-switching and dialectal variations. While MIRACL includes a diverse set of languages, it currently lacks a dedicated Urdu corpus. Our approach specifically addresses this gap for Urdu, leveraging the improved translation capabilities of IndicTrans2 to create a large-scale Urdu IR dataset. By fine-tuning the mT5 model on this dataset, we address the challenges posed by Urdu’s script and morphology. This work provides a foundation for future research in low-resource languages and contributes to the inclusivity of IR technologies. 3 Dataset Creation 3.1 Ms-Marco dataset The MS MARCO Passage ranking dataset [ 3 ] 5 5 5 https://microsoft.github.io/msmarco/Datasets.html is a cornerstone in IR research, comprising over 8.8 million passages and more than 500 thousand unique queries each paired with at least one relevant passage. The commonly used development set includes 6,980 queries, while the training set is organized into 39 million triples, each consisting of a query, a relevant passage, and a non-relevant passage. An example of it can be seen in Table 1 . This dataset’s scale and richness have significantly advanced English language IR models. However, replicating such progress for Urdu is challenging due to the absence of equivalent large-scale datasets or the prohibitive costs associated with translating such extensive corpora 6 6 6 estimated around $30,000 for MS Marco, using services like Azure Translator or Google Translation API with their standard rates of 10-20$ per million characters . 3.2 Translation Process To create the Urdu MS MARCO dataset, we translated the entire English MS MARCO dataset, including both passages and queries from the training and development sets, using the open-source IndicTrans2 [ 5 ] model. We utilized the distilled 200-million parameter version of IndicTrans2 to translate the entire MS MARCO dataset