Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios

Vera Pavlova, Mohammed Makhlouf · 2025

Read Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios by Vera Pavlova, Mohammed Makhlouf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15380.txt.

Introduction

Vera Pavlova and Mohammed Makhlouf build a search tool for Islamic texts in many languages. Search across languages has grown in the lab, but real tools that people can use are rare. The pair use the Quran, which exists in many tongues, as their test ground. Their goal is one small model that serves users well in each language.

They train eleven search models in four ways, from single language training to a mixed plan that joins two methods. The mixed plan works best across their tests. They also study how each plan shapes the model's inner word space, and they weigh the cost of running one model in real service. Their work points to a cheap, shared design.

Catalog record

CREATORVera Pavlova, Mohammed Makhlouf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15380

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios Vera Pavlova, Mohammed Makhlouf Abstract Despite recent advancements in Multilingual Information Retrieval (MLIR), a significant gap remains between research and practical deployment. Many studies assess MLIR performance in isolated settings, limiting their applicability to real-world scenarios. In this work, we leverage the unique characteristics of the Quranic multilingual corpus to examine the optimal strategies to develop an ad-hoc IR system for the Islamic domain that is designed to satisfy users’ information needs in multiple languages. We prepared eleven retrieval models employing four training approaches: monolingual, cross-lingual, translate-train-all, and a novel mixed method combining cross-lingual and monolingual techniques. Evaluation on an in-domain dataset demonstrates that the mixed approach achieves promising results across diverse retrieval scenarios. Furthermore, we provide a detailed analysis of how different training configurations affect the embedding space and their implications for multilingual retrieval effectiveness. Finally, we discuss deployment considerations, emphasizing the cost-efficiency of deploying a single versatile, lightweight model for real-world MLIR applications. The system is deployed online 1 1 1 https://rttl.ai/ . 1 Introduction MLIR is a challenging area of research that has seen significant advancements recently, mainly due to the use of large language models (LLMs) ( Nair et al. 2022 ; Lawrie et al. 2022 ) . However, there remains a considerable gap between research efforts and the actual deployment of MLIR systems in real-world scenarios. Many studies show impressive results in controlled environments or benchmark datasets, but typically focus on evaluating the IR model in a specific setting. However, many real-world applications often require a combination of various search scenarios within a single IR system —be it multilingual, cross-lingual, or monolingual. One example of such an application is a retrieval task for the Holy Quran. Retrieving passages from the Holy Quran is uniquely challenging. With translations in over 100 languages, it offers a rich parallel collection of high-quality human translations ( Bashir et al. 2023 ) . This unique feature provides an excellent opportunity to explore the multilingual potential of retrieval models and eliminates the bottleneck of applying machine translation (MT), simplifying and streamlining the evaluation process. This study examines training approaches for deploying a single retrieval model across diverse MLIR settings, enabling modern search capabilities in the Islamic domain. The goal is to help users efficiently locate relevant Quranic passages in multiple languages and access the cultural and religious heritage preserved within Islamic texts, serving both scholars and the general public. We utilize the XLM-R Base model ( Conneau et al. 2020 ) , a multilingual model trained for a general domain, as a backbone for retrieval. It is known that the performance of retrieval models typically deteriorates due to domain shift ( Thakur et al. 2021 ) . As a preliminary step, we conduct a brief domain adaptation of the XLM-R Base model using a small multilingual domain-specific corpus (approximately 100M words). This short round of pre-training resulted in significant performance improvements in retrieval tasks. Moreover, to reduce the model’s size, we conduct language reduction on the XLM-R Base model, which allows us to eliminate languages that are not needed for the current deployment, resulting in more than a 50% reduction in the model’s size. We prepare eleven retrieval models using this lightweight domain-specific multilingual large language model (MLLM) by applying four different training approaches: monolingual, cross-lingual, translate-train-all, and a proposed mixed approach that combines cross-lingual and monolingual techniques. Evaluation across monolingual, cross-lingual, and multilingual retrieval scenarios demonstrates that our proposed mixed training approach produces promising results in all settings. We conduct an in-depth analysis of the potential effects of different training configurations on the embedding space and their impact on multilingual retrieval. Additionally, we discuss the advantages of deploying a single lightweight model for various potential deployment scenarios. Our main contributions are: (1) We prepared eleven retrieval models trained using different approaches and conducted rigorous testing. This allowed us to evaluate how these models perform in various retrieval settings without limiting the evaluation to each model’s specific training approach. (2) We propose a mixed training approach that achieves competitive performance across monolingual, cross-lingual, and multilingual retrieval scenarios. (3) We deploy our model as a part of a free online multilingual search tool designed to explore Quranic text in multiple languages. 2 Preliminaries In this work, we use the term MLIR in its broadest sense. This includes monolingual IR in any language other than English, cross-lingual IR as a special case of MLIR, and MLIR itself, which enables the processing of queries in any language while retrieving relevant documents in multiple languages ( Oard and Dorr 1996 ; Oard and Dorr 1998 ) . We experimented with four languages: English, Arabic, Urdu, and Russian. The choice of languages is motivated by the availability of the evaluation dataset and diversity. Before we proceed to the details of the training approach and experiment, we briefly discuss the preparation of a lightweight domain-specific MLLM that serves as a backbone of the retrieval models. MLLMs provide cross-lingual functionality but are heavy to deploy in low-resource settings due to their large size ( Devlin et al. 2019 ; Lample and Conneau 2019 ; Conneau et al. 2020 ) . Language reduction is a promising approach in the deployment environment ( Abdaoui et al. 2020 ) . It decreases the model size by pruning only the embedding matrix and removing the languages that are not needed in deployment while preserving all encoder weights. We use XLM-R Base to perform language reduction and trim its size from 1.1 GB down to 481MB by keeping the vocabulary only of languages of interest: English, Arabic, Urdu, and Russian (XLM-R-4 model). We follow the technique of Pavlova and Makhlouf 2024 . The detailed steps and model comparison are listed in the Appendix A . 2.1 Domain Adaptation of MLLM While it is relatively easy to find a small amount of data for unsupervised pre-training, the absence of domain-specific labeled data for downstream tasks is a common problem. By leveraging continued pre-training and the integration of new domain-specific vocabulary ( Lee et al. 2019 ; Huang et al. 2019 ; Gu et al. 2020 ; Beltagy et al. 2019 ; Pavlova and Makhlouf 2023 ; Pavlova 2025 ) , we perform domain adaptation using a small corpus of 100 million words only. We combine a random subset of 50 million words from The Open Islamicate Texts Initiative (domain-specific corpus in Arabic) ( Romanov and Seydi 2019 ) with texts in English, Russian, and Urdu, mainly consisting of Tafseer and Hadith, also totaling 50 million words. We train a new Islamic tokenizer based on this corpus, add new domain-specific tokens to the existing vocabulary of the XLM-R-4 model, and continue a short round of pre-training on this assembled multilingual Islamic corpus (XLM-R-4-ID model). For more details on the hyperparameters, refer to Appendix A . As we will show below, this short pre-training round significantly boosted model performance on retrieval tasks. Models EN AR UR RU avg. XLM-R-EN 0.365 0.057 0.291 0.305 0.254 EN-monolingual 0.377 0.430 0.373 0.339 0.380 AR-monolingual 0 .436 0.416 0.400 0.337 0 .397 ENq-ARc 0.441 0.358 0.381 0.333 0.378 ARq-ENc 0.133 0.406 0 .427

Read the complete text