Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction

Mohamed Basem, Islam Oshallah, Ali Hamdi, Khaled Shaban, Hozaifa Kassab · 2025

Read Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction by Mohamed Basem, Islam Oshallah, Ali Hamdi, Khaled Shaban, Hozaifa Kassab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06971.txt.

Introduction

Basem and four co-authors build a two-stage ask tool for the Quran in this twenty twenty-five paper. The holy book's old tongue is rich and hard for search tools to read. Few drill sets exist, so fine-tuned tools strain to learn the field.

Inside, stage one blends tuned tongue tools to rank the right text bits first. Stage two has big chat tools pull the reply from those bits by coached asks. Rank grades on the year's shared task beat all past tools in the run. Pull grades also topped the field by a wide gap in that task. The pair shows blend rank plus coached pulls can serve thin-data fields.

Catalog record

CREATORMohamed Basem, Islam Oshallah, Ali Hamdi, Khaled Shaban, Hozaifa Kassab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06971

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Two-Stage Quranic QA via Ensemble Retrieval and Instruction-Tuned Answer Extraction Mohamed Basem, Islam Oshallah, Ali Hamdi, Khaled Shaban, Hozaifa Kassab Abstract Quranic Question Answering presents unique challenges due to the linguistic complexity of Classical Arabic and the semantic richness of religious texts. In this paper, we propose a novel two-stage framework that addresses both passage retrieval and answer extraction. For passage retrieval, we ensemble fine-tuned Arabic language models to achieve superior ranking performance. For answer extraction, we employ instruction-tuned large language models with few-shot prompting to overcome the limitations of fine-tuning on small datasets. Our approach achieves state-of-the-art results on the Quran QA 2023 Shared Task, with a MAP@10 of 0.3128 and MRR@10 of 0.5763 for retrieval, and a pAP@10 of 0.669 for extraction, substantially outperforming previous methods. These results demonstrate that combining model ensembling and instruction-tuned language models effectively addresses the challenges of low-resource question answering in specialized domains. I Introduction The Holy Qur’an, revealed over 1,400 years ago, remains the primary source of guidance for over 1.8 billion Muslims worldwide. Beyond its religious significance, the Qur’an represents a masterpiece of Classical Arabic literature, containing profound linguistic, historical, and ethical insights that continue to be studied by scholars across multiple disciplines [ 1 ] . However, accessing and understanding specific Quranic content in response to natural language queries presents significant computational challenges that have only recently begun to be addressed by the natural language processing community [ 2 , 3 ] . Question Answering (QA) systems have emerged as a cornerstone technology for information access across diverse domains, from general knowledge retrieval to specialized applications in legal, medical, and educational contexts [ 4 , 5 ] . The development of effective QA systems for religious texts, particularly the Qur’an, represents both a meaningful application with substantial social impact and a challenging technical problem that pushes the boundaries of current Arabic NLP capabilities [ 2 , 6 ] . Quranic Question Answering presents unique challenges that distinguish it from conventional QA tasks. First, the linguistic complexity of Classical Arabic—with its rich morphology, complex syntax, and archaic vocabulary—poses significant difficulties for models primarily trained on Modern Standard Arabic or contemporary dialects [ 7 ] . Second, the semantic richness and multi-layered meanings inherent in Quranic verses require sophisticated understanding that goes beyond surface-level text matching. Third, the limited availability of large-scale, high-quality annotated datasets for Quranic content constrains the development and evaluation of machine learning models [ 8 , 9 , 10 ] . Finally, the cultural and religious sensitivity of the domain requires both precision and humility. Quranic interpretation has traditionally been reserved for scholars, and AI outputs risk being mistaken for authoritative tafsīr, potentially misleading users or fueling sectarian disputes. To avoid this, systems should present answers as informational aids rather than interpretive rulings [ 11 ] . Recent advances in transformer-based language models have shown promising results for Arabic NLP tasks [ 14 , 15 , 12 , 13 ] . However, these models face particular challenges when applied to Classical Arabic texts, where the linguistic gap between training data and target domain significantly impacts performance [ 16 ] . While ensemble methods have proven effective in improving robustness across various NLP tasks [ 17 ] , their application to low-resource, high-complexity domains like Quranic QA remains underexplored [ 18 ] . Similarly, the emergence of instruction-tuned large language models offers new possibilities for few-shot learning in specialized domains, yet their effectiveness for Classical Arabic text understanding has not been thoroughly investigated [ 19 , 20 , 21 ] . The Qur’an QA 2023 Shared Task was designed to address these challenges through two complementary tasks: (1) Passage Retrieval, which requires identifying the most relevant Quranic verses for a given natural language question, and (2) Machine Reading Comprehension (MRC), which focuses on extracting precise answer spans from retrieved passages [ 22 ] . Despite the participation of multiple research teams [ 23 , 24 , 25 , 26 ] , the shared task results revealed significant room for improvement in both tasks, particularly in handling the linguistic complexity and semantic nuances of Quranic texts. This paper proposes a two-stage framework combining dataset expansion with tailored modeling: ensemble methods improve retrieval, while instruction-tuned LLMs with prompting enhance extraction. Our main contributions are: • Dataset Enhancement Strategy: We expand the original dataset from 251 to over 2,000 questions through systematic paraphrasing and integration of diverse Quranic resources, significantly improving model training coverage and linguistic diversity. • Ensemble-Based Retrieval Framework: We develop and evaluate multiple ensemble strategies for combining fine-tuned Arabic language models, achieving substantial improvements in passage retrieval accuracy. • Instruction-Tuned Extraction Approach: We demonstrate that few-shot prompting with instruction-tuned large language models significantly outperforms fine-tuned approaches for answer extraction in low-resource settings. • State-of-the-Art Results: Our framework achieves new benchmarks on the Quran QA 2023 Shared Task, with MAP@10 of 0.3128 and MRR@10 of 0.5763 for passage retrieval, and pAP@10 of 0.669 for answer extraction. The remainder of this paper is organized as follows: Section II reviews related work in Arabic QA and Quranic text processing. Section III describes our methodology, including dataset enhancement, model architectures, and ensemble strategies. Section IV presents comprehensive experimental results and analysis. Section V discusses implications and limitations, and Section VI concludes with future research directions. II Related Work This section surveys prior research relevant to our study on Quranic Question Answering (QA), covering Arabic QA systems, ensemble methods for retrieval, instruction-tuned language models for answer extraction, and specific efforts in Quranic QA. We highlight how these works inform our approach to tackling the linguistic and semantic challenges of Classical Arabic in the Qur’an. II-A Arabic QA Datasets The field of Arabic Question Answering (QA) has advanced significantly, driven by the adoption of deep learning techniques and transformer-based models [ 27 ] . Datasets like Arabic-SQuAD, which rely on machine-translated questions, suffer from quality issues that limit model performance in specialized domains [ 27 ] . To address this, recent efforts have introduced more robust and domain-specific resources. For example, the ArabicaQA dataset [ 10 ] contains over 89,000 answerable and 3,700 unanswerable questions derived from Arabic Wikipedia. These models are typically fine-tuned on domain-specific datasets such as the Quranic Passage Collection (QPC), the Arabic Reading Comprehension Dataset (ARCD), and newer large-scale corpora aimed at improving generalization in non-English QA. II-B Arabic QA Systems Several Arabic QA systems have been evaluated on benchmarks such as Arabic-SQuAD and ARCD. Mozannar et al. [ 28 ] reported F1 scores of 48% on Arabic-SQuAD and 51% on ARCD, highlighting the challenges in processing Arabic text even with advanced neural architectures. In addition, early efforts focused on religious texts. Durachman and Adriani [ 29 ] built a system to extract answers from Indonesian Quran translations, achieving only 60% accuracy. Similarly, Bhaskar et al. [ 30 ] proposed a rule-based multi-task QA system combining biomedical tasks using ensemble methods and NLTK-style pipelines, but lacked deep semantic modeling in general-purpose or Quranic Arabic. II-C Ensemble Approaches Ensemble methods improve QA systems, as shown in MRQA 2019 [ 17 ] where combining BiDAF, QANet,and BERT enhanced eneralization. However, Quranic QA presents unique challenges since retrieving isolated verses (āyāt) risks unintended doctrinal

Read the complete text