Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models
Zahra Khalila 1 , Arbi Haza Nasution 1 , Winda Monika 4 , Aytug Onan 5 , Yohei Murakami 6 , Yasir Bin Ismail Radi 7 , Noor Mohammad Osmani 10 · 2025
Read Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models by Zahra Khalila 1 , Arbi Haza Nasution 1 , Winda Monika 4 , Aytug Onan 5 , Yohei Murakami 6 , Yasir Bin Ismail Radi 7 , Noor Mohammad Osmani 10 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16581.txt.
Introduction
Zahra Khalila and co-authors test whether retrieval support makes language models trustworthy on Quranic questions, in a 2025 paper. General models often hallucinate, inventing answers that drift from authoritative sources, a serious flaw in religious study. The team examined thirteen open-source models in three size bands, from small three-billion-parameter versions to a seventy-billion flagship.
Each model answered from a purpose-built dataset describing all 114 surahs, with meanings, historical context, and qualities, using retrieval-augmented generation to fetch relevant knowledge first. Human judges scored context relevance, faithfulness, and answer relevance. Large models consistently beat smaller ones at capturing a question's meaning and staying grounded, though one small model did surprisingly well on faithfulness. Scale helped, but retrieval shaped the answers.
Catalog record
| CREATOR | Zahra Khalila 1 , Arbi Haza Nasution 1 , Winda Monika 4 , Aytug Onan 5 , Yohei Murakami 6 , Yasir Bin Ismail Radi 7 , Noor Mohammad Osmani 10 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16581
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models Zahra Khalila 1 , Arbi Haza Nasution 1 , Winda Monika 4 , Aytug Onan 5 , Yohei Murakami 6 , Yasir Bin Ismail Radi 7 , Noor Mohammad Osmani 10 Abstract Accurate and contextually faithful responses are critical when applying large language models (LLMs) to sensitive and domain-specific tasks, such as answering queries related to quranic studies. General-purpose LLMs often struggle with hallucinations, where generated responses deviate from authoritative sources, raising concerns about their reliability in religious contexts. This challenge highlights the need for systems that can integrate domain-specific knowledge while maintaining response accuracy, relevance, and faithfulness. In this study, we investigate 13 open-source LLMs categorized into large (e.g., Llama3:70b, Gemma2:27b, QwQ:32b), medium (e.g., Gemma2:9b, Llama3:8b), and small (e.g., Llama3.2:3b, Phi3:3.8b). A Retrieval-Augmented Generation (RAG) is used to make up for the problems that come with using separate models. This research utilizes a descriptive dataset of Quranic surahs including the meanings, historical context, and qualities of the 114 surahs, allowing the model to gather relevant knowledge before responding. The models are evaluated using three key metrics set by human evaluators: context relevance, answer faithfulness, and answer relevance. The findings reveal that large models consistently outperform smaller models in capturing query semantics and producing accurate, contextually grounded responses. The Llama3.2:3b model, even though it is considered small, does very well on faithfulness (4.619) and relevance (4.857), showing the promise of smaller architectures that have been well optimized. This article examines the trade-offs between model size, computational efficiency, and response quality while using LLMs in domain-specific applications. I Introduction Natural language processing (NLP) has been transformed as a result of the development of large language models (LLMs), which made it possible for these models to handle a wide range of activities. These include summarization and translation, as well as answering domain-specific questions [ 1 ] . Further, these models can even serve as a good annotator for a number of NLP tasks [ 2 ] . Recent studies have explored the use of NLP in various domain-specific including Quranic studies, legal system, and medical field focusing on developing question-answering system. Alnefie et al. (2023) [ 3 ] has evaluated the effectiveness of GPT-4 in answering Quran-related questions and highlighting challenges in context understanding and answer accuracy. However, their work is limited by its reliance on a general-purpose LLM without domain-specific fine-tuning, which affects responce precision for nuance religious queries. Retrieval-Augmented Generation (RAG) has been succesfully applied in general knowledge such as medical domains to mitigate these issues [ 4 ] , and legal domain such as research conducted by Pipitone and Alami (2024) [ 5 ] introduced LegalBench-RAG to evaluate retrieval accuracy in legal question-answering tasks. While significant progress has been made across these domains, challenges related to data quality, retrieval precision, domain-specific adaptation, and computational efficiency persist. This study builds upon these works by benchmarking open-source LLMs using the RAG framework to address the challenges in Quranic knowledge retrieval [ 6 ] while highlighting the balance between model size, performance, and efficiency. Using LLMs in religious or culturally sensitive environments has certain challenges [ 7 ] , including ensuring the accuracy, contextual relevance, and authenticity of the generated responses. These issues are particularly significant when engaging with content derived from religious texts, as distortion or hallucination may result in misunderstandings and a loss of faith in AI systems [ 8 ] . This paper examines the role of LLMs in quranic studies [ 9 ] . We use a dataset from a book about the 114 surahs of the Qur’an [ 10 ] , rather than the Qur’an text itself. This dataset provides thorough information about each surah, including meaning, provenance of revelation, and historical context. Descriptive insights are essential for offering relevant and accurate answers to questions regarding quranic studies. Since Qur’an is the sacred revelation that remains intact in today’s world without any human involvement, and the Qur’an is the only source to link human beings with their creator, an attempt to identify the reliable and trustworthy LLMs is really important. As they provide useful resources for the readers, accuracy and truthfulness must be given due importance while exploring the provided information from such sources. In order to overcome the difficulties associated with hallucination and accuracy, we have implemented a framework known as Retrieval-Augmented Generation (RAG) [ 11 , 12 , 13 ] . This method integrates LLM semantics with a vector database to retrieve relevant data from descriptive datasets [ 14 , 15 ] . The RAG method guarantees that responses derive from authoritative sources that are contextually appropriate, thereby reducing the probability of delivering content that is unsubstantiated or irrelevant. Citations are supplied in each response, which enables users to trace the information back to the descriptive dataset. This further enhances the level of trust and transparency [ 16 ] . The objectives of this research are threefold: (1) to compare 13 open-source LLMs in terms of their ability to respond accurately and faithfully to questions about quranic studies [ 17 ] , (2) to assess the effectiveness of the RAG approach in reducing hallucination and ensuring response relevance [ 18 ] , and (3) to provide insights into the use of descriptive datasets for religious education and AI-based knowledge systems [ 19 ] . The evaluation criteria consist of context relevance, response faithfulness, and answer relevance, and they are evaluated using human evaluation. [ 14 ] . This study provides a robust framework for integrating LLMs with descriptive datasets, thereby contributing to the expanding field of domain-specific AI applications [ 20 ] . It emphasizes the strengths and limitations of current LLMs in managing sensitive religious topics and establishes a foundation for future developments in AI-driven educational and informational tools. The rest of this paper is organized as follows: Section 1 introduces the challenges of using LLMs for Quranic studies and outlines the research objectives. Section 2 reviews related work, discussing previous studies on LLM applications in religious text analysis and RAG-based retrieval systems. Section 3 provides a detailed description of the experimental setup, covering the dataset, NLP tasks and evaluation guidelines, dataset selection and curation, human evaluators, metrics for quality evaluation, large language models, and hardware and software configuration. Section 4 presents the experimental results of various LLM models. Section 5 discusses key findings, including performance insights based on model size, the effectiveness of the RAG framework, the trade-off between computational resources and response quality, the surprising performance of Llama3.2:3b, and implications for domain-specific tasks. Section 6 concludes the paper by summarizing the main contributions and providing suggestions for future research. II Materials and Methods This section outlines the methodical strategy employed in our research. We begin by analyzing the dataset, detailing its source, structure, and descriptive content. Subsequently, the system’s responsibilities are thoroughly delineated, encompassing the formulation of solutions to user concerns concerning Islamic doctrines [ 21 ] . In addition, we provide a summary of the rules that have been set for human evaluators who are responsible for evaluating the outputs of the system. To ensure that the evaluations are reliable and consistent, we have produced these guidelines.