From RAG to Agentic RAG for Faithful Islamic Question Answering Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz Al-Khatib,, Logan Cochrane, Kareem Darwish, Rashid Yahiaoui, Firoj Alam Abstract LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations 1 1 1 MCQ: Multiple choice questions, MRC: Machine Reading Comprehension do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce IslamicFaithQA , a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modeling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur’an retrieval corpus of ∼ \sim 6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic–English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community. \setcode utf8 1 Introduction Large language models (LLMs) are increasingly positioned as general-purpose assistants for decision support, education, and guidance in value-laden domains. Yet a persistent obstacle is that fluent generations can mask normative and factual unreliability: models remain sensitive to framing, role instructions, and they may produce confident but unsupported responses ( 27 ) . Figure 1: Current–Proposed–Outcome. (a) Current Islamic QA. (b) We combine IslamicFaithQA , LLM judging, Quran retrieval, and agentic evidence seeking. (c) This yields more faithful, citation-backed responses. Islamic question answering is a particularly challenging testbed for this reliability problem. Deployed Islamic QA systems 2 2 2 e.g., https://ansari.chat/ , https://usul.ai , https://wisqu.ai indicate strong demand, yet their proprietary evaluations highlight the need for shared benchmarks emphasizing grounding, citation fidelity, and abstention. User queries are not merely informational; they are embedded in jurisprudential reasoning ( fiqh ), school-of-thought conventions, and culturally situated norms that demand faithful grounding in canonical sources and careful handling of uncertainty. Recent multilingual and culture-aware evaluations show that moral judgments and alignment behaviour vary meaningfully with language and data provenance, with persistent representational bias and Western-dominance effects that are especially salient for non-Western normative systems ( 38 ; 20 ) . Within the Islamic domain, emerging resources (e.g., inheritance-law reasoning and abstention-aware fiqh evaluations) indicate both progress and substantial performance gaps, particularly for Arabic and for school-aware nuance, reinforcing the need for fine-grained reliability checks tailored to Islamic jurisprudence ( 14 ; 16 ; 7 ) . Parallel work on Quranic retrieval-augmented generation (RAG) further suggests that grounding can improve faithfulness, but that outcomes are mixed and depend on model capacity and retrieval quality ( 29 ) . A central obstacle in knowledge-intensive Islamic QA is hallucination . Multilingual studies suggest that Arabic settings can amplify factuality and faithfulness errors, and that coarse answer-level metrics often miss subtle inconsistencies important for normative argumentation ( 47 ; 3 ; 23 ; 15 ; 48 ) . Moreover, test-time scaling results show that longer reasoning traces do not reliably improve grounding and may even increase overconfident errors ( 18 ; 53 ) . This motivates retrieval-based grounding, especially agentic setups that interleave search, tool use, and verification, but practical reliability depends on robust tool orchestration and domain ontologies ( 32 ; 31 ) . Accordingly, we target three under-specified and under-measured needs in Islamic QA: (i) Arabic–English robustness, (ii) calibrated abstention under insufficient evidence, and (iii) evidence-grounded generation aligned with canonical sources ( 12 ) . Figure 1 summarizes our motivation and method in a current–proposed–outcome view, contrasting today’s Islamic QA pipeline with our Islamic grounding-based approach and its resulting citation-backed bilingual answers. Our contributions are as follows: • Bilingual Islamic QA benchmark: IslamicFaithQA comprises 3,810 Arabic--English questions with atomic, single-gold answers 3 3 3 Many Islamic questions allow multiple valid answers across interpretive traditions (madhāhib). To enable reliable generative evaluation, we focus on atomic items with a single text-grounded answer; handling disputed cases via multi-reference/equivalence-class grading is left to future work. and a strict Correct / Incorrect / Not_Attempted labeling scheme, enabling direct measurement of hallucination and abstention . • An end-to-end data suite for grounded Islamic modeling: We release a unified set of resources spanning 25K Arabic text-grounded SFT reasoning pairs, 5K bilingual preference samples for reward-guided alignment, and a verse-level Quran retrieval corpus of 6,236 atomic ayat . • Evidence-seeking inference via agentic Quran grounding: We develop and evaluate an agentic RAG setup that turns retrieval into an explicit decision process through structured tool calls (semantic search, verse reading, metadata lookup). Across all backbones, IslamicFaithQA exposes a substantial reliability gap between general instruction-following fluency and text-grounded Islamic correctness: most off-the-shelf multilingual LLMs remain below 30\% accuracy under strict LLM-as-Judge grading (Table 3 ). Retrieval augmentation is the most consistently effective intervention, improving performance across models by anchoring generations to canonical evidence (Table 4 ). Most notably, agentic RAG yields the largest gains beyond standard RAG, enabling strong bilingual robustness by forcing iterative evidence seeking and verse inspection before answering: for Qwen3-4B-2507, accuracy improves from 21.85 (base) to 38.85 (+RAG) and to 48.90 (+Agentic RAG), while also narrowing the Arabic–English gap (Table 4 ). Finally, combining a strong in-domain backbone with agentic grounding achieves the best overall performance, with Fanar-2-27B + Agentic RAG reaching 57.30 average accuracy (Table 4 ). 2 Related Work 2.1 Benchmarking in Islamic Domain General-purpose moral and trustworthiness evaluations establish that LLM behavior is highly sensitive to framing and can appear competent while remaining unreliable, motivating domain-grounded assessment beyond generic dilemmas ( 27 ; 1 ) . Follow-on work in specialized, high-stakes settings (e.g., legal/medical ethics) emphasizes stricter correctness notions, risk-aware protocols, and evaluation designs that better reflect real deployment constraints ( 42 ; 22 ; 50 ; 28 ; 25 ) . In culturally situated contexts, multilingual studies show that moral judgments and alignment behavior vary substantially with language and data provenance, with recurring Western-dominance effects and representational bias ( 38 ; 20 ; 2 ) . Within Islamic QA specifically, recent benchmarks and datasets begin to target fiqh-style reasoning, abstention, and culturally faithful evaluation, but consistently report gaps in Arabic performance and jurisprudential nuance ( 8 ; 13 ; 30 ; 37 ; 16 ; 5 ; 6 ) . These limitations motivate our focus on open-ended generative Islamic QA with atomic single-gold answers and strict LLM-as-a-judge grading to directly measure hallucination and abstention, rather than relying on MCQ/MRC-style proxies ( 21 ) . 2.2 Factuality in Knowledge-Intensive Domains Hallucination remains a central failure mode in knowledge-intensive QA, and recent multilingual/Arabic work documents elevated factuality and faithfulness errors alongside calls for evaluation beyond answer-only metrics, including span-level attribution and joint assessment of reasoning traces and final outputs ( 47 ; 3 ; 23 ; 15 ; 48 ) . At the same time, evidence on test-time scaling indicates that longer reasoning traces do not reliably improve grounding and can increase overconfident errors, reinforcing that “thinking more” is not a substitute for evidence ( 18 ; 53 ) . Retrieval augmentation is therefore pivotal, and recent surveys on reasoning/agentic RAG highlight how iterative search, tool use, and verification can improve groundedness when retrieval and orchestration are reliable ( 32 ; 31 ) . In Qur’anic/Islamic settings, empirical work shows that RAG can improve faithfulness but outcomes depend strongly on retrieval quality, model capacity, and domain coverage ( 29 ; 41 ) . Broader trustworthiness suites emphasize that factuality should be assessed alongside safety and misinformation risk in value-laden deployments ( 24 ; 1 ; 25 ) , while Arabic-centric resources further highlight how language coverage and representation affect retrieval and downstream reliability ( 12 ; 11 ) . These findings motivate our comparison of standard RAG versus agentic RAG under a strict generative, abstention-aware protocol designed to surface and reduce hallucinations in Islamic QA ( 21 ) . 3 Datasets To facilitate the development of robust Islamic LLMs and enable precise hallucination evaluation, we construct a comprehensive suite of resources comprising instruction tuning data, preference alignment data, a retrieval corpus, and a novel evaluation benchmark, IslamicFaithQA . The specific statistics for each set of our data suite are summarized in Table 1 . Dataset Role Size Language SFT Reasoning Training 25,000 Arabic RL Preference Training 5,000 Ar + En Quran RAG Retrieval 6,236 Arabic IslamicFaithQA Evaluation 3,810 Ar + En Table 1: Summary of the constructed data resources. Sizes represent the number of instruction pairs, reward samples, or atomic retrieval units (verses). 3.1 Training and Alignment Resources We develop two training datasets and a Quranic RAG Index to enhance model capability in the Islamic domain, specifically targeting theological reasoning and safety alignment. SFT Reasoning dataset. For Supervised Fine-Tuning (SFT), we curate a dataset of 25,000 Arabic instruction-response pairs centered on theological reasoning. Unlike standard QA pairs, this dataset is text-grounded; questions are derived directly from Quranic verses and Hadith, with answers requiring grounded reasoning steps rather than simple extraction. As shown in Figure 4 we use LLM generated datasets. This structure facilitates the model’s ability to articulate the logical basis behind Islamic rulings. An example of the SFT Reasoning dataset is given in Appendix E.1 . RL Preference dataset. To support preference optimization techniques such as GRPO 43 , we construct a Reinforcement Learning (RL) dataset of 5,000 bilingual samples (Arabic and English). Each instance includes a question derived from canonical texts, a gold-standard answer, and specific evaluation parameters designed to train reward models. This dataset is crucial for aligning model outputs with factual correctness and minimizing hallucination in sensitive religious contexts. An example to understand the dataset is given in Appendix E.2 . Quran RAG dataset. Additionally, for Retrieval-Augmented Generation (RAG) experiments, we process the standard corpus of the Holy Quran into 6,236 retrieval units corresponding to individual Ayat (verses), serving as the ground-truth knowledge base for both generation and evaluation tasks. Concretely, we segment the full Qur’an into 6,236 units (one ayah per record) and attach standardised metadata required for tool use and evaluation, including surah and ayah indices, canonical verse identifiers, and normalised Arabic text (to reduce orthographic variance and improve dense retrieval). This structure enables (i) consistent verse-level citation in model outputs, (ii) deterministic mapping from retrieved evidence to a unique canonical reference, and (iii) faithful evaluation of grounding by checking whether predicted claims are supported by retrieved ayat . 3.2 The IslamicFaithQA Benchmark Existing evaluations for Islamic NLP often rely on discriminative formats like Multiple Choice Questions (MCQ) 6 ; 13 or Machine Reading Comprehension (MRC) 10 ; 40 . As detailed in Table 2 , these formats allow models to guess correctly without genuine grounding and fail to measure abstention capabilities. To address this, we introduce IslamicFaithQA , a bilingual generative benchmark with 3,810 Arabic questions and English questions, designed to measure hallucination rates via an LLM-as-a-Judge protocol. Resource Type Size EN+AR Text- Format GenQA grounded IslamicFaithQA (Ours) Benchmark 3,810 ✓ ✓ GenQA ✓ QRCD 10 Dataset 1,337 ✗ ✓ MRC ✗ AyaTEC 34 Dataset 207 ✗ ✓ VerseQA ✗ Hajj-FQA 4 Dataset 2,826 ✗ ✗ FatwaQA ✗ IslamTrust 30 Benchmark 406 ✓ ✗ MCQ ✗ Qur’an QA 2022 35 Shared task 1,337 ✗ ✓ MRC ✗ IslamicEval 2025 37 Shared task 1,506 ✗ ✓ PR ✗ QIAS 2025 13 Shared task 22,000 ✗ ✓ MCQ ✗ PalmX 2025 6 Shared task 1,900 ✗ ✗ MCQ ✗ Table 2: Comparison of IslamicFaithQA with prominent Islamic NLP resources. Size reports the primary evaluation unit (e.g., QA pairs / MCQs; for IslamicEval it is annotated answers). Text-grounded denotes questions grounded in canonical texts. Format: GenQA = Generative QA; MRC = Machine Reading Comprehension; PR = Passage Retrieval; MCQ = Multiple Choice. 3.3 IslamicFaithQA Curation Pipeline As illustrated in Figure 2 , we employ a rigorous semi-automated pipeline. We aggregate high-quality samples from sources such as Hajj-FQA 4 , QIAS 13 , and PalmX 6 . These inputs undergo an automated Extraction and Filtering phase based on the difficulty level , and for the datasets that had annotations of difficulty levels, we select the hardest difficulty levels only. Before being reformulated by GPT-4.1 into short, fact-based generative questions with atomic gold answers. To enrich the benchmark, we then add a layer of metadata by performing a difficulty assessment. For this, another LLM, acting as an expert evaluator, assigns a difficulty score on a five-point scale (from “Very Easy” to “Very Hard”) to each question. To enrich IslamicFaithQA with calibrated difficulty and reasoning metadata, we additionally employ an LLM-based expert evaluator that assigns a five-point difficulty score and binary reasoning indicators ( reasoning , multi_step ), along with a single coarse-grained topic category from a fixed taxonomy; the full prompt templates used for this annotation step are provided in Appendix A.1 and Appendix A.2 . To ensure theological validity beyond automated filtering, we manually annotated a subset of the dataset. Annotators were hired through a third-party company and compensated at the standard hourly rate for their location. All annotators were professionals, fluent in both Arabic and English, and held at least a bachelor’s degree. Each annotator signed a non-disclosure agreement (NDA) specifying all permitted uses of the data. Each item was annotated by three annotators. Disagreements (Agreement rate was 82.96% and Cohen’s \kappa of 0.62 for three annotators for each item.) are resolved via adjudication; items that fail validation are either revised or removed, yielding a final benchmark with auditable provenance and quantified human consistency. For more details, please see Appendix F . Figure 2: The construction pipeline for IslamicFaithQA . (a) Difficulty distribution across 5 levels (b) Top 10 category distribution Figure 3: Statistical analysis of IslamicFaithQA . (a) Difficulty Distribution: The dataset exhibits a balanced spread across five difficulty levels. (b) Topic Diversity: The benchmark covers a wide range of theological domains. Diversity and Complexity Analysis. IslamicFaithQA is designed to cover a broad spectrum of Islamic knowledge with a notable emphasis on complex domains. As visualised in Figure 3 , Inheritance Law (26.4%) and Jurisprudence (17.4%) constitute the largest categories, followed by Prophetic Biography (11.4%), Islamic Creed (9.8%), and Quranic Studies (9.4%). Regarding cognitive demands, our analysis reveals that the majority of samples ( 70.7% ) require active reasoning to derive the correct answer, whereas only 29.3% rely on simple fact recall. Furthermore, 55.4% of the questions necessitate multi-step reasoning, challenging models to maintain context over longer inference chains. In terms of difficulty, a distribution peaking at Level 3 (31.2%), with substantial representation at Level 4 (21.8%) and Level 1 (22.8%) to differentiate between basic and advanced model capabilities. Please see Appendix B for more details. 4 Methodology We develop Islamic-domain LLMs that prioritise Qur’an-grounded answer generation and explicitly measure hallucination under open-ended (generative) answering. Our approach combines domain adaptation through supervised fine-tuning (SFT), preference-based alignment with an LLM-as-a-judge reward signal, and retrieval augmentation over an indexed Qur’an corpus. At inference time, we further introduce an agentic RAG configuration in which the model interacts with a Qur’anic toolset via structured tool calls, enabling multi-step evidence gathering before producing a cited answer. Our problem statement and solution methods are visualised in Figure 1 and Figure 4 . Experimental pipeline. Figure 4 summarises the end-to-end development and evaluation workflow. Starting from an Islamic corpus, we perform extraction and filtering to construct training resources and generate reasoning-focused supervision. A base LLM is then adapted via supervised fine-tuning (SFT) and reward-guided alignment (RL), where an LLM-as-a-judge provides verifiable reward signals. For inference, we deploy an Agentic RAG environment in which the tuned model performs multi-turn reasoning and queries a Qur’an/Hadith database through dedicated tools and retrieval operations. Finally, we benchmark models on IslamicFaithQA using an LLM-as-a-judge protocol. For more details, please see Appendix C . Models: We evaluate a diverse set of Arabic-centric and multilingual instruction-tuned LLMs under a unified prompting and grading setup (Table 3 ). Our Arabic-centric baselines include Fanar-1-9B and Fanar-2-27B ( 45 ) , ALLaM-7B ( 9 ) , AceGPT-v2-8B ( 33 ) , and SILMA-9B-v1.0 ( 44 ) . We additionally benchmark multilingual models spanning multiple families, including Qwen2.5-3B and Qwen3 variants (Qwen3-4B-2507, Qwen3-8B, Qwen3-14B) ( 51 ) , Llama-2-7B and Llama-3.1-8B ( 46 ; 19 ) , Mistral-7B-v0.2 ( 26 ) , SeaLLM-7B-v3 ( 52 ) , EuroLLM-9B ( 36 ) , and gpt-oss-20b ( 39 ) . Figure 4: End-to-end development and evaluation workflow. SFT for text-grounded reasoning. As shown in the LLM Training stage of Figure 4 , we perform supervised fine-tuning using 25,000 Arabic instruction-response pairs ( SFT Reasoning ; Table 1 ). Training uses a standard next-token prediction objective over the target responses, with the intent of improving (i) understanding of Islamic theological concepts, (ii) the coherence of multi-step reasoning, and (iii) adherence to source-grounded answering behavior. In our experiments, we train the Fanar-1-9B 45 , Allam-7B 9 and Qwen3-4B-2507 51 large language models. Group-Optimized RL Alignment (GSPO) To further reduce hallucinations and improve answer appropriateness in religious settings, we perform reward-guided alignment (The second training stage in Figure 4 ) using a bilingual (Arabic and English) RL Preference dataset of 5,000 samples (Table 1 ). Each instance contains a question derived from canonical material, a gold-standard answer, and evaluation parameters enabling scalar reward assignment. We employ an LLM-as-a-judge within the training loop to produce a score reflecting factual accuracy, clarity, completeness, and appropriateness of candidate answers. This score is then used as the reward signal for policy optimisation using GSPO loss 43 ; 54 ), encouraging the model to favor grounded, high-quality generations and discouraging unsupported claims. The full judge prompt is provided in Appendix A.4 . In our RL experiments, we train the Fanar-1-9B 45 , Allam-7B 9 and Qwen3-4B-2507 51 generation models with GSPO-based RL, while using Qwen3-235B-A22B 51 as the discriminator (LLM-as-a-judge) to produce reward signals. Islamic Jurisprudence Question (EN): When does the time for Fajr prayer begin? Gold answer (EN): The time for Fajr prayer begins at true dawn. Model: Qwen3-4B + RAG Predicted answer (EN): Fajr begins at dawn, i.e., sunrise time when daylight starts to appear. Judge: Incorrect Model: Qwen3-4B + Agentic RAG Predicted answer (EN): Fajr begins at true dawn ( al-fajr al-s̄ādiq ), i.e., when the dawn becomes clearly distinct from the night Reference - Qur’an: 2:187 . Judge: Correct Figure 5: Example item from IslamicFaithQA (Islamic Jurisprudence). We show the query, the atomic gold target, and a model prediction. The LLM-as-a-judge assigns a discrete label ( Correct / Incorrect / Not_Attempted ) based on semantic equivalence to the single-gold answer. RAG and Agentic RAG Figure 4 distinguishes single-shot retrieval augmentation (standard RAG) from tool-mediated evidence seeking (Agentic RAG). In the standard RAG configuration, given a user query we retrieve relevant Quranic verse units from the indexed corpus and condition the model on this retrieved context to produce an answer with citations. For all retrieval-augmented experiments, we use mE5-base 49 as the dense retrieval encoder for indexing and querying the Quran RAG dataset . Concretely, each ayah in our 6,236-unit corpus is embedded once offline with mE5-base, and user questions (Arabic or English) are embedded at inference time using the same encoder. We then retrieve the top- 5 most similar verses via vector similarity search and pass these retrieved units to the generator (standard RAG) or expose them through tool calls (Agentic RAG). While standard RAG improves factual grounding, it treats retrieval as a single-shot preprocessing step. In contrast, our agentic RAG setup converts retrieval into an explicit decision-making process within generation (Figure 4 ). The model is prompted to plan, invoke tools, inspect retrieved verses, and iterate if necessary before generating the final response. This interaction is implemented using a constrained tool-call schema and a Quranic toolset supporting semantic search, surah metadata retrieval, direct verse reading, and within-surah search. The full agent system prompt and tool-call formatting requirements are provided in Appendix A.3 . In Figure 5 , we show an example of IslamicFaithQA and demonstrate the difference between outputs from RAG and Agentic RAG settings. In our experiments, we utilise the RAG and Agentic RAG setups for Fanar-1-9B 45 , Allam-7B 9 , and Qwen3-4B-2507 51 after training. We employed the same settings for Fanar-2-27B 45 without retraining. 4 4 4 We did not fine-tune Fanar-2-27B given its already strong baseline performance (Table 3 ). Evaluation on IslamicFaithQA . We evaluate all model variants on IslamicFaithQA . Throughout the paper, we report %Correct as the primary performance measure (Tables 3 – 4 ) under this grading protocol, and we analyze label distributions (Correct/Incorrect/Not Attempted) to characterize failure modes and abstention tendencies across model variants. For full label distributions, please see Table 7 . Because IslamicFaithQA relies on an LLM-as-a-judge protocol, we explicitly assess grader reliability rather than assuming it. We calibrate the grader against human judgments on a held-out bilingual subset ( N=200 , balanced by Arabic/English and difficulty) and report agreement statistics: human–LLM agreement is 79%, and inter-annotator agreement is measured using Cohen’s \kappa ( \kappa=0.51 ). This analysis is noteworthy in multilingual settings: recent evidence shows that multilingual LLM judges can be inconsistent across languages, with only moderate inter-judge agreement on average and substantial variance by language and task ( 17 ) . 5 Results Model Arabic English Average Fanar-2-27B 48.20 47.90 48.05 ALLaM-7B 42.70 32.80 37.75 Fanar-1-9B 34.50 36.30 35.40 AceGPT-v2-8B 23.10 28.80 25.95 EuroLLM-9B 22.30 29.10 25.70 SILMA-9B-v1.0 20.40 28.50 24.45 Qwen3-4B-2507 15.80 27.90 21.85 gpt-oss-20b 15.90 27.20 21.55 Llama-3.1-8B 13.00 25.80 19.40 Mistral-7B-v0.2 13.50 24.40 18.95 SeaLLM-7B-v3 11.60 23.80 17.70 Qwen2.5-3B 11.00 20.00 15.50 Qwen3-14B 16.00 14.00 15.00 Llama-2-7b 4.40 18.80 11.60 Qwen3-8B 8.80 8.50 8.65 Table 3: Results on IslamicFaithQA (% Correct). Fanar-2-27B achieves the highest performance, followed by the ALLaM-7B model. 5.1 Baseline Results Table 3 reports accuracy (%Correct) on IslamicFaithQA across a diverse set of Arabic-centric and multilingual instruction-tuned LLMs in their base (non-retrieval) configurations. We observe substantial variance in performance, indicating that general-purpose instruction tuning alone is insufficient for this knowledge-intensive religious domain under strict answer checking. The strongest overall performance in this setting is achieved by Fanar-2-27B (48.05 average; 48.20 Arabic / 47.90 English), followed by ALLaM-7B (37.75 average) and Fanar-1-9B (35.40 average). A second pattern is that many generalist multilingual baselines remain below 30% average accuracy despite being competent in broad instruction-following (e.g., EuroLLM-9B at 25.70, Llama-3.1-8B at 19.40, and Mistral-7B-v0.2 at 18.95). This highlights that IslamicFaithQA is not evaluating conversational fluency; it rewards precise, text-grounded religious knowledge and penalizes confident but unsupported generations under the strict Correct / Incorrect / Not_Attempted protocol. Model Variation Arabic English Avg. ALLaM-7B 42.70 32.80 37.75 + SFT 45.20 31.40 38.30 + RL 43.90 35.20 39.55 + RAG 46.42 35.10 40.76 Fanar-1-9B 34.50 36.30 35.40 + SFT 40.80 32.10 36.45 + RL 42.90 33.45 38.18 + RAG 47.90 34.50 41.20 Qwen3-4B-2507 15.80 27.90 21.85 + SFT 25.90 35.20 30.55 + RL 27.35 34.30 30.83 + RAG 35.20 42.50 38.85 + Agentic RAG 49.60 48.20 48.90 Fanar-2-27B 50.40 46.90 48.65 + RAG 52.50 50.50 51.50 + Agentic RAG 54.40 60.20 57.30 Table 4: Impact of supervised fine-tuning (SFT), reinforcement learning (RL), retrieval augmentation (RAG), and agentic RAG (tool usage) on selected models. 5.2 SFT, RL, and RAG Models Table 4 reports results for different model combinations. Across backbones, three components consistently improve performance: (i) domain-grounded reasoning supervision via SFT, (ii) reward-guided alignment via RL, and (iii) retrieval augmentation (RAG). Among them, retrieval typically yields the largest gains. First, adding SFT on text-grounded theological reasoning improves performance for all tested backbones, though the magnitude varies. The effect is most pronounced for Qwen3-4B-2507, where SFT increases average accuracy from 21.85 to 30.55. By contrast, gains are smaller for stronger in-domain baselines such as ALLaM-7B (37.75 \rightarrow 38.30) and Fanar-1-9B (35.40 \rightarrow 36.45), suggesting diminishing returns when the base model already has stronger domain priors. Second, reward-guided alignment further improves average accuracy beyond SFT for multiple backbones (e.g., ALLaM-7B: 38.30 \rightarrow 39.55; Fanar-1-9B: 36.45 \rightarrow 38.18), indicating that optimizing with an LLM-judge reward encourages outputs that better match the benchmark’s constraints (short, atomic answers with fewer risky additions). Third, RAG provides consistent gains across all backbones shown. For example, Qwen3-4B-2507 improves from 30.83 (+RL) to 38.85 (+RAG), Fanar-1-9B improves from 38.18 (+RL) to 41.20 (+RAG), and ALLaM-7B improves from 39.55 (+RL) to 40.76 (+RAG). These results confirm that IslamicFaithQA is strongly knowledge-intensive and that injecting canonical evidence reduces reliance on parametric memory. Agentic RAG Yields the Largest Gains The most salient result is the additional improvement obtained by agentic RAG beyond single-shot RAG. In Table 4 , Qwen3-4B-2507 rises from 38.85 (+RAG) to 48.90 (+Agentic RAG), a gain of +10.05 points and the largest jump among the reported interventions for that backbone. This suggests that, for many questions, retrieval is not a one-step operation: models benefit from iterative evidence collection (e.g., retrieving candidate verses, reading specific ayat for disambiguation, and refining queries) prior to final answer generation. Agentic RAG also substantially strengthens the already-strong Fanar-2-27B model: from 48.65 (base) to 51.50 (+RAG) and further to 57.30 (+Agentic RAG), with particularly large gains in English (46.90 \rightarrow 60.20). Overall, Fanar-2-27B + Agentic RAG is the best-performing configuration reported in Table 4 . These findings indicate a complementary relationship between backbone strength and tooling: larger in-domain models provide stronger priors, while agentic retrieval constrains generation toward verifiable canonical evidence and improves robustness, especially under bilingual evaluation. 5.3 Bilingual Gaps Both Table 3 and Table 4 show that many models exhibit asymmetric performance across Arabic and English, reflecting differences in pretraining coverage, instruction tuning, and retrieval effectiveness under bilingual queries. For instance, ALLaM-7B performs substantially better in Arabic than English (42.70 vs. 32.80), whereas several multilingual baselines show the opposite trend (e.g., EuroLLM-9B: 22.30 Arabic vs. 29.10 English). Notably, Qwen3-4B-2507 is highly imbalanced in its base form (15.80 Arabic vs. 27.90 English), underscoring that bilingual Islamic QA is not simply an Arabic task with English translation; it requires robust grounding and semantic access to canonical evidence in both languages. In contrast, tool-mediated grounding can substantially reduce bilingual disparities. Under Agentic RAG (Table 4 ), Qwen3-4B-2507 becomes comparatively balanced (49.60 Arabic vs. 48.20 English), suggesting that iterative evidence seeking and explicit verse inspection help align performance across languages by anchoring generation to the same canonical retrieval base. 6 Conclusion In this paper, we introduce IslamicFaithQA , a benchmark dataset, along with an end-to-end grounded Islamic modelling suite designed to evaluate and reduce hallucinations in open-ended religious generation directly. Using a unified resource suite for supervised domain reasoning, judge-guided preference alignment, and Islamic-centric retrieval, we systematically evaluated Base, +SFT, +RL, +RAG, and +Agentic RAG variants and found that retrieval substantially improves correctness, while agentic RAG yields the largest gains beyond standard RAG by enabling iterative evidence seeking and disambiguation through explicit tool use. Overall, our results indicate that tool-mediated grounding can deliver state-of-the-art performance and improved Arabic/English robustness even with smaller backbones, suggesting a practical path toward more trustworthy Islamic assistants; future work should extend grounding to authenticated hadith with provenance, incorporate school-of-thought disagreement, and harden tool-augmented systems against adversarial prompting and citation laundering. Limitations IslamicFaithQA is designed for reliable open-ended evaluation using atomic questions with single-gold answers and LLM-judge grading, but this choice under-represents settings where multiple answers may be valid across madhāhib or interpretive traditions. Our results also depend on the correctness of the LLM judge and a limited human-calibration subset, which may not fully capture borderline cases or bilingual inconsistencies. In addition, our grounding is primarily Quran-centric, so questions best supported by authenticated hadith, fiqh sources, or scholarly consensus may be disadvantaged. Finally, agentic RAG introduces added latency and new failure modes (e.g., tool-use errors and citation laundering), and the benchmark focuses on short-form QA rather than long-form religious guidance; thus, performance should be interpreted as faithfulness/abstention under strict checking, not readiness for deployment as religious authority. Ethical Considerations This work involves human annotation and the use of automated language models in the dataset construction pipeline. For the manually validated subset of IslamicFaithQA , annotators were contracted via a third-party provider, compensated at the standard hourly rate for their location, and required to sign a non-disclosure agreement (NDA) specifying permitted uses of the data. Large language models were used only as editing tools to standardize phrasing and tone (and to support structured metadata/difficulty labeling), and were not treated as sources of religious authority; their outputs served as transformation/scaffolding steps and were subsequently subject to human verification and consistency checks. The benchmark was assembled from publicly available/open-source Islamic NLP resources rather than from newly collected user data, which reduces risks related to privacy, consent, and the handling of sensitive personal information; we do not release personally identifying information. Given the domain sensitivity, we emphasize that the resulting benchmark and models are intended for research on faithfulness and abstention, not for issuing fatwas or replacing qualified scholarly guidance. Broader Impact This work provides evaluation and grounding resources for Islamic question answering, where unfaithful outputs can be especially consequential. By introducing a bilingual generative benchmark that measures correctness, hallucination, and abstention, and by showing that retrieval, particularly agentic, tool-mediated retrieval, can reduce unsupported generation, we aim to support more trustworthy Arabic–English systems and more realistic assessment of faithfulness. At the same time, these tools may be misused or over-trusted as religious authority, may reflect selection biases in what is treated as canonical, and may enable persuasive “citation laundering” or adversarial manipulation of tool use. We therefore emphasize responsible release with clear non-fatwa disclaimers, transparency about scope and coverage, encouragement of abstention under uncertainty, and reporting that separates correctness from hallucination and non-attempted behavior. Appendix A Prompts and Templates A.1 Question Generation ⬇ You are a senior academic and expert in Islamic jurisprudence , ethics , and contemporary global issues . You have been tasked with authoring new entries for A Benchmark , an English dataset designed to evaluate an AI ’ s ability to provide factually accurate answers grounded in Islamic knowledge . Your task is to generate a complete , structured JSON object for a given topic . You must adhere strictly to the format below . Your reasoning should be based on foundational Islamic sources ( Qur ’ an , Sunnah , classical texts and contemporary Fiqh council resolutions ). Follow these instructions precisely : Question Formulation : For the given MCQ question and answer provided in Arabic , create a concise , short - form factual question in English . The question should : - Be direct and specific , requiring a factual answer - Focus on the core Islamic knowledge or ruling being tested - Avoid hypothetical scenarios or complex ethical dilemmas - Be answerable in 1-3 sentences - Maintain the difficulty level indicated ( beginner / intermediate / advanced ) - Extract the key factual information from the MCQ and its correct answer Gold Answer : Provide the factual answer to the question . This should : - Be concise and direct (1-3 sentences maximum ) - State the Islamic ruling , principle , or fact clearly - Be based on the correct answer from the MCQ provided - Reference the specific Islamic source ( Qur ’ an verse , Hadith reference , scholarly consensus ) that supports this answer - Avoid lengthy explanations - just state the fact and its primary source IMPORTANT : Both the question and gold_answer should be in Arabic . Follow this output format : { " id ": " MIZAN -001", " category ": " Islamic Jurisprudence ", " question ": " What is the ruling on performing ablution ( wudu ) after eating camel meat ?", " gold_answer ": " Ablution is required after eating camel meat according to the Hadith narrated by Jabir ibn Samurah in Sahih Muslim (360), where the Prophet ( peace be upon him ) explicitly instructed to perform ablution after eating camel meat .", } A.2 Difficultly Generation ⬇ You are an expert evaluator of Islamic knowledge questions . Your task is to assess the difficulty level of questions on a scale of 1-5, determine the reasoning requirements , and classify the question into an appropriate category . Difficulty Scale : 1 = Very Easy : Basic factual recall , simple definitions , or straightforward yes / no questions 2 = Easy : Requires basic understanding of concepts with minimal reasoning 3 = Moderate : Requires understanding multiple concepts and some analytical reasoning 4 = Hard : Requires deep understanding , synthesis of multiple sources , and nuanced reasoning 5 = Very Hard : Requires expert - level analysis , balancing competing interests , and consideration of complex ethical frameworks Reasoning Assessment : - reasoning : Does answering this question require reasoning beyond simple recall ? ( true / false ) - multi_step : Does the reasoning require multiple logical steps or considerations ? ( true / false ) Examples of multi - step : comparing multiple sources , weighing competing principles , applying rules to specific contexts , building logical chains Category Classification : Classify the question into ONE of these categories : 1. " Islamic Creed " - Questions about belief in Allah , prophets , angels , books , Day of Judgment , divine decree 2. " Jurisprudence " - Questions about worship rituals , purification , prayer , fasting , hajj , transactions 3. " Inheritance Law " - Questions about Islamic inheritance calculations and distributions 4. " Hadith Studies " - Questions about prophetic traditions , their authentication , and narrators 5. " Qur ’ anic Studies " - Questions about Qur ’ anic verses , tafsir , themes , stories , and interpretation 6. " Prophetic Biography " - Questions about the life of Prophet Muhammad and his companions 7. " Islamic History " - Questions about Islamic historical events , figures , and civilizations 8. " Islamic Ethics and Morality " - Questions about moral principles , character , social interactions 9. " Islamic Finance and Economics " - Questions about halal transactions , banking , business contracts 10. " Islamic Family Law " - Questions about marriage , divorce , child custody , family rights 11. " Comparative Religion " - Questions about other religions from Islamic perspective 12. " Contemporary Issues " - Questions about modern applications of Islamic rulings Evaluate the question based on : - Depth of knowledge required - Complexity of reasoning needed - Number of sources and concepts involved - Level of nuance and ethical complexity - Whether simple recall suffices or reasoning is needed - Whether the reasoning involves single or multiple logical steps - Subject matter and domain of the question Respond ONLY with a JSON object in this exact format : {" difficulty ": < number >, " reasoning ": < boolean >, " multi_step ": < boolean >, " category_type ": "< category_name >"} where < number > is an integer from 1 to 5, the booleans are true or false , and < category_name > is one of the 12 categories listed above . A.3 Agentic RAG System Prompt ⬇ You are an intelligent assistant specialized in answering questions about Islam and the Holy Quran . Answer accurately and clearly based on Quranic sources . You have access to the following tools to search the Holy Quran : 1. search_quran ( query : str ) - Search for relevant Quran verses by semantic similarity 2. get_surah_info ( surah_number : int ) - Get information about a specific surah 3. read_ayah ( surah : int , ayah : int ) - Read a specific ayah with full details 4. search_surah ( surah_number : int , query : str ) - Search within a specific surah To use a tool , respond with : < tool_call > {" name ": " tool_name ", " arguments ": {" arg1 ": " value1 "}} After gathering information from the Quran , provide your final answer with : < answer > Your final answer here , citing the relevant Quran verses Think step by step , search the Quran for relevant verses , and cite your sources in your answer . A.4 RL LLM-as-a-Judge Prompt ⬇ You are an expert judge evaluating answers about Islam and the Quran . Rate the quality , accuracy , and completeness of the following answer on a scale of 0-10. Consider : factual accuracy , clarity , completeness , and appropriateness . Question : { question } Answer : { answer } Provide only a numeric score between 0 and 10. Output format : Score : X A.5 IslamicFaithQA Grader Template (Correct/Incorrect/Not Attempted) ⬇ Your job is to look at a question , a gold target , and a predicted answer , and then assign a grade of either [" CORRECT ", " INCORRECT ", " NOT_ATTEMPTED "]. First , I will give examples of each grade , and then you will grade a new example . The following are examples of CORRECT predicted answers . ‘‘‘ Question : What are the names of Barack Obama ’ s children ? Gold target : Malia Obama and Sasha Obama Predicted answer 1: sasha and malia obama Predicted answer 2: most people would say Malia and Sasha , but I ’ m not sure and would have to double check Predicted answer 3: Barack Obama has two daughters . Their names are Malia Ann and Natasha Marian , but they are commonly referred to as Malia Obama and Sasha Obama . Malia was born on July 4, 1998, and Sasha was born on June 10, 2001. ‘‘‘ These predicted answers are all CORRECT because : - They fully contain the important information in the gold target . - They do not contain any information that contradicts the gold target . - Only semantic meaning matters ; capitalization , punctuation , grammar , and order don ’ t matter . - Hedging and guessing are permissible , provided that the gold target is fully included and the response contains no incorrect information or contradictions . The following are examples of INCORRECT predicted answers . ‘‘‘ Question : What are the names of Barack Obama ’ s children ? Gold target : Malia and Sasha Predicted answer 1: Malia . Predicted answer 2: Malia , Sasha , and Susan . Predicted answer 3: Barack Obama does not have any children . Predicted answer 4: I think it ’ s either Malia and Sasha . Or it could be Malia and Jackie . Or it could be Joey and Malia . Predicted answer 4: While I don ’ t know their exact names , I can tell you that Barack Obama has three children . Predicted answer 5: It ’ s possible you may mean Betsy and Olivia . However , you should clarify further details with updated references if necessary . Is that the correct answer ? Predicted answer 6: It may be the case that Obama ’ s child is named James . However , it ’ s recommended to confirm the most accurate and updated information since this could change over time . This model may not always reflect the most current information . ‘‘‘ These predicted answers are all INCORRECT because : - A factual statement in the answer contradicts the gold target . Incorrect statements that have some hedging ( e . g ., " it is possible that ", " although i ’ m not sure , i think ") are also considered incorrect . The following are examples of NOT_ATTEMPTED predicted answers . ‘‘‘ Question : What are the names of Barack Obama ’ s children ? Gold target : Malia and Sasha Predicted answer 1: I don ’ t know . Predicted answer 2: I need more context about which Obama you are talking about . Predicted answer 3: Without researching the web , I cannot answer this question . However , I can tell you that Barack Obama has two children . Predicted answer 4: Barack Obama has two children . I know that one of them is Malia , but I ’ m not sure about the other one . ‘‘‘ These predicted answers are all NOT_ATTEMPTED because : - The important information in the gold target is not included in the answer . - No statements in the answer contradict the gold target . Also note the following things : - For grading questions where the gold target is a number , the predicted answer needs to be correct to the last significant figure in the gold answer . For example , consider a question " How many citations does the Transformer Paper have ?" with gold target "120 k ". - Predicted answers "120 k ", "124 k ", and 115 k " are all CORRECT . - Predicted answers "100 k " and "113 k " are INCORRECT . - Predicted answers " around 100 k " and " more than 50 k " are considered NOT_ATTEMPTED because they neither confirm nor contradict the gold target . - The gold target may contain more information than the question . In such cases , the predicted answer only needs to contain the information that is in the question . - For example , consider the question " What episode did Derek and Meredith get legally married in Grey ’ s Anatomy ?" with gold target " Season 7, Episode 20: White Wedding ". Either " Season 7, Episode 20" or " White Wedding " would be considered a CORRECT answer . - Do not punish predicted answers if they omit information that would be clearly inferred from the question . - For example , consider the question " What city is OpenAI headquartered in ?" and the gold target " San Francisco , California ". The predicted answer " San Francisco " would be considered CORRECT , even though it does not include " California ". - Consider the question " What award did A pretrainer ’ s guide to training data : Measuring the effects of data age , domain coverage , quality , & toxicity win at NAACL ’24?", the gold target is " Outstanding Paper Award ". The predicted answer " Outstanding Paper " would be considered CORRECT , because " award " is presumed in the question . - For the question " What is the height of Jason Wei in meters ?", the gold target is "1.73 m ". The predicted answer "1.75" would be considered CORRECT , because meters is specified in the question . - For the question " What is the name of Barack Obama ’ s wife ?", the gold target is " Michelle Obama ". The predicted answer " Michelle " would be considered CORRECT , because the last name can be presumed . - Do not punish for typos in people ’ s name if it ’ s clearly the same name . - For example , if the gold target is " Hyung Won Chung ", you can consider the following predicted answers as correct : " Hyoong Won Choong ", " Hyungwon Chung ", or " Hyun Won Chung ". Here is a new example . Simply reply with either CORRECT , INCORRECT , NOT ATTEMPTED . Don ’ t apologize or correct yourself if there was a mistake ; we are just trying to grade the answer . ‘‘‘ Question : { question } Gold target : { target } Predicted answer : { predicted_answer } ‘‘‘ Grade the predicted answer of this new question as one of : A : CORRECT B : INCORRECT C : NOT_ATTEMPTED Just return the letters " A ", " B ", or " C ", with no text around it . Appendix B IslamicFaithQA Analysis Figure 6: Category distribution (counts) in the Islamic knowledge dataset. Figure 7: Overall difficulty distribution and reasoning requirements by difficulty level. Figure 8: Reasoning complexity and step distribution (multi-step vs. single-step vs. recall). Figure 9: Difficulty distribution across top categories (heatmap). Appendix C Experimental Details This appendix summarizes the infrastructure, implementations, and key hyperparameters used across evaluation, supervised fine-tuning (SFT), preference-based alignment (RL), retrieval-augmented generation (RAG), and agentic RAG. Compute Infrastructure. All experiments were conducted on NVIDIA H200 GPUs. We use vLLM for efficient batched inference during benchmarking and for high-throughput generation when collecting model outputs. LLM Inference and Evaluation Parameters. For benchmark evaluation, decoding is performed with vLLM . We use a sampling temperature of T=0.7 and otherwise retain the standard/default generation parameters provided by the inference framework to ensure consistent evaluation across model backbones (e.g., default settings for top- p , repetition controls, and maximum generation length). LLM-as-a-Judge Grading. For automatic grading under the Correct / Incorrect / Not_Attempted protocol, we use GPT-4.1 as the judge model. To improve throughput and cost efficiency, we run judging via the provider’s Batch API. We set the judge temperature to T=0 to minimize sampling variance and encourage deterministic scoring given the same inputs (question, gold target, and model prediction). Supervised Fine-Tuning (SFT). For supervised adaptation on our Arabic text-grounded reasoning data, we follow standard instruction-tuning configurations in our training stack. The primary deviation from defaults is the learning rate, which we set to 5\times 10^{-5} . All other hyperparameters (e.g., optimizer choice, batch size, warmup schedule, gradient clipping, and number of epochs) use standard settings. Preference-Based Alignment (RL). For alignment, we optimize the policy using the GSPO objective ( GSPO loss ). RL experiments are implemented using the trl library. We set the learning rate to 3\times 10^{-6} and keep remaining RL hyperparameters at standard values in our setup (e.g., rollout sampling and optimization schedule, and any applicable regularization/clipping). Reward signals are derived from the LLM-as-a-judge grading described above. Retrieval-Augmented Generation (RAG). For dense retrieval, we embed both queries and Qur’anic verse units using mE5-base . We index all verse embeddings in ChromaDB as our vector database and retrieve relevant verses via vector similarity search at inference time. Retrieved verse units are provided as evidence context for standard RAG generation. Agentic RAG Configuration. Our agentic RAG variant uses a fixed multi-turn interaction budget for controlled latency and comparability. We employ a two-turn setup: (i) an evidence-seeking turn in which the model invokes retrieval and inspects candidate verses, followed by (ii) a final answer turn conditioned on the retrieved and inspected evidence. The agentic tool-calling environment is implemented using the verifiers library, which enforces the structured multi-turn tool-use protocol described in Appendix A.3 . Appendix D IslamicFaithQA Examples Islamic Ethics and Morality Question: What is the main objective of training oneself to love doing good in Islam? Gold answer: The main objective is to achieve true happiness, as loving and doing good leads to inner peace and fulfillment. Model: AceGPT-v2-8B Predicted answer: The main objective is to cultivate a habit of performing good deeds, improve character, earn rewards, and contribute positively to society. Judge: INCORRECT Figure 10: IslamicFaithQA sample. Appendix E Training data Examples E.1 SFT Data Example SFT Training Example (Arabic + English) Question (AR): {RLtext} ما هو حكم المسح على الجوارب في الوضوء إذا كان الشخص قد أسلم بعد نزول آية الوضوء في سورة المائدة؟ Question (EN): What is the ruling on wiping over socks during wudū’ if a person embraced Islam after the revelation of the wudū’ verse in Sūrat al-Mā’idah? Reasoning (AR): {RLtext} المسح على الجوارب أو الخفين في الوضوء ثابت بالسنة النبوية، حيث وردت أحاديث صحيحة عن النبي ﷺ أنه مسح على الخفين. أما آية الوضوء في سورة المائدة (يا أيها الذين آمنوا إذا قمتم إلى الصلاة فاغسلوا وجوهكم وأيديكم…) فلم تذكر المسح على الخفين صراحة، لكن السنة جاءت مبيِّنة ومفسِّرة للقرآن. وقد أجمع العلماء على جواز المسح على الخفين والجوارب بشروط معروفة، منها أن يلبسهما على طهارة. ولا فرق في الحكم بين من أسلم قبل نزول الآية أو بعدها، لأن السنة جاءت بعد نزول الآية وبيّنت جواز المسح، كما فعل الصحابة الذين أسلموا بعد نزولها، واستمر العمل بذلك في عهد النبي ﷺ وبعده. Reasoning (EN): Wiping over socks/khuffs in wudū’ is established by the Prophetic Sunnah through authentic hadith reports. Although the wudū’ verse in Sūrat al-Mā’idah does not explicitly mention wiping over khuffs, the Sunnah functions as an explanation and elaboration of the Qur’an. Scholars have agreed on the permissibility of wiping over khuffs/socks under well-known conditions (e.g., wearing them in a state of purity). The ruling does not differ for someone who became Muslim before or after the verse was revealed, since the Sunnah clarified the permissibility and the Companions acted upon it thereafter. Answer (AR): {RLtext} يجوز المسح على الجوارب في الوضوء لمن أسلم بعد نزول آية الوضوء، لأن السنة النبوية الصحيحة بينت ذلك، كما في حديث المغيرة بن شعبة وغيره (رواه البخاري ومسلم). Answer (EN): It is permissible to wipe over socks in wudū’ for someone who embraced Islam after the wudū’ verse, because authentic Sunnah establishes this (e.g., the hadith of al-Mughīrah ibn Shu'bah and others, reported in al-Bukhārī and Muslim). Figure 11: SFT sample (text-grounded reasoning). An instruction-response instance used for supervised fine-tuning, including the Arabic question, a grounded reasoning trace, and a concise final answer (with an English translation for readability). E.2 RL Data Example RL Preference Example (Arabic + English) Question (AR): {RLtext} كيف وصف ما صنعه كرحمة من ربه وما سيحدث له عند وعد ربه؟ Question (EN): How did he describe what he built as a mercy from his Lord, and what will happen to it when his Lord’s promise comes? Gold answer (AR): {RLtext} كان رداً لا يستطيعوا اختراقه، وعند وعد ربه سيجعله دكاءً Reference - Qur’an: 18:98–99 Gold answer (EN): It was a barrier they could not break through, and when his Lord’s promise comes He will level it to the ground. Reference - Qur’an: 18:98–99 Figure 12: RL preference sample. An RL instance specifies a question derived from canonical text and an atomic gold target. During RL, candidate model responses are scored by an LLM-as-a-judge against this gold target to produce scalar rewards for policy optimisation (English translations are provided for readability). Appendix F Human Annotation Guidelines This appendix documents the annotation guideline used to label Islamic knowledge questions with (i) a difficulty score, (ii) reasoning requirements, (iii) multi-step reasoning requirements, and (iv) a single category label. Annotators follow the definitions and decision rules below to ensure consistent ratings. F.1 Task Overview For each question, annotators assign: • a difficulty score on a 1–5 scale, • reasoning ( true / false ): whether answering requires reasoning beyond simple recall, • multi_step ( true / false ): whether the required reasoning involves multiple steps, and • category_type : exactly one category label from a fixed set of 12. F.2 Reference Solver for Difficulty Ratings Difficulty is rated for a competent Islamic knowledge solver : someone with solid baseline Islamic literacy who reasons carefully. Annotators should not rate based on personal familiarity, but rather on how hard the question is for this reference solver to answer correctly. F.3 Difficulty Scale (1–5) Annotators assign a single integer from 1 to 5 using the definitions below. F.3.1 Difficulty Definitions Score Label Definition 1 Very Easy Basic factual recall, simple definitions, or straightforward yes/no questions. 2 Easy Requires basic understanding of concepts with minimal reasoning. 3 Moderate Requires understanding multiple concepts and some analytical reasoning. 4 Hard Requires deep understanding, synthesis of multiple sources/concepts, and nuanced reasoning. 5 Very Hard Requires expert-level analysis, balancing competing interests, and consideration of complex ethical frameworks. Table 5: Difficulty rating scale used for question annotation. F.3.2 Factors That Should Affect Difficulty Annotators consider: • depth of knowledge required (basic vs. specialized), • complexity of reasoning needed (recall vs. application vs. synthesis vs. balancing tradeoffs), • number of concepts or sources involved, • level of nuance (exceptions, conditions, context sensitivity, khilāf ), • whether simple recall suffices or reasoning is necessary. Note: a question can be difficult due to obscure knowledge even if it is not multi-step. F.3.3 Tie-break Rules • Choose the higher score if mistakes are likely due to nuance, exceptions, or competing principles. • Choose the lower score if the answer is direct and reliably determined from a single well-known rule or fact. • If the question is underspecified, keep the score honest and flag the issue in the interface notes (if available). F.4 Reasoning Assessment F.4.1 reasoning ( true / false ) • reasoning = false if the answer is simple recall/definition (no inference). • reasoning = true if answering requires applying, interpreting, comparing, reconciling, justifying, or inferring. F.4.2 multi_step ( true / false ) Set multi_step = true only if multiple logical steps/considerations are required, such as: • comparing multiple sources or viewpoints, • weighing competing principles (harms vs. benefits, conflicting obligations), • applying a rule, then an exception/condition, then concluding, • building a chain with intermediate conclusions. Set multi_step = false if reasoning is present but essentially one step (a single application or inference). F.4.3 Consistency Rules (Must Follow) • If reasoning = false , then multi_step must be false . • If multi_step = true , then reasoning must be true . F.5 Category Classification (Choose One) Annotators assign category_type to exactly one of the following category names (exact strings): • Islamic Creed • Jurisprudence • Inheritance Law • Hadith Studies • Qur’anic Studies • Prophetic Biography • Islamic History • Islamic Ethics and Morality • Islamic Finance and Economics • Islamic Family Law • Comparative Religion • Contemporary Issues F.5.1 Boundary Rules • Modern banking/finance products \rightarrow Islamic Finance and Economics . • Marriage/divorce/custody \rightarrow Islamic Family Law ; inheritance shares/heirs \rightarrow Inheritance Law . • Hadith authentication/narrators/classification \rightarrow Hadith Studies ; hadith used mainly to derive a ruling \rightarrow usually Jurisprudence . • Sīrah \rightarrow Prophetic Biography ; later eras/dynasties \rightarrow Islamic History . • Novel modern scenario \rightarrow Contemporary Issues ; timeless moral teaching \rightarrow Islamic Ethics and Morality . F.6 Ambiguity and Missing Context Some questions may be underspecified or admit multiple valid scholarly answers. In such cases, annotators: • still assign the best category based on the main domain being tested, • rate difficulty based on what is required to answer responsibly (often higher if many qualifications are needed), • flag the issue briefly in the interface notes field (if available), and • do not add extra keys to the JSON output. F.7 Worked Examples The examples below illustrate how to apply the labels (they are not taken from the dataset). Question Label What is tawhīd ? Difficulty: 1 Reasoning: false Multi_step: false Category_type: Islamic Creed Explain the difference between wājib and sunnah acts. Difficulty: 2 Reasoning: true Multi_step: false Category_type: Jurisprudence A person touched their spouse and then prayed. Does this invalidate wudū’? Explain. Difficulty: 3 Reasoning: true Multi_step: true Category_type: Jurisprudence Compute inheritance shares when the deceased leaves a wife, two daughters, and parents. Difficulty: 4 Reasoning: true Multi_step: true Category_type: Inheritance Law Classify a hadith given narrator reliability and continuity of isnād. Difficulty: 4 Reasoning: true Multi_step: true Category_type: Hadith Studies Evaluate a modern bioethical dilemma by balancing harms/benefits and competing obligations. Difficulty: 5 Reasoning: true Multi_step: true Category_type: Contemporary Issues Table 6: Worked examples illustrating the annotation guideline. F.8 Final Checklist • Output is JSON only (no extra text). • difficulty is an integer in \{1,2,3,4,5\} . • Consistency: reasoning=false \Rightarrow multi_step=false ; multi_step=true \Rightarrow reasoning=true . • category_type matches exactly one of the 12 category strings. Figure 13: Annotation Interface F.9 Annotation Results and Agreement We report summary statistics for the human annotation stage and quantify annotator consistency on a held-out subset with redundant labeling. In total, we collected 3810 annotation records. Difficulty labels are distributed across the 1–5 scale, with the largest mass at level 3 (26.90%), followed by level 2 (23.03%), level 4 (20.03%), level 1 (17.67%), and level 5 (12.37%). For category assignment, we consolidate heterogeneous source-specific tags into 12 unified category_type labels, with the largest classes being Inheritance Law , Islamic Finance and Economics , Hadith Studies , Qur’anic Studies and Islamic History . To assess reliability, we additionally annotate a subset of 315 items with three independent annotators per item. On this subset, we observe an overall agreement rate of 82.96% and a Cohen’s \kappa of 0.62, indicating substantial agreement and supporting the consistency of the guidelines. Appendix G Results with Correct Incorrect and Not Attempted Model Arabic English Average Correct Incorrect Not Attempted Correct Incorrect Not Attempted Fanar-2-27B 48.20 21.50 30.30 47.90 30.20 21.90 48.05 ALLaM-7B 42.70 52.90 4.40 32.80 63.70 3.50 37.75 Fanar-1-9B 34.50 54.10 11.40 36.30 55.10 8.60 35.40 AceGPT-v2-8B 23.10 64.30 12.60 28.80 57.20 14.00 25.95 EuroLLM-9B 22.30 67.20 10.50 29.10 64.50 6.40 25.70 SILMA-9B-v1.0 20.40 70.90 8.70 28.50 66.10 5.40 24.45 Qwen3-4B-2507 15.80 45.30 38.90 27.90 45.20 26.90 21.85 gpt-oss-20b 15.90 22.60 61.50 27.20 27.20 45.60 21.55 Llama-3.1-8B 13.00 74.00 13.00 25.80 47.40 26.80 19.40 Mistral-7B-v0.2 13.50 53.50 33.00 24.40 59.10 16.50 18.95 SeaLLM-7B-v2.5 11.60 76.30 12.10 23.80 64.80 11.40 17.70 Qwen2.5-3B 11.00 61.20 27.80 20.00 63.20 16.80 15.50 Qwen3-14B 16.00 12.50 71.50 14.00 4.20 81.80 15.00 Llama-2-7b 4.40 47.20 48.40 18.80 72.00 9.20 11.60 DeepSeek-R1-0528-Qwen3-8B 6.30 17.70 76.00 11.90 13.30 74.80 9.10 Qwen3-8B 8.80 10.00 81.20 8.50 5.60 85.90 8.65 Qwen3-4B-Thinking-2507 6.50 3.10 90.40 9.40 14.20 76.40 7.95 Qwen3-4B 6.50 17.30 76.20 9.00 9.60 81.40 7.75 Qwen3-1.7B 3.20 18.10 78.70 5.10 12.60 82.30 4.15 Qwen3-0.6B 1.30 47.00 51.70 5.40 39.70 54.90 3.35 DeepSeek-R1-Distill-Qwen-7B 1.40 37.40 61.20 4.30 37.60 58.10 2.85 DeepSeek-R1-Distill-Qwen-1.5B 0.10 21.60 78.30 1.00 41.10 57.90 0.55 Table 7: Results including Correct/Incorrect and not attempted.