From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation
Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, Wadii Boulila · 2025
Read From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation by Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, Wadii Boulila (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01920.txt.
Introduction
Serry Sibaee and co-authors set out new guidelines for testing Arabic language models, and a harder test to go with them. Their 2025 paper reviews existing Arabic test sets and finds gaps in language accuracy, cultural fit, and method. Models can score well on such tests while still missing deep cultural knowledge.
Their new mini data set holds hundreds of hard questions across ten broad domains and many sub-domains. They ran five leading models on it, and scores varied widely by domain. The best model reached only a modest top score, with strength in math, language, and Islamic topics. The authors argue that cultural skill should be graded beside raw technical skill in Arabic AI.
Catalog record
| CREATOR | Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, Wadii Boulila |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, Wadii Boulila Abstract This paper addresses critical gaps in Arabic language model evaluation by establishing comprehensive theoretical guidelines and introducing a novel evaluation framework. We first analyze existing Arabic evaluation datasets, identifying significant issues in linguistic accuracy, cultural alignment, and methodological rigor. To address these limitations in LLMs, we present the Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490 challenging questions spanning ten major domains (42 sub-domains, see Figure 1 ). Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet, Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant variations in model performance across different domains, with particular challenges in areas requiring deep cultural understanding and specialized knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30%, showing relative strength in mathematical theory in Arabic, Arabic language, and islamic domains. This work provides both theoretical foundations and practical insights for improving Arabic language model evaluation, emphasizing the importance of cultural competence alongside technical capabilities. \setcode utf8 1 Introduction The evaluation of Arabic large language models (LLMs) presents unique challenges that extend beyond conventional metrics of linguistic accuracy. As these models become increasingly prevalent in various applications, the need for comprehensive and culturally aware evaluation frameworks has become critical. Recent developments in Arabic LLM evaluation have produced several datasets, including GPTArEval Khondaker et al. 2023 , Ghafa Almazrouei et al. 2023 , and ArabicMMLU from openAI OpenAI 2024 , each attempting to address different aspects of model assessment. However, these efforts often fail to provide a comprehensive evaluation that includes both technical proficiency and cultural understanding. Figure 1: Representation of categories and subcategories of the proposed dataset. Current evaluation approaches frequently rely on translated content Romanou et al. 2024 or simplified metrics that fail to capture the nuances of Arabic language and culture OpenAI 2024 . This limitation is particularly evident in specialized domains such as Islamic studies, classical literature, and technical fields where cultural context and domain expertise are crucial. Furthermore, existing datasets often exhibit inconsistencies in linguistic standards and cultural representation, potentially resulting in misleading assessments of model capabilities. Our work addresses these challenges through three main contributions. First, we establish theoretical guidelines for Arabic evaluation datasets that encompass linguistic standards, cultural alignment, and methodological requirements. Second, we conduct a detailed analysis of existing evaluation datasets, identifying common pitfalls and areas for improvement. Third, we introduce the Arabic Depth Mini Dataset (ADMD), a specialized evaluation tool designed to assess both technical and cultural competencies across diverse domains. The ADMD represents a significant advancement in the evaluation of Arabic LLM, featuring carefully curated questions that demand deep understanding rather than surface-level pattern matching. By evaluating leading language models using this dataset, we provide insights into current model capabilities and limitations, particularly in handling complex Arabic queries that require cultural awareness and specialized knowledge. This paper is organized as follows: Section 2 reviews related work in Arabic LLM evaluation, Section 3 presents our theoretical guidelines, Section 4 analyzes existing evaluation datasets, Section 5 introduces the ADMD and presents evaluation results, and Section 6 discusses limitations and future work directions. Dataset Reviewed Handwritten Generated Translated GPTArEval Khondaker et al. 2023 \times ✓ \times \times Ghafa Almazrouei et al. 2023 \times ✓ \times ✓ ArabicMMLU Koto et al. 2024 ✓ \times \times ✓ AraDICE Mousi et al. 2025 \times ✓ \times \times ArSTEM Mustapha et al. 2024 ✓ \times ✓ \times Aya Expanse Dang et al. 2024 \times \times ✓ ✓ AraTrust Alghamdi et al. 2024 ✓ ✓ \times \times ILMAAM Nacar et al. 2025 ✓ \times ✓ \times ArabicaQA Abdallah et al. 2024 ✓ ✓ \times \times Balsam KSAA 2024 ✓ ✓ \times \times ADMD (Ours) ✓ ✓ \times \times Table 1: Comparison of Arabic LLM Evaluation Datasets based on annotation type and content origin. 2 Related Works The evaluation landscape for Arabic large language models (LLMs) has witnessed significant advancements through several benchmark initiatives, each with distinctive methodological approaches and inherent limitations Eriksson et al. 2025 . This section critically examines these evaluation frameworks while highlighting their methodological underpinnings and empirical contributions. GPTArEval Khondaker et al. 2023 represents a pioneering effort in Arabic LLM assessment, with its integration of ORCA Elmadany et al. 2023 datasets and emphasis on natural language understanding and generation capabilities. The framework provides valuable insights into model performance but exhibits constraints in addressing the full spectrum of Arabic linguistic nuances. In parallel, Ghafa Almazrouei et al. 2023 employs a translation-based methodology supplemented by native speaker revisions, while ArabicMMLU Koto et al. 2024 attempts to span diverse knowledge domains. Despite engaging ten native Arabic speakers in their validation processes, both datasets demonstrate substantial limitations in linguistic precision and comprehensive domain representation, with assessment complexity being constrained by the educational resources utilized in their development. The cultural dimension of Arabic LLM evaluation has been addressed through specialized datasets such as AraDICE Mousi et al. 2025 , which focuses specifically on dialectal variations and cultural contextual understanding, and ArSTEM Mustapha et al. 2024 , which emphasizes scientific knowledge assessment within Arabic linguistic frameworks. These initiatives reflect an emerging recognition of the importance of culturally nuanced evaluation metrics that extend beyond mere linguistic accuracy. Institutional contributions to Arabic LLM evaluation methodologies have come from teams developing models such as Jais Sengupta et al. 2023 and Allam Bari et al. 2024 , each implementing distinctive approaches to dataset curation and evaluation protocols. However, comprehensive assessment of closed-source models such as Allam and Fanar Team et al. 2025 remains challenging due to accessibility constraints. The Aya Expanse model Dang et al. 2024 distinguishes itself through methodological transparency regarding its utilization of translated and GPT-generated materials, establishing an important precedent for disclosure in evaluation dataset construction. Critical methodological analysis by Nacar et al. 2025 has illuminated significant deficiencies in existing benchmarks, particularly in ArabicMMLU OpenAI 2024 , encompassing linguistic inconsistencies, semantic imprecisions, and fundamental methodological flaws. In response to these identified shortcomings, AraTrust Alghamdi et al. 2024 was developed as a methodologically rigorous framework specifically designed to enhance reliability assessment for Arabic LLMs (as illustrated in Table 1 ). While Abdallah et al. 2024 has contributed substantially to the field with an extensive Arabic question-answering dataset comprising over 80,000 entries authored by native speakers, its reliance on Wikipedia articles as source material raises concerns regarding authoritative credibility. In contrast, Balsam KSAA 2024 represents the highest quality dataset produced through collaboration between prominent academic and governmental institutions across the Middle East, though its utility is limited by the relatively small number of samples within each category and it is not publicly available. Within the context of ongoing efforts to establish methodologically sound and linguistically accurate Arabic evaluation frameworks, our research makes two significant contributions: (1) a comprehensive theoretical analysis and empirical assessment of three critical evaluation datasets—Ghafa, ArabicMMLU, and INCLUDE—examining their methodological approaches, linguistic accuracy, and domain coverage; and (2) the introduction of the Arabic Depth Mini Dataset (ADMD), conceived as a foundational resource to address the current limitations in evaluating specialized domain knowledge within Arabic language contexts. The ADMD serves as an initial