A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment
Khalid N. Elmadani, Nizar Habash, Hanada Taha-Thomure · 2025
Read A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment by Khalid N. Elmadani, Nizar Habash, Hanada Taha-Thomure (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13520.txt.
Introduction
Khalid Elmadani, Nizar Habash, and Hanada Taha-Thomure present a large new corpus for grading how hard Arabic texts are to read, in a 2025 paper. The corpus holds 69,441 sentences and more than a million words. Each sentence is labeled by hand on a fine scale of 19 levels, from kindergarten to graduate study.
The set was built to balance genre, topic, and audience, and a large team of annotators labeled every sentence. Agreement between annotators was high, which supports the labels' quality. The authors also test automatic grading methods at several levels of detail. They release the corpus, its labeling guide, and the test results openly for research and teaching use.
Catalog record
| CREATOR | Khalid N. Elmadani, Nizar Habash, Hanada Taha-Thomure |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13520
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment Khalid N. Elmadani, Nizar Habash, Hanada Taha-Thomure Abstract This paper introduces the Balanced Arabic Readability Evaluation Corpus ( Barec ), 1 1 1 قراب bAriq is Arabic for ‘very bright and glittering’. a large-scale, fine-grained dataset for Arabic readability assessment. Barec consists of 69,441 sentences spanning 1+ million words, carefully curated to cover 19 readability levels, from kindergarten to postgraduate comprehension. The corpus balances genre diversity, topical coverage, and target audiences, offering a comprehensive resource for evaluating Arabic text complexity. The corpus was fully manually annotated by a large team of annotators. The average pairwise inter-annotator agreement, measured by Quadratic Weighted Kappa, is 81.8%, reflecting a high level of substantial agreement. Beyond presenting the corpus, we benchmark automatic readability assessment across different granularity levels, comparing a range of techniques. Our results highlight the challenges and opportunities in Arabic readability modeling, demonstrating competitive performance across various methods. To support research and education, we make Barec openly available, along with detailed annotation guidelines and benchmark results. 2 2 2 http://barec.camel-lab.com 1 Introduction Text readability impacts understanding, retention, reading speed, and engagement DuBay (2004) . Texts above a student’s readability level can lead to disengagement Klare (1963) . Nassiri et al. (2023) highlighted that readability and legibility depend on both external features (e.g., production, fonts) and content. Text leveling in classrooms helps match books to students’ reading levels, promoting independent reading and comprehension Allington et al. (2015) . Developing readability models is crucial for improving literacy, language learning, and academic performance. Readability levels have long been a key component of literacy teaching and learning. One of the most widely used systems in English literacy is Fountas and Pinnell Fountas and Pinnell (2006) , which employs qualitative measures to classify texts into 27 levels (A to Z+), spanning from kindergarten to adult proficiency. Similarly, Taha-Thomure (2017) ’s system for Arabic has 19 levels from Arabic letters \set@arabfont أ \arab@strut A to \set@arabfont ق \arab@strut Q. These fine-grained levels are designed for pedagogical effectiveness, ensuring young readers experience gradual, measurable progress, particularly in early education (K–6) Barber and Klauda (2020) . A key advantage is that they can be easily mapped to coarser levels with fewer categories, which may be more efficient for broader applications in readability research and automated assessments. In this paper we present the Balanced Arabic Readability Evaluation Corpus ( Barec ) – a large-scale fine-grained readability assessment corpus across a broad space of genres and readability levels. Inspired by the Taha/Arabi21 readability reference Taha-Thomure (2017) , which has been instrumental in tagging over 9,000 children’s books, Barec seeks to establish a standardized framework for evaluating sentence-level 3 3 3 We use sentence to refer to any standalone text segment, including phrases and single words (e.g., book titles). Arabic text readability across 19 distinct levels, ranging from kindergarten to postgraduate comprehension. Our contributions are: (a) a large-scale curated corpus with 69K+ sentences (1M+ words) spanning diverse genres; and (b) benchmarking of automatic readability assessment models across multiple granularities, including both fine-grained (19 levels) and collapsed tiered systems (e.g., five-level and three-level scales) to support various research and application needs, aligning with previous Arabic readability frameworks Al Khalil et al. (2018) ; Al-Khalifa and Al-Ajlan (2010) . 2 Related Work Automatic Readability Assessment Automatic readability assessment has been widely studied, resulting in numerous datasets and resources Collins-Thompson and Callan (2004) ; Pitler and Nenkova (2008) ; Feng et al. (2010) ; Vajjala and Meurers (2012) ; Xu et al. (2015) ; Xia et al. (2016) ; Nadeem and Ostendorf (2018) ; Vajjala and Lučić (2018) ; Deutsch et al. (2020) ; Lee et al. (2021) . Early English datasets were often derived from textbooks, as their graded content naturally aligns with readability assessment Vajjala (2022) . However, copyright restrictions and limited digitization have driven researchers to crowdsource readability annotations from online sources Vajjala and Meurers (2012) ; Vajjala and Lučić (2018) or leverage CEFR-based L2 assessment exams Xia et al. (2016) . Arabic Readability Efforts Arabic readability research has focused on text leveling and assessment across various frameworks. Taha-Thomure (2017) proposed a 19-level system for children’s books based on qualitative and quantitative criteria. Other efforts applied CEFR leveling to Arabic, including the KELLY project’s frequency-based word lists, manually annotated corpora Habash and Palfreyman (2022) ; Naous et al. (2024) , and vocabulary profiling Soliman and Familiar (2024) . El-Haj et al. (2024) introduced DARES, a readability assessment dataset collected from Saudi school materials. The SAMER project Al Khalil et al. (2020) developed a lexicon with a five-level readability scale, leading to the first manually annotated Arabic parallel corpus for text simplification Alhafni et al. (2024) . Automated readability assessment has also been explored through rule-based and machine learning approaches. Early models relied on surface-level features like word and sentence length Al-Dawsari (2004) ; Al-Khalifa and Al-Ajlan (2010) , while later work incorporated POS-based and morphological features Forsyth (2014) ; Saddiki et al. (2018) . The OSMAN metric El-Haj and Rayson (2016) leveraged script markers and diacritization, and recent efforts Liberato et al. (2024) achieved strong results using pretrained models on the SAMER corpus. Building on these efforts, we curated the Barec corpus across genres and readability levels, and manually annotated it at the sentence-level based on an adaptation of Taha/Arabi21 guidelines Taha-Thomure (2017) , offering finer-grained control and a more objective assessment of textual variation. Figure 1 : The Barec Pyramid illustrates the relationship across Barec levels and linguistic dimensions, three collapsed variants, and education grades. Table 1 : Representative subset of examples of the 19 Barec readability levels, with English translations, and readability level reasoning. Underlining is used to highlight the main keys that determined the level. 3 Barec Corpus Annotation In this section, we summarize the guidelines and annotation process. For more details, see Habash et al. (2025) . In the next section, we discuss corpus selection and statistics. 3.1 Barec Guidelines We present below a summarized version of the Barec annotation guidelines. A detailed account of the adaptation process from Taha-Thomure (2017) ’s guidelines is in Habash et al. (2025) . Readability Levels The readability level system of Taha-Thomure (2017) uses the Abjad order of Arabic letters for 19 levels: 1-alif , 2-ba , 3-jim , through to 19-qaf . This system emphasizes a finer distinction in the lower levels, where readability is more varied. The Barec pyramid (Figure 1 ) illustrates the scaffolding of these levels and their mapping to, guidelines components, school grades, and three collapsed versions of level size 7, 5, and 3. All four level types (19-7-5-3) are fully aligned to easy mapping from fine-grained to coarse-grained levels. We present results for these levels in Section 6 . Readability Annotation Principles The guidelines focus on readability and comprehension, considering the ease of reading and understanding for independent readers. The evaluation does not depend on grammatical analysis or rhetorical depth but rather on understanding basic, literal meanings. Larger texts may contain sentences at different readability levels, but we focus on sentence-level evaluation, ignoring context and author intent. Textual Features Levels are assessed in six key dimensions. Each of these specify numerous linguistic phenomena that are needed to