Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset
Rawan Bondok, 1 Mayar Nassar, 1, 2] Salam Khalifa, 1, 3] Kurt Micallef, 1, 4] Nizar Habash 1 · 2025
Read Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset by Rawan Bondok, 1 Mayar Nassar, 1, 2] Salam Khalifa, 1, 3] Kurt Micallef, 1, 4] Nizar Habash 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02656.txt.
Introduction
Rawan Bondok and four co-authors release a hand-checked data set for restoring the vowel marks on Arabic proper names in Wikipedia. Their paper appeared in two thousand twenty-five. Arabic script often omits those marks, leaving names ambiguous to pronounce and interpret. Foreign names carried into Arabic are the hardest cases.
Each name in the set is fully marked by hand and paired with its English Wikipedia gloss. The authors describe the challenges and rules they followed to build it. They then test a leading chat model on recovering the marks from the bare Arabic plus the English form. The model reached a fair but far from perfect score. That gap shows the task's difficulty and the need for better resources. The set is released for other researchers.
Catalog record
| CREATOR | Rawan Bondok, 1 Mayar Nassar, 1, 2] Salam Khalifa, 1, 3] Kurt Micallef, 1, 4] Nizar Habash 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02656
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset Rawan Bondok, 1 Mayar Nassar, 1, 2] Salam Khalifa, 1, 3] Kurt Micallef, 1, 4] Nizar Habash 1 Abstract Proper nouns in Arabic Wikipedia are frequently undiacritized, creating ambiguity in pronunciation and interpretation, especially for transliterated named entities of foreign origin. While transliteration and diacritization have been well-studied separately in Arabic NLP, their intersection remains underexplored. In this paper, we introduce a new manually diacritized dataset of Arabic proper nouns of various origins with their English Wikipedia equivalent glosses, and present the challenges and guidelines we followed to create it. We benchmark GPT-4o on the task of recovering full diacritization given the undiacritized Arabic and English forms, and analyze its performance. Achieving 73% accuracy, our results underscore both the difficulty of the task and the need for improved models and resources. We release our dataset to facilitate further research on Arabic Wikipedia proper noun diacritization. 1 1 1 https://github.com/CAMeL-Lab/CamelProp \noautomath 1 Introduction Arabic Wikipedia, like other language editions, has been a valuable resource for both its readers and NLP research. In this paper, we focus on a particular limitation rooted in Arabic’s abjad orthography, where diacritics are typically omitted Elgamal et al. (2024) except for children’s books and religious texts. This omission leads to ambiguity in pronunciation and interpretation, especially for proper nouns. Some Arabic Wikipedia articles address this issue by providing partial or full diacritization in their lead sentences. For instance, \set@arabfont نامع \arab@strut \varsigma mAn 2 2 2 Arabic HSB Romanization Habash et al. (2007) . can refer to either \set@arabfont ناَمُع \arab@strut \varsigma umaAn ‘Oman’ or \set@arabfont ناَّمَع \arab@strut \varsigma am \sim aAn ‘Amman’ depending on the diacritization (Figure 1 ). But more often than not, these diacritics are missing. In our dataset we found 99.45% of all entries had no diacritics. Our intention is to solve this limitation. Figure 1 : Four Arabic Wikipedia entries: (a) \set@arabfont نامع \arab@strut \varsigma mAn ‘Oman’, (b) \set@arabfont نامع \arab@strut \varsigma mAn ‘Amman’, (c) \set@arabfont ندنل \arab@strut lndn ‘London’, and (d) \set@arabfont ناوجخن \arab@strut nxjwAn ‘Nakhchivan’. All titles lack diacritics. Lead sentences do not consistently use diacritics: (a) \varsigma u mAn, (b) \varsigma a m \sim a An, and (c) l a nd a n; but (d) lacks diacritics, allowing multiple readings. The work presented in this paper lies at the intersection of three commonly but often independently studied Arabic NLP tasks: transliteration, diacritization, and lemmatization. Transliteration is the mapping of words, primarily proper nouns, from one script to another, usually in the context of machine translation Beesley (1997) ; Benites et al. (2020) ; Chen et al. (2018) . It poses challenges due to misalignments between scripts, differences in representing phonology and morphology, and historical ad hoc conventions. Diacritization , or diacritic restoration, aims at recovering omitted diacritics in languages that rely on them for disambiguation Alqahtani et al. (2019) ; Darwish et al. (2017) ; Abandah et al. (2015) . While both transliteration and diacritization have been well studied for Arabic, they are typically treated in isolation. An exception is the work of Mubarak et al. (2009) , which considers both in the context of Arabic to English proper noun transliteration. Lemmatization maps inflected words to their base forms. This is particularly important for morphologically rich languages such as Arabic Roth et al. (2008) . In the context of Wikipedia entries, providing the lemmas is useful to readers as it gives them a grounding on how to interpret and later inflect the word forms properly. More concretely, we focus here on mapping pairs of undiacritized Arabic proper nouns and their English glosses to fully diacritized and lemmatized Arabic forms. The task can be viewed as partial transliteration, where Roman-script vowels help infer (or transliterate into) Arabic diacritical marks. For example, \set@arabfont ناوجخن \arab@strut nxjwAn ‘Nakhchivan’ (from Figure 1 ) should ideally be mapped to \set@arabfont ناَوِجْخَن \arab@strut nax.jiwaAn, rather than incorrect alternatives like \set@arabfont ناَوَجْخِن \arab@strut nix.jawaAn or \set@arabfont ناَوِجْخُن \arab@strut nux.jiwaAn. We present a new dataset of 3,000 unique Arabic Wikipedia proper nouns annotated with gold lemma-level diacritizations. Each entry is paired with its English Wikipedia equivalent, enabling the study of joint diacritization and transliteration. We benchmark GPT-4o OpenAI et al. (2024) , which shows promising results but struggles with spelling variants and ambiguity. The dataset covers a range of named entities (people, places, and organizations) and includes 3,362 total pairs to reflect multiple valid diacritizations based on the gloss. Our contributions are: • A publicly available gold-standard dataset of Arabic Wikipedia proper nouns with English equivalents. 1 • A GPT-4o benchmark and detailed error analysis for Arabic proper noun diacritization. The remainder of this paper is structured as follows. Section 2 outlines Arabic linguistic aspects. Section 3 reviews related work. Sections 4 and 5 describe our dataset and annotation process. Section 6 presents evaluation results and error analysis. 2 Linguistic Background Diacritic Example Fatha \set@arabfont َب \arab@strut ba /ba/ Damma \set@arabfont ُب \arab@strut bu /bu/ Kasra \set@arabfont ِب \arab@strut bi /bi/ Shadda \set@arabfont ّب \arab@strut b \sim /bb/ Sukun \set@arabfont ْب \arab@strut b. /b/ Dagger Alif \set@arabfont ٰب \arab@strut bá /ba:/ Shadda + Fatha \set@arabfont َّب \arab@strut b \sim a /bba/ Shadda + Damma \set@arabfont ُّب \arab@strut b \sim u /bbu/ Shadda + Kasra \set@arabfont ِّب \arab@strut b \sim i /bbi/ Long vowel /a/ \set@arabfont اَب \arab@strut baA /ba:/ Long vowel /u/ \set@arabfont وُب \arab@strut buw /bu:/ Long vowel /i/ \set@arabfont يِب \arab@strut biy /bi:/ Shadda + Long vowel /a/ \set@arabfont اَّب \arab@strut bbaA /bba:/ Shadda + Long vowel /u/ \set@arabfont وُّب \arab@strut bbuw /bbu:/ Shadda + Long vowel /i/ \set@arabfont يِّب \arab@strut bbiy /bbi:/ Glide w \set@arabfont ْوَب \arab@strut baw. /baw/ Glide y \set@arabfont ْيَب \arab@strut bay. /bay/ Table 1 : Examples of Arabic diacritics, their transliterations, and phonological values. We exclude nunation diacritics as they are not used in our lemmas. Input Arabic Gloss Lemma Arabic Transformation \set@arabfont تسلا \arab@strut Alst Al-Sit \set@arabfont ّتِس \arab@strut sit \sim DET → \phi \set@arabfont ساولا \arab@strut AlwAs Elvas \set@arabfont ساَوْلِإ \arab@strut Ǎil.waAws Bare Alif → Alif Hamza \set@arabfont مجعلا \arab@strut Al \varsigma jm Al-Ajam \set@arabfont مَجَع \arab@strut \varsigma ajam DET → \phi \set@arabfont ةاظغلا \arab@strut Al \gamma ĎA \hbar Al-Ghadhah \set@arabfont ةاَظَغ \arab@strut \gamma aĎaA \hbar DET → \phi \set@arabfont نويليوزنف \arab@strut fnzwylywn Venezuelans \set@arabfont ّيِليِوْزِنِف \arab@strut finiz.wiyliy \sim 3MP → \phi \set@arabfont نويتوبيجلا \arab@strut Aljybwtywn Djiboutians \set@arabfont ّيِتوُبيِج \arab@strut jiybuwtiy \sim DET+3MP → \phi Table 2 : Examples of lemmatization transformations from Arabic input (inflected) words to canonical lemmas, with English glosses and corresponding changes. 2.1 Arabic Diacritization Arabic orthography follows an Abjad system Daniels (2013) , where letters encode consonants and diacritical marks represent short vowels, nunation (case endings), gemination, and vowel absence. Diacritic clusters are typically limited to a Shadda ( \set@arabfont › \sim ) followed by a short vowel or nunation diacritic. Three letters, \set@arabfont ا \arab@strut A, \set@arabfont و \arab@strut w, and \set@arabfont ي \arab@strut y (henceforth AWY), encode long vowels when preceded by a matching short vowel and not followed by any diacritic: \set@arabfont ا \arab@strut \set@arabfont ¸ aA (/a:/), \set@arabfont و \arab@strut \set@arabfont ˛ uw (/u:/), and \set@arabfont ي \arab@strut \set@arabfont ¸ iy (/i:/). These letters are often used with foreign