DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding

Shubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan, Mohamed bin Zayed University of AI, NUCES, NUST, Australian National University{shubham.patle, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github.io/DuwatBench/ · 2026

Read DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding by Shubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan, Mohamed bin Zayed University of AI, NUCES, NUST, Australian National University{shubham.patle, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github.io/DuwatBench/ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19898.txt.

Introduction

Shubham Patle and colleagues build a benchmark for Arabic calligraphy understanding. Arabic calligraphy blends language with art; letters stretch and curve into decoration. Multimodal models have advanced across many languages and scripts. Their grasp of Arabic script, especially in stylised calligraphic forms, remains largely untested.

Their benchmark, DuwatBench, bridges language and visual heritage. It tests whether models can read and interpret calligraphic works. Such art carries religious and cultural weight across the Arab world. A model that fails on calligraphy fails on real cultural material. The benchmark gives researchers a way to measure and close that gap. The gap is plain to see.

Catalog record

CREATORShubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan, Mohamed bin Zayed University of AI, NUCES, NUST, Australian National University{shubham.patle, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github.io/DuwatBench/
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding Shubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan, Mohamed bin Zayed University of AI, NUCES, NUST, Australian National University{shubham.patle, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github.io/DuwatBench/ Abstract Arabic calligraphy represents one of the richest visual traditions of the Arabic language, blending linguistic meaning with artistic form. Although multimodal models have advanced across languages, their ability to process Arabic script, especially in artistic and stylized calligraphic forms, remains largely unexplored. To address this gap, we present DuwatBench, a benchmark of 1,272 curated samples containing about 1,475 unique words across 6 classical and modern calligraphic styles, each paired with sentence-level detection annotations. The dataset reflects real-world challenges in Arabic writing, such as complex stroke patterns, dense ligatures, and stylistic variations that often challenge standard text recognition systems. Using DuwatBench, we evaluated 13 leading Arabic and multilingual multimodal models and showed that while they perform well in clean text, they struggle with calligraphic variation, artistic distortions, and precise visual–text alignment. By publicly releasing DuwatBench and its annotations, we aim to advance culturally grounded multimodal research, foster fair inclusion of Arabic language and visual heritage in AI systems, and support continued progress in this area. Our dataset 1 1 1 https://github.com/mbzuai-oryx/DuwatBench and code 2 2 2 https://huggingface.co/datasets/MBZUAI/DuwatBench are publicly available. 2 2 footnotetext: Equal contribution. Figure 1: DuwatBench Taxonomy and Distribution. DuwatBench encompasses six principal Arabic calligraphy styles (Thuluth, Diwani, Kufic, Naskh, Ruq’ah, and Nasta’liq). The benchmark includes a distribution of collected samples ranging from 706 in Thuluth to 67 in Nasta’liq. Beyond style diversity, the taxonomy incorporates categories, including non-religious terms, Quranic words, devotional expressions and hadith, names of the Prophet and companions, names of Allah, and person or place names. This organization provides a structured basis for evaluating both the visual variation and semantic depth of Arabic calligraphy. Feature Allaf Adam Sal&King Kaoudja Calliar MOJ-DB HICMA DuwatBenach (Ours) Real-world images ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ Style diversity (multiple styles) ✗ ✓ ✓ ✓ ✓ ✗ ✓ ✓ Word / Sentence-level text ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ Full text transcriptions ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ Bounding boxes for detection ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ Complex artistic backgrounds ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✓ Publicly available ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ Table 1: Comparison of existing Arabic calligraphy datasets. Existing resources typically focus on isolated aspects such as limited style coverage, word- or sentence-level annotations, or constrained availability. In contrast, DuwatBench uniquely integrates multiple underrepresented dimensions like script diversity, word- and sentence-level transcriptions, bounding boxes for detection, and complex artistic backgrounds, while ensuring public accessibility. Allaf 6 , Adam 1 , Sal&King 34 , Kaoudja 25 , Calliar 10 , MOJ-DB 39 , HICMA 21 . 1 Introduction Arabic calligraphy sits at the meeting point of language and visual art. Letters are bound, stretched, and ornamented; diacritics and ligatures introduce significant variation in glyph shapes and baselines; style rules vary across schools such as Thuluth, Diwani, Naskh, and Kufic. These properties make calligraphy culturally and technically challenging for machine perception. Prior works ( 28 , 4 ) have noted the added difficulty of Arabic script due to positional letter forms, dense ligaturing, and heavy use of diacritics, especially when written artistically rather than as plain text. Despite growing interest in Arabic handwriting and manuscripts, existing datasets only partially address calligraphic use cases. Calliar 10 provides pen-trace data across multiple text levels but lacks the textures and visual complexity of real artworks. HICMA 21 includes real-world calligraphy images with sentence-level and style labels, but is limited to five scripts, focuses mainly on recognition, and lacks layout annotations as well as evaluation on large multimodal models (LMMs). Earlier works on style classification 25 or letter-level corpora 1 ; 34 remain too narrow to support comprehensive multimodal assessment. Meanwhile, Arabic-capable multimodal models are advancing rapidly. Cross-lingual CLIP variants 14 and multilingual VLMs 17 demonstrate strong retrieval and captioning capabilities across languages; Arabic-specific CLIP adaptations such as AraCLIP 3 report clear gains on Arabic image–text benchmarks; and recent Arabic OCR and VLM systems 13 ; 38 ; 20 show improved handling of diacritics and high-resolution text. However, none of these models have been evaluated on stylized Arabic calligraphy with complex artistic backgrounds and diverse style rules. A benchmark centered on calligraphy can thus reveal whether these systems can generalize to culturally grounded, visually demanding Arabic text, or still rely on surface-level cues. We introduce DuwatBench, a 1.27K-sample benchmark for Arabic calligraphy designed to evaluate LMMs under realistic conditions. The dataset encompasses over 1,475 unique words drawn from both religious and cultural domains, including Quranic verses, devotional phrases, greetings, and poetic or proverbial expressions. It features 6 major calligraphic styles: Thuluth, Diwani, Naskh, Kufic, Nasta’liq, and the more modern Ruq’ah, capturing the visual diversity of Arabic writing across historical and contemporary contexts (see Figure 1 ). Each image is paired with transcriptions and detection-level annotations, enabling both recognition and localization analysis. Unlike existing datasets that primarily target OCR-like tasks (Table 1 ), DuwatBench integrates real-world calligraphy with complex backgrounds, multiple scripts, full transcriptions, and bounding box annotations. Arabic calligraphy lies at the edge of visual–textual alignment: while meaning is preserved, artistic strokes, ligatures, and curved layouts often distort the patterns leveraged by models trained on clean Latin text. Evaluating multimodal models on DuwatBench therefore provides a rigorous test of whether Arabic or multilingual pretraining transfers to artistic calligraphy, distinguishing genuine understanding from pattern memorization. Our contributions are threefold: (1) we release a curated benchmark of 1.27K samples spanning 6 calligraphic styles and 1,475 unique words, enriched with transcriptions and artistic contexts; (2) we introduce detection-level annotations for word localization, complementing prior classification- and OCR-based datasets; and (3) we establish systematic baselines on state-of-the-art Arabic and multilingual LMMs, revealing consistent failure cases such as style misinterpretation, sensitivity to diacritics and hamza, curved text alignment, and interference from background clutter. DuwatBench bridges Arabic visual heritage and modern AI, enabling inclusive evaluation, style-aware modeling, and applications in preservation, education, and cultural search. Figure 2: Examples of DuwatBench Calligraphic Styles. Representative samples from the six calligraphic styles in DuwatBench: Thuluth, Diwani, Kufic, Ruq’ah, Naskh, and Nasta’liq. Each entry displays the artwork with bounding box annotations, transcription, and metadata such as style, text content, theme, and word count. These examples highlight the dataset’s diversity in structure, composition, and artistic context, spanning both religious and non-religious inscriptions. 2 DuwatBench: The Arabic Calligraphy Dataset 2.1 Dataset Taxonomy Our dataset corpus is structured along two main dimensions: textual categories and calligraphic styles (see Figure 1 ). On the content side, the largest portion consists of religious material, including Quranic verses, devotional invocations, and the Names of Allah, complemented by references to the Prophets and companions, as well as smaller portions of hadith and personal dedications. Non-religious expressions are also well represented, such as greetings, motifs, and cultural quotes common in decorative and public settings (see Figure 6 ). This balance ensures fair representation of both spiritual and cultural expressions in the dataset. In terms of visual representation, the dataset covers 6 major

Read the complete text