JEEM: Vision-Language Understanding in Four Arabic Dialects

Karima Kadaoui 1 Hanin Atwany 1 Hamdan Al-Ali 1 Abdelrahman Mohamed 1 Ali Mekky 1 Sergei Tilga 2 Natalia Fedorova 2 Ekaterina Artemova 2 Hanan Aldarmaki 1 Yova Kementchedjhieva 1 1 MBZUAI 2 Toloka AI · 2025

Read JEEM: Vision-Language Understanding in Four Arabic Dialects by Karima Kadaoui 1 Hanin Atwany 1 Hamdan Al-Ali 1 Abdelrahman Mohamed 1 Ali Mekky 1 Sergei Tilga 2 Natalia Fedorova 2 Ekaterina Artemova 2 Hanan Aldarmaki 1 Yova Kementchedjhieva 1 1 MBZUAI 2 Toloka AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21910.txt.

Introduction

Kadaoui and a large team present a test set from twenty twenty-five for image and text models in Arabic. The set covers four Arabic dialects from Jordan, the Emirates, Egypt, and Morocco. Its tasks are writing captions for images and answering questions about them. The content is rich in local culture and daily life. The team use it to judge how well models handle dialect shifts. They test five open Arabic models and one famed paid model.

The Arabic models did poorly on both image sense and dialect text. The paid model ranked best but varied by dialect and lagged on image sense. The authors say current models leave Arabic users underserved. They call for broader training data and fairer test sets.

Catalog record

CREATORKarima Kadaoui 1 Hanin Atwany 1 Hamdan Al-Ali 1 Abdelrahman Mohamed 1 Ali Mekky 1 Sergei Tilga 2 Natalia Fedorova 2 Ekaterina Artemova 2 Hanan Aldarmaki 1 Yova Kementchedjhieva 1 1 MBZUAI 2 Toloka AI
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JEEM: Vision-Language Understanding in Four Arabic Dialects Karima Kadaoui 1 Hanin Atwany 1 Hamdan Al-Ali 1 Abdelrahman Mohamed 1 Ali Mekky 1 Sergei Tilga 2 Natalia Fedorova 2 Ekaterina Artemova 2 Hanan Aldarmaki 1 Yova Kementchedjhieva 1 1 MBZUAI 2 Toloka AI Abstract We introduce JEEM , a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: J ordan, The E mirates, E gypt, and M orocco 1 1 1 Our data is available at https://huggingface.co/datasets/toloka/JEEM . . JEEM includes the tasks of image captioning and visual question answering, and features culturally rich and regionally diverse content. This dataset aims to assess the ability of VLMs to generalize across dialects and accurately interpret cultural elements in visual contexts. In an evaluation of five prominent open-source Arabic VLMs and GPT-4V, we find that the Arabic VLMs consistently underperform, struggling with both visual understanding and dialect-specific generation. While GPT-4V ranks best in this comparison, the model’s linguistic competence varies across dialects, and its visual understanding capabilities lag behind. This underscores the need for more inclusive models and the value of culturally-diverse evaluation paradigms. 1 Introduction Figure 1 : A sample from JEEM (Moroccan set). Vision-language models (VLMs) have recently achieved notable improvements in tasks such as image captioning (IC) and visual question answering (VQA), benefiting from large multimodal training datasets and parameter scaling LlamaTeam (2024) ; Beyer et al. (2024) ; OpenAI et al. (2024) . However, these models often struggle to generalize across culturally diverse and dialect-rich environments due to the over-representation of specific geographic regions De Vries et al. (2019) ; Gustafson et al. (2023) and standardized language varieties in their training datasets Pouget et al. (2024) . Similarly, existing evaluation datasets predominantly feature Western-centric images and English text Liu et al. (2021) ; Wang et al. (2024) , while their non-English counterparts are often derived from the former, either through translation or relabeling of the same images Changpinyo et al. (2023) . This results in biased evaluation, which conceals the suboptimal performance of VLMs in geographically and dialectally diverse settings Bhatia et al. (2024) . Recognizing this gap, recent work has focused on the creation of culturally diverse multilingual VQA benchmarks, incorporating images and questions from various countries and languages ( Liu et al., 2021 ; Pfeiffer et al., 2022 ; Changpinyo et al., 2023 , inter alia) . Among these, Arabic is rarely included, and when it is, it appears either in its standardized form (Modern Standard Arabic) Tang et al. (2024) or a single dialect, such as Egyptian Romero et al. (2024) . This approach overlooks the cultural and dialectal diversity found among the \sim 400 million speakers of this language. Arabic is an official language in 25 countries across North Africa and the Middle East. Despite the shared language, each country has a different history, geography, and consequently culture. These differences manifest in the objects, locations, and activities that visually characterize each region, as well as the lexical terms and implicit meanings associated with them. For example, the traditional clothing item in the Gulf, the ‘kandura’ (a long white robe worn by men) differs subtly from the ‘djellaba’ worn in Upper Egypt, each reflecting regional identity and invoking different societal norms. On a linguistic level, differences are found not only in terms of lexicon, but also in phonetics and syntax, sometimes making mutual intelligibility challenging even among native Arabic speakers. To address the challenges posed by the cultural and dialectal diversity of Arabic, we introduce JEEM , a benchmark dataset spanning one representative dialect from each dialectal region Habash (2010) : J ordanian (Leventine), E gyptian, E mirati (Khaleeji), and M oroccan (Maghrebi). JEEM comprises two core tasks: image captioning and visual question answering. These tasks enable the evaluation of VLMs in terms of their ability to recognize and appropriately reason about cultural elements, such as traditional clothing, local artifacts, and social settings, while utilizing dialectal language. We benchmark five VLMs on JEEM and measure performance in terms of standard count-based metrics, GPT4-based evaluation, and human evaluation. This comprehensive evaluation protocol allows us to reliably compare the five VLMs, identifying performance gaps in all models, including the top-ranking one, GPT-4o. We also evaluate the automatic metrics against human judgments and provide recommendations for future evaluations on JEEM and other Arabic benchmark datasets. 2 Related Work 2.1 Why Culture Matters Prior studies reported performance disparities across cultures on machine learning tasks such as object recognition De Vries et al. (2019) ; Gustafson et al. (2023) , geolocalization Pouget et al. (2024) , mutlimodal retrieval Kádár et al. (2018) ; Buettner and Kovashka (2024) and visual question-answering Romero et al. (2024) . These disparities are commonly attributed to biases in the data on which models are trained, which tends to over-represent high-income geographic regions, and in particular Western ones De Vries et al. (2019) ; Gustafson et al. (2023) ; Pouget et al. (2024) . People from different cultures use different objects, have different traditions, and occupy different physical environments, resulting in different visual experiences and associations. They also show perceptual differences with respect to specificity and importance Nisbett and Masuda (2013) . Culture determines whether a general or a more specific term will be used to refer to an object, how the importance of background objects will be ranked with respect to foreground objects, and what objects will be mentioned in a caption or omitted Nisbett and Masuda (2013) ; Buettner and Kovashka (2024) . 2.2 Vision-Language Resources in Arabic Image Captioning Early work on Arabic IC Jindal (2017) ; Mualla and Alkheir (2018) relied on the machine translation of existing datasets (primarily MSCOCO Lin et al. (2014) and Flickr8K Hodosh et al. (2013) ), sometimes including human validation ElJundi et al. (2020) or human translation for a subset of the data Al-muzaini et al. (2018) . AraCOCO Mohamed et al. (2023) features 500 images from the MS COCO test set, captioned by Arabic speakers. While the annotators often mentioned details that did not appear in the original English caption, attesting to the difference in cultural perspectives, the captioned images were not sourced locally from the Arab-speaking world. Moreover, these captions are in MSA and follow the same short simplistic format found in COCO, lacking in dialectal and cultural understanding. Visual Question-Answering While several multilingual datasets focus on culturally relevant VQA, many exclude Arabic Gao et al. (2015) ; Gupta et al. (2020) ; Pfeiffer et al. (2022) ; Changpinyo et al. (2023) . VAQA Kamel et al. (2023) relabels MS COCO images in MSA, again limiting the cultural relevance and dialectal coverage of the data. Some works in text-only QA address this issue through manual cultural alignment Alyafeai et al. (2024) or sourcing data from Arab countries directly Koto et al. (2024) , but lack the visual component. Others incorporate images but are limited to a single Arabic variety Tang et al. (2024) ; Romero et al. (2024) , or rely on synthetic questions that are not always visually grounded Alwajih et al. (2024) . 3 Dataset Construction JEEM consists of images originating from four Arabic-speaking countries covering four distinct dialectal regions: Jordan (Levantine), Emirates (Gulf), Egypt (Egyptian), and Morocco (Maghrebi).

Read the complete text