Beyond Understanding: Evaluating the Pragmatic Gap in LLMs’ Cultural Processing of Figurative Language
Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab · 2025
Read Beyond Understanding: Evaluating the Pragmatic Gap in LLMs’ Cultural Processing of Figurative Language by Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23828.txt.
Introduction
Mena Attia and four co-authors test whether large language models truly grasp culturally grounded figurative language, in Arabic and in English. Idioms and proverbs pack local knowledge into few words, making them a sharp probe of cultural reasoning. The team built tasks on Egyptian Arabic idioms, proverbs from many Arabic dialects, and English proverbs, checking understanding, practical use, and connotation separately.
Across twenty-two models, a steady hierarchy appeared. Scores on Arabic proverbs trailed English proverbs by about four percent, and Egyptian idioms trailed Arabic proverbs by about ten percent. Using expressions appropriately in context proved harder than understanding them, with accuracy dropping about fourteen percent, though example sentences recovered much of the loss. Models also wobbled on connotation, agreeing with human judges at best about eighty-six percent of the time. The authors release Kinayat, the first Egyptian idiom dataset for both skills.
Catalog record
| CREATOR | Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs’ Cultural Processing of Figurative Language Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab Abstract We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural nuance. Using figurative language as a proxy for cultural nuance and local knowledge, we design evaluation tasks for contextual understanding, pragmatic use, and connotation interpretation in Arabic and English. We evaluate 22 open- and closed-source LLMs on Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs. Our results show a consistent hierarchy: the average accuracy for Arabic proverbs is 4.29% lower than for English proverbs, and performance for Egyptian idioms is 10.28% lower than for Arabic proverbs. For the pragmatic use task, accuracy drops by 14.07% relative to understanding, though providing contextual idiomatic sentences improves accuracy by 10.66%. Models also struggle with connotative meaning, reaching at most 85.58% agreement with human annotators on idioms with 100% inter-annotator agreement. These findings demonstrate that figurative language serves as an effective diagnostic for cultural reasoning: while LLMs can often interpret figurative meaning, they face challenges in using it appropriately. To support future research, we release Kinayat , the first dataset of Egyptian Arabic idioms designed for both figurative understanding and pragmatic use evaluation. \setcode utf8 1 Introduction Proverb {arabtext} \< ما تروحش تبيع المية في حارة السقايين> Literal Translation Don’t sell water in the village of water-sellers Figurative Meaning Don’t try to outsmart the experts. Historical Context Historically, water-sellers in Egypt lived together in villages and brought water from the Nile to sell across neighborhoods. New sellers would mistakenly try to sell water in the water-sellers’ village, failing to realize the abundance of water there made it an unsuitable market. Cultural Significance Featured in the popular song “Haret El-Sa’ayeen” (1966) by Hussein Al-Sayed, sung by Sherifa Fadel, later by Mohammed Mounir with a performance on Arab Idol (2013) surpassing 132 million views on YouTube as of August 2025 MBC 2013 . Table 1: Egyptian Proverb: Water-Sellers Village Large language models (LLMs) have demonstrated remarkable progress in multilingual text understanding and generation. Yet their ability to process culturally grounded meaning —how language encodes local knowledge, social values, and pragmatic nuance—remains poorly understood. Figurative language offers a natural lens for studying this capability. Idioms and proverbs are among the most pervasive forms of figurative expression, deeply rooted in collective experience and cultural identity. They rely on shared world knowledge and pragmatic reasoning that extend beyond literal semantics. For instance, Table 1 presents an Egyptian proverb. Without contextual knowledge, the reference to the village of water-sellers offers little clue to its intended message or its appropriate use. For models to truly understand language, they must internalize this cultural substrate rather than rely solely on surface statistics or translation mappings. Most prior studies have centered on figurative comprehension—whether a model can explain an idiom or proverb—but not on pragmatic use, which requires context sensitivity, affective inference, and social appropriateness. In this work, we address these gaps by introducing a comprehensive evaluation framework that uses figurative language as a proxy for cultural nuance and knowledge. We design tasks that test models’ abilities to (1) interpret figurative expressions, (2) use them appropriately in context, and (3) infer their connotative and affective dimensions. Our evaluation covers 22 open- and closed-source models spanning multiple architectures and parameter scales. While we leverage existing resources such as Jawaher (Arabic) and MAPS (English) for proverb interpretation, our central contribution is the introduction of Kinayat , a new dataset of Egyptian Arabic idioms annotated for both figurative meaning and pragmatic use. Our work makes the following key contributions: 1. We introduce a pragmatic use task to assess the ability of LLMs to employ figurative language appropriately in context; 2. We present a unified cross-lingual evaluation suite probing figurative interpretation, contextual appropriateness, and connotative inference; 3. We present Kinayat , a novel dataset of Egyptian Arabic idioms to evaluate figurative language understanding and pragmatic use. Our results demonstrate that performance consistently degrades from English proverbs to Arabic proverbs and finally to colloquial idioms, highlighting a systemic weakness in handling culturally-specific figurative language. We find that performance in Arabic proverbs is on average 4.29% lower than in English proverbs, indicating a gap in cross-linguistic performance. Within Arabic, models performed 10.28% worse on idioms from our Kinayat dataset (Egyptian dialect) compared to Modern Standard Arabic (MSA) proverbs. 1 1 1 MSA is the standard Arabic used in formal settings, while Egyptian dialect is a spoken vernacular, a related variant of MSA, used in Egypt. In our novel pragmatic use task, we uncover a significant ‘Pragmatics Gap’ in LLMs. Our core finding shows that model accuracy drops by an average of 14 percentage points when tasked with applying an idiom in context compared to simply explaining its meaning. Namely, the average accuracy is 14.07% lower than in the corresponding understanding task, with a maximum accuracy of 85.33%, suggesting that the use of idioms appropriately in context remains difficult for current models. However, providing the idiom sentence as additional context in the multiple-choice understanding prompt improves average accuracy by 10.66%. Finally, models exhibit notable limitations in grasping connotative meaning, achieving at most 85.58% agreement with humans on samples with 100% inter-annotator agreement. 2 Related Work Dataset Languages Figurative Type MAPS Liu et al. 2024 English, Chinese, German, Russian, Bengali, Indonesian Proverbs MABL Kabra et al. 2023 Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili, Yoruba Metaphors Fig-QA Liu et al. 2022 English Metaphors FLUTE Chakrabarty et al. 2022b English Sarcasm, Simile, Metaphor, and Idioms PUB Sravanthi et al. 2024 English Implied answers, Presuppositions, Metonymy Jawaher Magdy et al. 2025 Arabic (20 dialects) Proverbs Fann or Flop Alghallabi et al. 2025 Arabic Poetry Kinayat (ours) Arabic Idioms Table 2: Comparison of existing figurative language datasets. Recent years have seen a surge of interest in evaluating and improving the ability of LLMs to interpret and generate figurative language. Several benchmarks have been proposed to assess various dimensions of figurative understanding, including metaphor, idiom, proverb, and poetry interpretation. Table 2 provides a comparative summary of these datasets. Several English-focused datasets have been developed to evaluate literal versus figurative reasoning. The Fig-QA dataset Liu et al. 2022 frames figurative language understanding as a multiple-choice question answering task. Chakrabarty et al. 2022a evaluate LLM ability to interpret idioms and similes plausibly by continuing narratives. Other efforts include FLUTE Chakrabarty et al. 2022b and follow-up work that explores metaphor interpretation using chain-of-thought prompting and psychologically informed reasoning He et al. 2022 ; Prystawski et al. 2023 ; Jang et al. 2023 , showing that while LLMs can often identify meaning at the surface level, they frequently fail to capture implicit moral or social connotations. The PUB benchmark Sravanthi et al. 2024 specifically evaluates pragmatic competence, testing the model’s ability to distinguish between literal and contextually appropriate figurative meanings in conversation. In the multilingual setting, the MAPS dataset Liu et al. 2024 evaluates figurative understanding across six languages on proverbs, and the ProverbEval dataset Azime et al. 2025 evaluates LLMs of cultural proverbs for four Ethiopian languages and English, while MABL Kabra et