MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

Ghazal Kalhor, Behnam Bahrak · 2026

Read MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs by Ghazal Kalhor, Behnam Bahrak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22050.txt.

Introduction

Ghazal Kalhor and Behnam Bahrak test whether language models understand Persian proverbs. Models now sit inside daily life, so they must handle conversational language well, including figurative speech. Most figurative language research covers high-resource languages, leaving low-resource ones unmeasured.

Their 2026 preprint introduces MasalBench, a benchmark for contextual and cross-cultural understanding of Persian proverbs. A proverb's meaning rarely sits in its words alone; it depends on culture and situation. Testing models on such material reveals whether their multilingual skill is deep or only grammatical. Benchmarks like this push developers to serve smaller language communities with the same care given to English.

Catalog record

CREATORGhazal Kalhor, Behnam Bahrak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs Ghazal Kalhor, Behnam Bahrak Abstract In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs’ understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs’ contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at \fontspec_if_language:nT ENG \addfontfeature Language=Latexhttps://github.com/kalhorghazal/MasalBench . \fontspec_if_language:nT ENG \addfontfeature Language=Latexency to select semantically coherent interpretations, a pattern more pronounced in reasoning-focused models. In contrast, identifying the correct English equivalent of a Persian proverb proves more challenging, and although instruction tuning provides some benefit, its impact is limited due to the need for cultural knowledge and analogical reasoning. \fontspec_if_language:nT ENG \addfontfeature Language=Latex4 Benchmark Construction To construct our benchmark, we collect Persian proverbs and their meanings from Foote Koozegari: Persian Proverbs and Their Stories ( rahmandoost2011foote ) , a comprehensive reference that documents the etymologies, narratives, and interpretations of over 4,000 Persian proverbs. For this task, we use Gemini 2.5 Pro ( comanici2025gemini ) with the PDF versions of the book (both volumes), as the texts are provided in image format rather than as digital text. \fontspec_if_language:nTENG\addfontfeatureLanguage=Latex3 \fontspec_if_language:nTENG\addfontfeatureLanguage=Latex3 \fontspec_if_language:nT ENG \addfontfeature Language=Latex3 Whenever we use Gemini 2.5 Pro for automated content generation, we set the temperature to 1 and top-p to 0.95 to ensure a balance of creativity and coherence. We choose Gemini because it has strong performance on Persian text recognition and understanding. \fontspec_if_language:nTENG\addfontfeatureLanguage=Latex4 \fontspec_if_language:nTENG\addfontfeatureLanguage=Latex4 \fontspec_if_language:nT ENG \addfontfeature Language=Latex4 \fontspec_if_language:nT ENG \addfontfeature Language=Latexhttps://mcinext-mizan-llm-leaderboard.hf.space/ We then manually review the extracted content to correct potential typos and select 1,000 proverbs that are widely used and well known in Iranian culture, based on the authors’ cultural familiarity and their frequency of occurrence in everyday language, literature, and media. All manual reviews and verifications are carried out by the authors, who are native Persian speakers familiar with the language and its cultural context. The overall pipeline for constructing the benchmark is illustrated in Figure . \fontspec_if_language:nT ENG \addfontfeature Language=Latex4.1 Contextual Understanding To prepare the dataset for this task, we follow two steps. First, for 100 of our proverbs, we manually design short, natural, and conversational dialogues in two parts between two people. In each dialogue, the second person uses the given proverb in response to the first person. We also provide an explanation of the second person’s purpose for using the proverb. We then automate this step for the remaining proverbs using Gemini 2.5 Pro, by providing the proverbs and their meanings. Second, for 40 dialogues, we manually design three distractor interpretations of the proverb: (1) a literal interpretation, (2) a logical but incorrect interpretation, and (3) an unrelated interpretation. We then automate this step for the remaining dialogues using Gemini, by providing the dialogue, the question (asking for the second person’s purpose for using the proverb), and the correct purpose. All automatically generated dialogues and distractor options are manually verified by the authors for correctness and naturalness. These steps result in a dataset of 1,000 multiple-choice questions for assessing LLMs’ understanding of proverbs in context. \fontspec_if_language:nT ENG \addfontfeature Language=Latex4.2 Cross-Cultural Understanding To construct the dataset for this task, we provide Gemini 2.5 Pro with each Persian proverb and its meaning, and instruct it to generate two well-known and recognized English proverbs: (1) an equivalent proverb and (2) an irrelevant proverb. The equivalent proverb is most similar in meaning and usage to the given Persian proverb, while the irrelevant proverb has no real connection in meaning or usage to the Persian proverb, but may appear similar in form (for example, in terms of words, tone, or imagery), and therefore serves as a convincing distractor. If no clear or culturally appropriate English equivalent exists for a given Persian proverb, we exclude that proverb from this task and instead sample additional proverbs from the source collection. We then manually verify the results to ensure that the generated proverbs are authentic English expressions and that the equivalents accurately match the Persian proverbs in meaning. Additionally, for each proverb, we add a question asking which English proverb is equivalent to it. Through this process, we compile a dataset of 700 binary-choice questions that assesses LLMs’ cross-cultural understanding of proverbs. \fontspec_if_language:nT ENG \addfontfeature Language=Latex5 Experiments \fontspec_if_language:nT ENG \addfontfeature Language=Latex5.1 Language Model for Evaluation We evaluate several well-known multilingual LLMs of varying sizes with proficiency in Persian, including Llama 4 Scout ( meta2025llama4scout ) , Llama 3.3 70B Instruct ( meta2024llama3 ) , Qwen 2.5 72B Instruct ( qwen2025technical ) , Qwen QwQ 32B ( qwen2025qwq ) , DeepSeek V3.1 ( deepseek2025v31 ) , DeepSeek R1 ( deepseek2025r1 ) , GPT-4.1 mini ( openai2025gpt41 ) , and GPT-4o mini ( openai2024gpt4omini ) , on our benchmark. \fontspec_if_language:nT ENG \addfontfeature Language=Latex5.2 Prompting Setup To design the prompts for both tasks, we randomize the order of the options to avoid positional bias. We employ zero-shot prompting, and each prompt runs independently, with the LLMs not performing any web searches to answer them. We set the temperature to 0 to ensure deterministic responses without added creativity, and top-p to 1 so that all possibilities are considered. The maximum number of tokens is limited to 5, as the LLMs are expected to return only the letter corresponding to the correct option. Appendix provides example prompts used to construct MasalBench and evaluate the performance of LLMs on this benchmark. \fontspec_if_language:nT ENG \addfontfeature Language=Latex6 Main Results \fontspec_if_language:nT ENG \addfontfeature Language=Latex6.1 Contextual Level Table presents the accuracy of different LLMs on contextual understanding. As shown, all models perform highly on this task, achieving accuracy above 0.90, which indicates that contextual understanding of Persian proverbs is a relatively easy task for these state-of-the-art LLMs and that they are already robust at metaphorical reasoning in non-English contexts. DeepSeek V3.1 turns out to be the best-performing model with an accuracy score of 0.943, which could be explained by the fact that it has the largest size (671B parameters) among the models. However, size is not the only determining factor of LLM performance on this task, as the lowest accuracy score is assigned to Qwen QwQ 32B, even though it has a larger parameter size compared to the GPT models. We also observe that instruction-tuned models perform better than their reasoning-focused counterparts within the same families, which suggests that instruction tuning plays a significant role in improving cultural contextualization. Model Accuracy Llama 4 Scout 0.906 Llama 3.3 70B Instruct 0.927 Qwen 2.5 72B Instruct 0.911 Qwen QwQ 32B 0.903 DeepSeek V3.1 0.943 DeepSeek R1 0.927 GPT-4.1 mini

Read the complete text