CulturalFrames : Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics

Shravan Nayak Mehar Bhatia Xiaofeng Zhang, [0.5em] Verena Rieser Lisa Anne Hendricks Sjoerd van Steenkiste, [0.5em] Yash Goyal Karolina Stańczak Aishwarya Agrawal, [0.5em] Mila – Quebec AI Institute, Université de Montréal, McGill University, Google Research, Google DeepMind, Samsung - SAIT AI Lab, · 2025

Read CulturalFrames : Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics by Shravan Nayak Mehar Bhatia Xiaofeng Zhang, [0.5em] Verena Rieser Lisa Anne Hendricks Sjoerd van Steenkiste, [0.5em] Yash Goyal Karolina Stańczak Aishwarya Agrawal, [0.5em] Mila – Quebec AI Institute, Université de Montréal, McGill University, Google Research, Google DeepMind, Samsung - SAIT AI Lab, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08835.txt.

Introduction

Shravan Nayak and co-authors introduce CulturalFrames in 2025, a benchmark for testing whether text-to-image models respect cultural expectations. The study covers 10 countries and five social and cultural domains. It asks whether models meet both cues stated plainly in a prompt and cues that are only implied by the cultural setting.

The benchmark holds 983 prompts, 3,637 images from four leading models, and more than ten thousand human ratings. Raters found that models missed cultural expectations about 44 percent of the time, and stated cues were missed even more often. Standard scoring tools also matched human judgment poorly. The work gives builders a concrete test and clear directions for fairer image models.

Catalog record

CREATORShravan Nayak Mehar Bhatia Xiaofeng Zhang, [0.5em] Verena Rieser Lisa Anne Hendricks Sjoerd van Steenkiste, [0.5em] Yash Goyal Karolina Stańczak Aishwarya Agrawal, [0.5em] Mila – Quebec AI Institute, Université de Montréal, McGill University, Google Research, Google DeepMind, Samsung - SAIT AI Lab,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08835

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CulturalFrames : Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics Shravan Nayak Mehar Bhatia Xiaofeng Zhang, [0.5em] Verena Rieser Lisa Anne Hendricks Sjoerd van Steenkiste, [0.5em] Yash Goyal Karolina Stańczak Aishwarya Agrawal, [0.5em] Mila – Quebec AI Institute, Université de Montréal, McGill University, Google Research, Google DeepMind, Samsung - SAIT AI Lab, Montreal, ETH AI Center[0.3em] Abstract The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit as well as implicit cultural expectations . To this end, we introduce CulturalFrames , a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CulturalFrames comprises 983 prompts, 3,637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that T2I models not only fail to meet the more challenging implicit expectations but also the less challenging explicit expectations. Across models and countries, cultural expectations are missed an average of 44% of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68%, while implicit expectation failures are also significant, averaging 49%. Furthermore, we demonstrate that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, providing actionable directions for developing more culturally informed T2I models and evaluation methodologies. 1 Introduction Visual media such as advertisements, posters, and public imagery play a central role in encoding and transmitting cultural values ( McLuhan 1966 ) . They often depict culturally specific elements (e.g., traditional attire, religious symbols) and embed societal norms and values (e.g., expectations around family structure, gender roles, and etiquette), thus reflecting and influencing the cultures from which they originate ( Hall 1980 ) . Text-to-image (T2I) models are emerging as a significant component of this visual media ecosystem, now adopted across diverse domains like education, marketing, and storytelling ( Dehouche and Dehouche 2023 ; Loukili et al. 2025 ; Maharana et al. 2022 ) . This magnifies the cultural implications of their outputs for global audiences ( Wan et al. 2024 ; Hartmann et al. 2025 ) and raises a critical question: how accurately, and with what depth, do these models depict diverse cultures? While T2I models may generate visually plausible outputs for cultural prompts (e.g., “a bride and groom exchanging vows at their Hindu wedding,” Figure 1 ), they often capture explicit details at the expense of crucial, implicit elements integral to the cultural context (such as a sacred fire or officiating priest). Indeed, T2I model performance hinges on accurate cultural representation, which can foster familiarity and trust. Inaccuracies, however, risk reinforcing stereotypes, exclusion, or propagating dominant narratives ( Naik and Nushi 2023 ) . Figure 1: Examples from CulturalFrames benchmark for three selected countries: India, China, and Poland. We ask annotators to evaluate the generated images with respect to both explicit and implicit cultural expectations. This necessitates evaluation practices that not only verify faithfulness to the explicit expectations ( i.e. , expectations based on the words in the prompt) but also assess the inference and contextualization of implicit cultural expectations ( i.e. , expectations based on the cultural context mentioned in the prompt). However, current T2I evaluation methodologies predominantly focus on the former by assessing explicit prompt-image consistency using automated metrics ( Hu et al. 2023 ; Hessel et al. 2021 ; Ku et al. 2024a ) . 1 1 1 The only prior work evaluating appropriate contextualization of sensitive content is Akbulut et al. 2025 , which focuses on image-to-text for historical events. Further, existing benchmarks for evaluating T2I models are designed around prompts that emphasize attributes like realism ( Saharia et al. 2022 ) , compositionality ( Huang et al. 2023 ; Huang et al. 2025 ) , and safety ( Lee et al. 2023 ) , typically using generic or Western-centric prompts. Consequently, current evaluation methods and benchmarks lack adequate representation of culturally nuanced and expectation-rich scenarios critical to diverse cultural contexts. In response to these limitations, we perform a comprehensive study to evaluate how T2I models represent cultural expectations across diverse contexts. We introduce CulturalFrames , a novel benchmark comprising 983 prompts across 10 countries, with 3,637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. The curated prompts are grounded in real-life situations and cover five culturally significant domains: greetings , etiquette , dates of significance , religion , and family life , which are explicitly designed to test representation of both explicit and implicit cultural expectations . Using the collected prompts, we first generate images with four state-of-the-art T2I models, two open-source and two closed-source. Second, we conduct evaluations employing human annotators with relevant cultural backgrounds, who provide fine-grained judgments of the generated images with respect to the prompt in order to assess T2I models’ performance. We find that state-of-the-art T2I models not only struggle with depicting the more challenging implicit expectations but also the comparatively simpler explicit ones. In fact, models fail to meet cultural expectations 44% of the time on average across countries. Among these instances, the failure rate for explicit expectations is unexpectedly high, averaging 68%, while the rate for implicit expectations is also substantial at an average of 49%. Furthermore, we compare these human assessments with existing T2I evaluation metrics to demonstrate that current measures correlate poorly with human judgments of cultural alignment, while differing in their internal reasoning. Collectively, our findings lead to a discussion on actionable directions for developing more culturally informed T2I models and evaluation methodologies. These include utilizing our prompts for future evaluations, leveraging the full CulturalFrames (prompts, images, and annotations) for model alignment, and using explicit instructions for metrics. 2 Related Work Evaluating T2I models. A suite of benchmarks has been proposed for text-to-image generation. DrawBench ( Saharia et al. 2022 ) and PartiPrompts ( Yu et al. 2022 ) evaluate overall image fidelity and complex scene rendering. The T2I-CompBench series ( Huang et al. 2023 ; Huang et al. 2025 ) focus specifically on compositional challenges. Human assessment and considerations for bias and fairness are addressed by ImagenHub ( Ku et al. 2024c ) , HEIM ( Lee et al. 2023 ) , and GenAI Arena ( Jiang et al. 2024 ) . Traditional metrics assess image quality and diversity using embedding-based metrics, e.g., FID ( Heusel et al. 2018 ) , Inception Score ( Salimans et al. 2016 ) , and the text-image alignment via pretrained vision-language embeddings, e.g., CLIPScore ( Hessel et al. 2021 ) and DinoScore ( Ruiz et al. 2023 ) . More recently, reward models trained on human preferences such as HPSv2 ( Wu et al. 2023a ) , ImageReward ( Xu et al. 2023 ) , and PickScore ( Kirstain et al. 2023 ) have shown improved correlation with human judgments. Concurrently, further metrics leverage LLMs and VLMs for evaluating prompt consistency and image-text alignment through question-answering or reasoning, such as TIFA ( Hu et al.

Read the complete text