Culture is Everywhere: A Call for Intentionally Cultural Evaluation

Juhyun Oh ⋄ Inha Cha † Michael Saxon ‡ Hyunseung Lim ⋄ Shaily Bhatt ∗ Alice Oh ⋄ ⋄ KAIST † Georgia Institute of Technology ‡ University of Washington ∗ Carnegie Mellon University [email protected] · 2025

Read Culture is Everywhere: A Call for Intentionally Cultural Evaluation by Juhyun Oh ⋄ Inha Cha † Michael Saxon ‡ Hyunseung Lim ⋄ Shaily Bhatt ∗ Alice Oh ⋄ ⋄ KAIST † Georgia Institute of Technology ‡ University of Washington ∗ Carnegie Mellon University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01301.txt.

Introduction

Tests of how well language models handle culture often shrink it to trivia questions with short answers. Juhyun Oh, with five co-authors, argues that this trivia-centered habit fails as models grow stronger and spread wider. Culture is not a static list of facts; it shapes even tasks that look neutral.

The authors call for intentionally cultural evaluation, a systematic check of the cultural assumptions built into every part of testing. They map what kinds of cultural questions arise, how they arise, and under what circumstances. They also stress researcher positionality, the way a test maker's own background shapes the test. Oh and the team close with directions for more inclusive, culturally aware language research beyond current benchmarks.

Catalog record

CREATORJuhyun Oh ⋄ Inha Cha † Michael Saxon ‡ Hyunseung Lim ⋄ Shaily Bhatt ∗ Alice Oh ⋄ ⋄ KAIST † Georgia Institute of Technology ‡ University of Washington ∗ Carnegie Mellon University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01301

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Culture is Everywhere: A Call for Intentionally Cultural Evaluation Juhyun Oh ⋄ Inha Cha † Michael Saxon ‡ Hyunseung Lim ⋄ Shaily Bhatt ∗ Alice Oh ⋄ ⋄ KAIST † Georgia Institute of Technology ‡ University of Washington ∗ Carnegie Mellon University [email protected] Abstract The prevailing “trivia-centered paradigm” for evaluating the cultural alignment of large language models (LLMs) is increasingly inadequate as these models become more advanced and widely deployed. Existing approaches typically reduce culture to static facts or values, testing models via multiple-choice or short-answer questions that treat culture as isolated trivia. Such methods neglect the pluralistic and interactive realities of culture, and overlook how cultural assumptions permeate even ostensibly “neutral” evaluation settings. In this position paper, we argue for intentionally cultural evaluation : an approach that systematically examines the cultural assumptions embedded in all aspects of evaluation, not just in explicitly cultural tasks. We systematically characterize the what, how, and circumstances by which culturally contingent considerations arise in evaluation, and emphasize the importance of researcher positionality for fostering inclusive, culturally aligned NLP research. Finally, we discuss implications and future directions for moving beyond current benchmarking practices, discovering important applications that we don’t yet know exist, and involving communities in evaluation design through HCI-inspired participatory methodologies. 1 Introduction Language model-based applications are growing in adoption across the world. To ensure they are adopted responsibly and effectively, an understanding of their cultural impacts and sensitivities is important. Cultural misalignments in AI can perpetuate stereotypes, marginalize underrepresented voices, and fail to address the needs of diverse user communities Blodgett et al. 2020 . In response, the NLP and ML communities have begun to focus on culturally-aligned NLP, a subfield that aims to develop and evaluate systems capable of understanding and appropriately applying cultural knowledge in context Adilazuarda et al. 2024 ; Liu et al. 2024 ; Zhou et al. 2025 . The overarching goal is to create NLP systems that can effectively respond to and operate within varied cultural settings Bhatt and Diaz 2024 . In this paper, we concentrate specifically on evaluation, as it increasingly shapes the direction of LLM development and deployment across diverse cultural contexts. A key challenge, however, is that any decision in the evaluation pipeline—no matter how technical or routine—can carry cultural assumptions or consequences. For example, the tasks selected for evaluation often reflect the developers’ cultural context, which may not align with the needs of users from different backgrounds Hershcovich et al. 2022 . Metrics assumed to be universal, such as what counts as “well-structured” writing, can vary significantly across cultures. Even expectations around interaction style and communication can differ Folk et al. 2025 ; Ge et al. 2024 , affecting how users perceive model outputs. Despite this, the community often overlooks these cultural contingencies , focusing attention only on the most obvious or explicit cultural questions (those labeled as “cultural tasks” or “multilingual settings”). As a result, most current evaluation practices reduce culture to static facts, trivia, or proxies like nationality—primarily testing models through isolated factual questions Zhou et al. 2025 or their performance on culturally-cued prompts Mukherjee et al. 2024 . While knowledge of cultural facts is important, it fails to recognize the cultural contingencies embedded in seemingly “neutral” evaluation choices. In this position paper, we argue that every evaluative choice should be examined for culturally contingent considerations , not just those in explicitly cultural domains. We argue for a shift toward intentionally cultural evaluation : a systematic approach that foregrounds cultural context throughout the evaluation process. By this, we mean making the cultural context of every evaluative decision explicit and deliberate, rather than leaving cultural influences implicit or accidental. To challenge the current focus on only the most obvious choices like explicitly cultural tasks or multilingual settings, we systematically distinguish and discuss three key aspects of evaluation: (1) what is evaluated ( section 2 ), (2) how it is evaluated ( section 3 ), and (3) in what circumstances ( section 4 ) the desideratum is defined. We also examine the critical role of researcher positionality in shaping these evaluative choices ( section 5 ). Finally, we outline the broader implications of our proposed approach for NLP research and practice ( section 6 ). Contributions. Our work offers several key contributions. We characterize the cultural contingencies in evaluation and propose building blocks for intentionally cultural evaluation. We find most evaluations reflect a narrow set of cultural assumptions, shaped by those who define the tasks and metrics. The design of “what” gets evaluated is frequently informed by dominant Anglocentric perspectives, reifying specific knowledge types and communicative norms while marginalizing others. We show that standard computational practices, such as static reference examples or aggregate metrics, are poorly equipped to assess culturally grounded variation, and argue for reimagining these methods to support more flexible, context-sensitive judgments of model quality. Crucially, we argue that culture in evaluation is not merely static content to be measured but is fundamentally tied to the circumstances of evaluation. We show how culture is both embedded in the very language of evaluation and enacted through culturally-contingent interactional patterns. As such, evaluating only static outputs misses key aspects of cultural alignment. Finally, we call for greater reflection on the positionality of those evaluating. Evaluation of cultural competence in NLP is not neutral—it is shaped by the positionality of researchers and by systemic biases embedded in the broader AI/ML ecosystem. Researchers from lower-resource or non-Anglophone contexts often face pressure to conform to English-centric benchmarks to gain visibility, placing additional burdens on their work and constraining the development of research agendas grounded in local cultural contexts. This marginalization limits the diversity of perspectives represented in NLP and reinforces existing inequities. Further, we suggest implications for moving beyond decontextualized methodologies toward more situated and culturally responsive methods, surfacing “unknown unknowns,” and co-constructing evaluation practices with affected communities. We ground our suggestions using findings from HCI studies. In doing so, we support a broader shift in NLP evaluation toward ‘thick evaluation’ Qadri et al. 2025 —an approach that prioritizes context-sensitive, community-aligned assessments of AI systems. 2 What to evaluate To move toward culturally intentional evaluation, we must ask: What tasks contain important, culturally contingent considerations? Current evaluations suffer from (a) overly narrow conceptions of ‘cultural’ tasks and (b) externally imposed definitions of relevance, thus failing to capture true cultural competence in real-world contexts. 2.1 Narrow Definitions of “Cultural Tasks” Current evaluation practices suffer from a flawed dichotomy. On one hand, explicitly “cultural” tasks are often reduced to testing factual knowledge, a “culture as trivia” approach Zhou et al. 2025 that neglects the complex interaction patterns and behavioral expectations core to cultural competence. On the other hand, widely-used benchmarks such as MMLU Hendrycks et al. 2021 and HELM Liang et al. 2023 , designed to assess foundational LLM performance, are often presented as culturally neutral. However, recent analyses reveal demonstrate that performance on these benchmarks in fact requires considerable culturally contingent knowledge and assumptions. Singh et al. 2025 found that 28% of MMLU requires culturally-sensitive knowledge to answer correctly, demonstrating that accounting for cultural context can change system rankings. We argue that cultural tasks should be expanded to include any task whose successful execution depends on cultural

Read the complete text