CURE: Cultural Understanding & Reasoning Evaluation – A Framework for “Thick” Culture Alignment Evaluation in LLMs
Truong Vo, Sanmi Koyejo · 2025
Read CURE: Cultural Understanding & Reasoning Evaluation – A Framework for “Thick” Culture Alignment Evaluation in LLMs by Truong Vo, Sanmi Koyejo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12014.txt.
Introduction
Truong Vo and Sanmi Koyejo build CURE, a benchmark suite for whether language models understand culture thickly, in lived situations, rather than thinly, as trivia about norms. Deployed models serve culturally diverse users, but most evaluations probe decontextualized statements or force choices that real cultural competence never faces.
Benchmarks present realistic situational contexts demanding culturally grounded reasoning. Beyond exact match scoring, four new metrics, coverage, specificity, connotation, and coherence, grade different facets of response quality. Frontier models are analyzed under this richer lens. Alignment claims about culture need instruments sensitive to context, and CURE is an attempt to build one.
Catalog record
| CREATOR | Truong Vo, Sanmi Koyejo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12014
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CURE: Cultural Understanding & Reasoning Evaluation – A Framework for “Thick” Culture Alignment Evaluation in LLMs Truong Vo, Sanmi Koyejo Abstract Large language models (LLMs) are increasingly deployed in culturally diverse environments, yet existing evaluations of cultural competence remain limited. Existing methods focus on de-contextualized correctness or forced-choice judgments, overlooking the need for cultural understanding and reasoning required for appropriate responses. To address this gap, we introduce a set of benchmarks that, instead of directly probing abstract norms or isolated statements, present models with realistic situational contexts that require culturally grounded reasoning. In addition to the standard Exact Match metric, we introduce four complementary metrics (Coverage, Specificity, Connotation, and Coherence) to capture different dimensions of model’s response quality. Empirical analysis across frontier models reveals that thin evaluation systematically overestimates cultural competence and produces unstable assessments with high variance. In contrast, thick evaluation exposes differences in reasoning depth, reduces variance, and provides more stable, interpretable signals of cultural understanding. Introduction Foundational theories in sociology and anthropology conceptualize culture in ways that resist straightforward measurement: as shared systems of meaning shaped through symbolic interpretation within and across cultures ( 4 ) , as “toolkits” of strategies that actors draw on to navigate social life ( 17 ) , or as informal institutions that shape behavior by providing unwritten rules and constraints ( 10 ) . These conceptualizations emphasize that true cultural competence requires accounting for behavioral norms, contextual flexibility, and implicit social contracts, not merely stated beliefs or values. However, researchers working on AI cultural alignment often draw upon established quantitative frameworks from the social sciences, particularly survey-based tools like the World Values Survey ( 8 ) , Values Survey Module ( 7 ) , and OpinionQA ( 14 ) . While these approaches offer reproducibility and scalability, they often capture only surface-level cultural signals, such as whether individuals endorse particular values, agree with abstract statements, or select from predefined response options. Recent evidences suggest they may be measuring randomness rather than genuine cultural representation ( 11 ) . Another issue of this probing method, focusing on accuracy with survey-based questionnaire, is the consistent finding that many LLMs, regardless of anthropological prompting ( 1 ) , tend to converge on a ”moderate cultural middle ground” or a ”global average” culture ( 16 ) . While this is often interpreted as a form of cultural bias (e.g., a bias towards a generic, globalized culture), it can also be understood as a statistical artifact of the evaluation method itself. In response to the limitations of surface-level (“thin”) cultural evaluation for language models, we introduce an operationalization for “thick” cultural assessment. Building on the distinction between “thin descriptions” (factual observation) and “thick descriptions” (interpretation of context and meaning) ( 5 ) , we argue that cultural evaluation of LLMs must go beyond correctness or preference agreement. Effective evaluation should capture how well model responses reflect embedded norms, symbolic meaning, and situational appropriateness within specific cultural contexts. To this end, our framework requires models not only to make decisions but to generate free-form justifications, which are assessed against richer cultural criteria. We operationalize thick evaluation through a four-scenario benchmark, each designed to target a distinct diagnostic dimension of cultural reasoning. By situating norms in realistic contexts rather than abstract prompts, our benchmark measures both the accuracy of model outputs (“thin”) and the depth of their reasoning (“thick”), providing a comprehensive assessment of cultural competence. Overall, we address the limitations of surface-level, survey-style benchmarks that primarily test factual recall or binary norm classification. In contrast, we propose a thick cultural evaluation framework that emphasizes situated reasoning, symbolic understanding, and role-sensitive behavior in context. Our main contributions are: • We develop a contextual, persona-driven, and free-response benchmark that elicits context-sensitive cultural reasoning in multiple value-laden scenarios. • We propose and validate new metrics for evaluating culture reasoning, leveraging an LLM-as-a-Judge to score responses and conducting human-in-the-loop validation to ensure reliability and calibration of the automatic grader. • We integrate both thin-with-context culture benchmark (multiple-choice accuracy) and thick (free-response reasoning) cultural evaluation into the HELM benchmark, enabling direct comparison of frontier and open-source LLMs. Related Work To move beyond traditional value surveys, several benchmarks have emerged that aim to evaluate cultural knowledge and norm adherence in more applied, context-sensitive settings. NormAD ( 13 ) tests a model’s ability to judge the social acceptability of actions in narrative scenarios drawn from 75 countries, offering a broader geographic range and cultural diversity. CASA ( 12 ) places AI agents in simulated web-based environments, evaluating their ability to detect and respond to culturally grounded norm violations in tasks such as online shopping. DailyDilemmas ( 2 ) probes a model’s implicit value preferences by presenting forced-choice moral dilemmas that involve trade-offs between competing norms or values, although these are often framed through a Western-centric lens. CultureBank ( 15 ) adopts a bottom-up strategy by curating a repository of user-contributed narratives, anchoring evaluations in lived cultural experience rather than abstract norms. While these existing benchmarks mark an important shift toward more context-aware cultural evaluation, they largely rely on a single evaluation metric-typically whether the model’s response aligns with a predefined target norm or label. This often flattens the complexity of cultural competence into binary judgments (e.g., correct vs. incorrect, acceptable vs. unacceptable), ignoring the deeper layers of interpretation, symbolic meaning, and contextual sensitivity that shape culturally appropriate behavior ( 6 ) . As a result, these benchmarks fall short in assessing whether a model can explain why a behavior is appropriate or how it should adapt across cultural settings. Benchmark Construction Drawing on the concept of ”thick description” ( 5 ) , we identify three categories of evaluation that distinguish between thin and thick culture competence in LLMs. These categories clarify the spectrum of evaluation approaches, from surface-level judgments to contextually rich assessments requiring deeper reasoning and justification. • Thin (Norm-Only). These approaches rely on categorical questions detached from context (e.g., WVS, VSM, OpinionQA). While highly scalable, they risk encouraging superficial pattern-matching and often fail to capture the complexity of situational nuance. • Thin-with-Context. These methods embed categorical judgments within more realistic scenarios that include personas or narrative frames. Representative examples include NormAd, CASA, and CultureBank. To enhance this category, we develop SpecNorm, a new dataset captures subgroup-level variation through ethnicity, religion, and regional cues. • Thick Evaluation. Moving beyond binary judgments, this approach requires models to produce 2–4 sentence justifications grounded in persona details, situational context, and relevant social norms. Each benchmark in this category targets a distinct diagnostic dimension: Coherence (CultureBank), Connotation (CASA), Coverage (NormAd), and Specificity (SpecNorm). These categories support a more nuanced understanding of cultural evaluation in LLMs, offering increasing interpretability and cultural sensitivity as one moves from thin to thick evaluation. In our work, we implement both Thin-with-Context and Thick Evaluation settings across four datasets (Table 1 ), yielding eight scenarios on the HELM Leaderboard to compare accuracy and reasoning depth. Dataset Size Countries Key Feature NormAd 2,600 75 Narrative social behavior SpecNorm 3,766 145 Subgroup-specific (ethnicity, religion, region) CASA 1,198 17 Symbol connotation CultureBank 22,990 120+ Persona-guided reasoning Table 1: Benchmark datasets for paired thin with context and thick cultural evaluation 1. SpecNorm Benchmark SpecNorm is a benchmark