Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs’ Cultural Intelligence with CQ-Bench
Ziyi Liu, Priyanka Dey, Jen-tse Huang, Zhenyu Zhao, Bowen Jiang, Rahul Gupta, Yang Liu, Yao Du, Jieyu Zhao · 2025
Read Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs’ Cultural Intelligence with CQ-Bench by Ziyi Liu, Priyanka Dey, Jen-tse Huang, Zhenyu Zhao, Bowen Jiang, Rahul Gupta, Yang Liu, Yao Du, Jieyu Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01127.txt.
Introduction
Ziyi Liu and co-authors built this 2025 benchmark to test whether language models grasp unspoken cultural values. People rarely state their values outright in daily talk. Models that serve users worldwide need to read those hints well. Older tests lean on stated norms, so they miss this quieter skill.
Their test, CQ-Bench, uses short stories with several speakers in conversation. The values come from two large world surveys and span ethics, faith, and social views. A build pipeline checked each story for fit, consistency, and hiddenness of the values, reaching high agreement with human judges. Three tasks of rising difficulty ask models to spot attitudes and pick out values. Top models matched humans at picking values but lagged at reading attitudes with nuance.
Catalog record
| CREATOR | Ziyi Liu, Priyanka Dey, Jen-tse Huang, Zhenyu Zhao, Bowen Jiang, Rahul Gupta, Yang Liu, Yao Du, Jieyu Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01127
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs’ Cultural Intelligence with CQ-Bench Ziyi Liu, Priyanka Dey, Jen-tse Huang, Zhenyu Zhao, Bowen Jiang, Rahul Gupta, Yang Liu, Yao Du, Jieyu Zhao Abstract Cultural Intelligence (CQ) refers to the ability to understand unfamiliar cultural contexts—a crucial skill for large language models (LLMs) to effectively engage with globally diverse users. Existing studies often focus on explicitly stated cultural norms, but fail to capture the subtle, implicit values that are common in daily conversation. To address this gap, we introduce CQ-Bench , a benchmark specifically designed to assess LLMs’ capability to infer implicit cultural values from natural conversational contexts. CQ-Bench consists of multi-character conversation-based stories using values from the World Value Survey and the GlobalOpinions , with topics including ethical, religious, social, etc. Our automatic dataset construction pipeline integrates rigorous validation procedures (incorporation, consistency, and implicitness checks), achieving a 94.5% human–model agreement in the final validation. To leverage CQ-Bench data, we design three tasks of increasing complexity: attitude detection, value selection, and value extraction. These tasks evaluate whether models can detect attitude and recognize values embedded within natural dialogues rather than relying on explicit cultural knowledge. We find that while frontier models like o1 reach human-level performance in value selection (0.809 F 1 F_{1} ), they still fall short in nuanced attitude detection (0.622 F 1 F_{1} ). Notably, fine-tuning a smaller LLaMA-3.2-3B on only 500 culturally-rich examples improves performance by over 10% , even outperforming o3-mini in some cases. Using CQ-Bench , we provide insights into the current challenges in LLMs’ CQ research and suggest practical pathways for enhancing LLMs’ cross-cultural reasoning abilities. 1 1 1 The code is available at: https://github.com/uscnlp-lime/CQ-Bench 1 Introduction Large language models (LLMs) have demonstrated impressive capabilities in understanding and generating culturally relevant text ( Li et al., 2024a ; Li et al., 2024c ; Li et al., 2024b ; Putri et al., 2024 ) . Prior research on LLM cultural alignment primarily focuses on modeling differences between national cultures ( Pujari and Goldwasser, 2024 ; Kharchenko et al., 2024 ; Shi et al., 2024 ; Wang et al., 2024 ) or aligning models with culturally specific norms ( Zhong et al., 2024 ; Rozen et al., 2024 ; Johnson et al., 2022 ; Kim et al., 2024 ; Monazzah et al., 2025 ) . Figure 1: An illustration of CQ-Bench . We construct three distinct tasks based on conversation-style stories to assess the cultural intelligence of LLMs in CQ-Bench . However, cultural values in real-world interactions are not solely determined by demographic characteristics. Individuals within the same cultural group can hold diverse and even conflicting beliefs ( Fischer and Poortinga, 2012 ) , and LLMs risk oversimplifying human perspectives if they rely merely on broad demographic generalizations Saha et al. (2025a) . This limitation becomes particularly problematic in human-AI interactions, where successful communication relies not only on an LLM’s ability to recognize explicit cultural markers but also on its capacity to infer implicit cultural values. While LLMs can produce diverse responses when a persona and values are explicitly provided ( Grassi et al., 2024 ; Wang et al., 2025 ; Huang et al., 2024 ; Jiang et al., 2025 ) , real-world communication often lacks such explicit signals. Humans do not typically express their values in a structured debate format Chatterji et al. (2025) ; instead, values are subtly embedded into casual conversations and personal anecdotes. Cultural intelligence (CQ) refers to an outsider’s ability to interpret unfamiliar and ambiguous cultural cues ( Blanchard and Mohammed, 2024 ; Earley and Ang, 2003 ) . For LLMs, CQ is crucial for engaging in meaningful conversations with individuals from diverse backgrounds. Unlike traditional assessments of value understanding that focus on detecting explicit statements about cultural norms ( Ren et al., 2024 ; Kiesel et al., 2023 ) , it emphasizes deeper contextual reasoning—mirroring real-life interactions where implicit beliefs are embedded in everyday speech and action. To address these gaps, we introduce CQ-Bench , a benchmark designed to evaluate LLMs’ ability to infer implicit cultural values within conversational contexts (Figure 1 ). We propose a structured automatic pipeline for generating multi-turn, multi-character conversations that naturally embed cultural values. We design additional incorporation, consistency, and implicitness checks using GPT-4o as a judge to ensure dataset quality. Human evaluation shows a 94.5% agreement with the GPT-4o’s judgment. We then evaluate various LLMs on three increasingly challenging tasks based on CQ-Bench : attitude detection, value selection, and value extraction, to measure their ability to recognize, interpret, and extract cultural values from conversations. While larger models generally show more promising results than smaller models in most tasks, our detailed analysis reveals notable performance variations across different categories of values. For example, models extract political values well (above 0.7 F_{1} score overall) but perform much worse in religious values (below 0.6), highlighting substantial room for improving LLMs’ understanding of culture values. Our contributions are as follows: • Benchmark Design: We are the first to benchmark LLMs’ cultural intelligence via CQ-Bench , which assesses their ability to infer implicit cultural values in conversation. • Automatic Dataset Generation Pipeline: We introduce a pipeline that automatically generates high-quality synthetic datasets. The strong agreement between our model and human evaluations demonstrates the robustness of the pipeline and ensures reproducibility for future research. • Comprehensive Evaluation: We construct three escalating tasks—attitude detection, value selection, and value extraction—to systematically evaluate open-sourced and close-sourced models’ CQ. We further perform both supervised and reinforcement fine-tuning with GRPO to enhance the reasoning and alignment. 2 CQ-Bench Evaluating a model’s cultural intelligence goes beyond simply identifying a broad value in a short speech Ren et al. (2024) . To reflect real-world scenarios, we design comprehensive tasks that examine implicit cultural value understanding from multiple angles. As shown in Figure 1 , CQ-Bench features three tasks of growing difficulty: attitude detection, value selection, and value extraction. We will first introduce the definition of culture value and conversation setup in CQ-Bench and then explain the tasks in detail. 2.1 Culture Value Cultural values are defined as values inherently linked to culture and expressed through different attitudes. Each cultural value consists of two components: (1) Statement , which presents or solicits an opinion, and (2) Attitude , which signifies agreement or disagreement with the statement. As shown in Figure 1 , “The only acceptable religion is my religion” is a statement while “Disagree” is an attitude. Different statements offer multiple attitude options, aligning with the settings in the original questionnaires. The statements in this study are sourced from the World Values Survey (WVS) ( Haerpfer et al., 2022 ) and the GlobalOpinion dataset ( Durmus et al., 2023 ) . The WVS is a global research project that explores individuals’ values and beliefs, how their evolution over time, and their sociopolitical implications. The GlobalOpinion dataset contains a subset of survey questions about global issues and public opinions, adapted from the WVS and the Pew Global Attitudes Survey. We manually select values that either focus on personal beliefs or characterize societal and community attributes. 2.2 Conversation Setting In CQ-Bench , each conversation story features 4–5 characters, with randomly selected cultural values implicitly embedded in the narrative. We consider three distinct