Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models

Haeun Yu Seogyeong Jeong Siddhesh Pawar, Jisu Shin Jiho Jin Junho Myung Alice Oh Isabelle Augenstein · 2025

Read Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models by Haeun Yu Seogyeong Jeong Siddhesh Pawar, Jisu Shin Jiho Jin Junho Myung Alice Oh Isabelle Augenstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08879.txt.

Introduction

Haeun Yu and co-authors present CulTrace in a 2025 paper, a method for watching how a large language model thinks about culture on the inside. Earlier studies judged cultural awareness only from the model's answers, which shows what goes wrong but not why. CulTrace instead traces activity inside the model layer by layer while it answers cultural questions, mapping when and where the relevant culture gets picked out.

The team finds a steady three-stage pattern: the model first takes in the question's topic, then settles on the culture involved, and only then narrows to an answer. That middle step runs late and muddy for cultures that are rare in training data, which explains many wrong answers. The authors argue that fixing cultural bias will require working on these internal steps, not just scoring final outputs.

Catalog record

CREATORHaeun Yu Seogyeong Jeong Siddhesh Pawar, Jisu Shin Jiho Jin Junho Myung Alice Oh Isabelle Augenstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models Haeun Yu Seogyeong Jeong Siddhesh Pawar, Jisu Shin Jiho Jin Junho Myung Alice Oh Isabelle Augenstein Abstract The growing deployment of large language models (LLMs) across diverse cultural contexts necessitates a better understanding of how the overgeneralization of less documented cultures within LLMs’ representations impacts their cultural understanding. Prior work only performs extrinsic evaluation of LLMs’ cultural competence, without accounting for how LLMs’ internal mechanisms lead to cultural (mis)representation. To bridge this gap, we propose Culturescope , the first mechanistic interpretability-based method that probes the internal representations of LLMs to elicit the underlying cultural knowledge space. CultureScope utilizes a patching method to extract the cultural knowledge. We introduce a cultural flattening score as a measure of the intrinsic cultural biases. Additionally, we study how LLMs internalize Western-dominance bias and cultural flattening, which allows us to trace how cultural biases emerge within LLMs. Our experimental results reveal that LLMs encode Western-dominance bias and cultural flattening in their cultural knowledge space. We find that low-resource cultures are less susceptible to cultural biases, likely due to their limited training resources. Our work provides a foundation for future research on mitigating cultural biases and enhancing LLMs’ cultural understanding. Our codes and data used for experiments are publicly available 1 1 1 https://github.com/copenlu/CultureScope . 1 Introduction Large language models (LLMs) are increasingly being used in culturally diverse contexts, where understanding and responding appropriately to each culture is essential ( Pandya and Holia 2023 ; Salemi et al. 2023 ; Liu et al. 2025 ) . However, the cultural knowledge acquired by LLMs is largely shaped by the data they are trained on, which is predominantly Western-centric ( Santurkar et al. 2023 ) . This results in severe geographic imbalances, where some regions receive disproportionate attention while others remain marginalized ( Nguyen et al. 2023a ) . While LLMs do acquire some cross-cultural patterns during training ( Hershcovich et al. 2022 ; Arora et al. 2023 ) , this learning often results in overgeneralization. For example, when asked about a popular leisure activity for retired men in Azerbaijan, GPT-4 responded with chess—a plausible answer, but one that reflects general stereotypes of post-Soviet or Eastern European regions rather than Azerbaijan specifically ( Myung et al. 2024 ) . Figure 1: Given the question about the popular domestic vacation spot in Greece, Culturescope first generates an answer to cultural Question at the Inference stage. Then, it reads the hidden representation from the Inference stage and elicits the cultural knowledge used for ‘Zakynthos’ at the Scoping-in stage. We finalize a list of cultural knowledge after the Filtering stage. Culturescope unveils the internal mechanism of LLMs that cannot be revealed through the Inference stage alone. However, these patterns of overgeneralization directly perpetuate cultural biases such as cultural flattening. Measuring such overgeneralizations through model outputs alone proves insufficient since generated outputs mask the underlying mechanisms that lead to cultural (mis)representation. To facilitate the examination of the underlying mechanisms, we propose to study LLMs’ cultural understanding with mechanistic interpretability (MI) techniques. MI techniques provide us with methods that can directly examine how cultural biases discovered by the extrinsic evaluation Santurkar et al. 2023 is internally processed within model representations, revealing where and how harmful generalizations emerge. We are the first to propose an approach for intrinsic cultural bias evaluation. In this work, we introduce Culturescope , a method to probe internal representations and surface the cultural knowledge activated during cultural understanding tasks (§ 4.1 ). By surfacing this knowledge, we reveal the internal cultural knowledge space from which model outputs are generated. Figure 1 illustrates an overview of Culturescope. To examine the intrinsic ‘cultural flattening’ embedded in the parameter space, we introduce a cultural flattening (CF) score, which quantifies the degree of intersection between cultural knowledge decoded by Culturescope (§ 4.2 ). We implement our framework on two cultural understanding tasks, cultural commonsense Question Answering (QA) and extractive QA, across three different LLMs. We further challenge the model’s cultural understanding by creating multiple-choice questions (MCQs) with hard negatives (§ 3.2.3 ). Culturally nuanced answers from high-resource cultures or geographically proximate countries are selected as hard negative options to simulate the Western-dominance bias and the cultural flattening. This setup refrains LLMs to leverage surface-level elimination strategies based on the overgeneralization Khan et al. 2025 . Analyzing selected options by LLMs with the attention map method Yuksekgonul et al. 2024 allows us to examine whether extrinsic and intrinsic cultural biases align, by revealing which options the model internally attends to (§ 5.1 ). High CF scores of Western and high-resource cultures show that LLMs encode Western-dominance bias and cultural flattening in their cultural knowledge space (§ 6.2 ). Model performances on MCQs with hard negatives demonstrate that low-resource cultures are less likely to be affected by the biases, likely due to their limited training resource (§ 6.3 ). This implies that LLMs struggle with low-resource cultures due to the lack of parametric knowledge. In § 6.4 , we find that LLMs over-attend to tokens from Western and high-resource cultures. This suggests Western-dominance bias is more internalized than the cultural flattening. Our work provides useful signal to future research on mitigating the internalized patterns for biases to build a better culturally aligned model. 2 Related Work Evaluating Cultural Understanding of LLMs Previous work has proposed evaluation datasets and frameworks to assess LLMs’ cultural understanding ability acquired during pre-training Keleg and Magdy 2023 ; Naous and Xu 2025 ; Pawar et al. 2025 . BLEnD Myung et al. 2024 provides a multilingual commonsense QA dataset spanning 16 countries and regions, designed to uncover cross-cultural disparities in everyday knowledge. CAMeL Naous et al. 2024 compares LLM behavior in Arabic versus Western settings across tasks like story generation, NER, and sentiment analysis, exposing systematic cultural biases in LLMs. Other multilingual benchmarks Zhou et al. 2025 ; Hasan et al. 2025 ; Wang et al. 2024 ; Cao et al. 2024 construct culturally localized evaluation datasets that span domains such as cuisine, proverbs, news, and reasoning. Across these datasets, performance gaps are consistently observed between high-resource and underrepresented languages and cultures, often linked to pre-training data imbalances that favor dominant regions Naous and Xu 2025 . While these efforts highlight important cross-cultural disparities, they perform an extrinsic evaluation, overlooking the underlying mechanism and cultural knowledge space embedded in LLMs. To address this gap, our paper aims to reveal how culture is embedded, entangled, or flattened within the models’ inner representations. Mechanistic Interpretability MI techniques are developed to explain the inner workings of LLMs by identifying responsible model components, such as neurons and attention heads Meng et al. 2022 ; Geva et al. 2023 ; Yu et al. 2024 . Leveraging their transparency, recent studies have employed MI techniques to investigate how specific behaviors emerge in LLMs. For instance, they have been used to uncover and manipulate components associated with social biases, enabling both diagnostic and steering interventions Liu et al. 2024 ; Durmus et al. 2024 ; Yang et al. 2024 . Despite growing interest in the cultural capabilities of LLMs, no prior work has explored cultural understanding through the lens of MI. Our study fills this

Read the complete text