Attributing Culture-Conditioned Generations to Pretraining Corpora

Huihan Li, Arnav Goel, Keyu He, Xiang Ren · 2024

Read Attributing Culture-Conditioned Generations to Pretraining Corpora by Huihan Li, Arnav Goel, Keyu He, Xiang Ren (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20760.txt.

Description

Abstract In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these biases may stem from uneven cultural representation in pretraining corpora.

Catalog record

CREATORHuihan Li, Arnav Goel, Keyu He, Xiang Ren
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attributing Culture-Conditioned Generations to Pretraining Corpora Huihan Li, Arnav Goel, Keyu He, Xiang Ren Abstract In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these biases may stem from uneven cultural representation in pretraining corpora. This work investigates how pretraining leads to biased culture-conditioned generations by analyzing how models associate entities with cultures based on pretraining data patterns. We propose the MEMOed framework ( MEMO rization from pr e training d ocument) to determine whether a generation for a culture arises from memorization. Using MEMOed on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance. We hope that the MEMOed framework and our insights will inspire more works on attributing model performance on pretraining data. 1 1 1 https://github.com/huihanlhh/CultureGenAttr [Disclaimer: This analysis does not represent any views or beliefs of the authors. Our findings reflect trends observed specifically within OLMo-7B ’s pretraining data and are limited to this dataset. We make no claims about whether these results reflect real-world conditions.] 1 1 footnotetext: Equal Contribution 1 Introduction In open-ended generative tasks like narrative writing or dialogue, language models often show bias against marginalized social groups based on gender, race, or culture ( Gallegos et al., 2024 ; Manvi et al., 2024 ; Li et al., 2024b ) . Cultural bias is particularly notable due to the vast number of cultures to account for. Cultures are often unevenly represented in the pretraining corpora, with some mentioned more frequently than others, irrespective of their real-world prevalence ( Li et al., 2024a ) . Recent studies reveal that models favor entities ( Naous et al., 2023 ) and opinions ( Ryan et al., 2024 ) from frequently represented cultures in pretraining while showing inadequate knowledge and templated answers for less frequent ones ( Li et al., 2024b ) . Figure 1: Four types of culture-symbol associations in culture-conditioned generations Such biases in culture-conditioned generations can be linked to studies showing that LLMs’ memorization and generalization are constrained by pretraining data imbalances. Zhang et al. (2024) find that these imbalances cause models to overgeneralize to high-frequency knowledge, overshadowing lower-frequency knowledge. Similarly, Chang et al. (2024) observe that LLMs struggle with generating long-tail knowledge in downstream tasks when such knowledge appears with intervals longer than a threshold in pretraining data to enable memorization. Building on these findings, we analyze culture biases by examining how models associate entities, referred to as “symbols,” with cultures based on patterns in pretraining data (e.g., “kimono” associated with Japan). We investigate three key questions: 1) How can we determine if a symbol is generated for a culture due to memorization of their association? 2) If not memorization, what other factors drive the model’s association? 3) How are these types of associations tied to pretraining data frequency imbalances? To address the first question, we propose MEMOed ( MEMO rization from pr e training d ocument), a framework to determine if symbols in culture-conditioned generations arise from the model memorizing culture-symbol relationships in pretraining data. MEMOed involves two steps: 1) it identifies contributory documents in the pretraining corpora for the culture-symbol association 2) it classifies the symbol as memorized if the percentage of contributory documents is significant (§ 3.3 ). To answer the second question, we analyze OLMo-7B ( Groeneveld et al., 2024 ) and its pretraining data Dolma ( Soldaini et al., 2024 ) , indexed by Elazar et al. (2024) and Liu et al. (2024) 2 2 2 While our analysis is applicable to any model, our analyses are constrained by open-sourced models with searchable pretraining data. Ablation on OLMo-7B-0424 shows consistent conclusions and can be found in Appendix F.2 . Following ( Li et al., 2024b ) , we collect OLMo-7B ’s culture-conditioned generations about 110 cultures on food and clothing topics and extract topic-related symbols. Using MEMOed , we find that 46% of food symbols and 26% of clothing symbols are generated due to memorization of culture-symbol relationships in pretraining data. For the remaining symbols, we identify three other types of culture-symbol associations: 1) Diffuse Association : symbols not strongly tied to any specific culture but appearing in over half of all cultures’ generations ( e.g. , “t-shirt”), 2) Cross-culture Generalization : symbols memorized with one culture but generated for others ( e.g. , “kimono” is a memorized symbol for Japan but generated for Korea), and 3) Weak Association Generalization : broader conceptual representations derived from synthesis or reinterpretation of memorized symbols and diffuse association symbols ( e.g. , “robe” as a generalized reference to “kimono,” a symbol memorized for Japan). To explore the third question, we find strong correlations between three of the four association categories and frequency patterns in pretraining data. Memorized associations strongly correlate with a culture’s occurrence frequency, indicating that low-frequency cultures in the “long-tail” lack sufficient pretraining supervision for memorization. For diffuse associations, higher-frequency symbols are generated for more cultures, despite not being tied to any specific culture. Cross-culture generalization shows that cultures with higher pretraining frequencies are more likely to have their memorized symbols generated for others. Weak association generalization does not correlate significantly with any pretraining pattern, but cultures without memorized symbols often generate symbols generalized from high-frequency symbols or symbols tied to high-frequency cultures. In summary, our work presents a generation attribution framework that allows researchers to clearly trace culture-conditioned generations to memorization of patterns in pretraining data. Our findings suggest that language models are unable to reliably and evenly recall knowledge about global cultures in downstream generations, and resort to repeating a small set of high-frequency symbols. Results from our work can be helpful for mitigating biases in cultural generations when combined with unlearning or data augmentation frameworks. We hope that we inspire more works on attributing model performance to pretraining data. 2 Related Works Memorization and Generalization. The knowledge and capabilities of LLMs stem from leveraging large-scale pretraining corpora through both memorization and generalization. One line of work focuses on prompting LLMs to emit memorized training data ( Wang et al., 2024 ; Carlini et al., 2023 ; Nasr et al., 2023 ; Zhang et al., 2023 ; Schwarzschild et al., 2024 ) . Carlini et al. (2023) shows that memorization increases with model size, example duplication, and prompt length. Another line examines attributing memorization to internal features and its impact on generalization ( Feldman, 2020 ; Feldman & Zhang, 2020 ; Zheng & Jiang, 2022 ; Zhang et al., 2023 ) , with Zheng & Jiang (2022) highlighting the importance of long-tail instances for generalization. Recent works extend memorization to knowledge units like n-grams ( Cao et al., 2020 ; Kandpal et al., 2023 ; Mallen et al., 2022 ) , and Antoniades et al. (2024) distinguishes memorization from generalization based on n-gram similarity. Additionally, research explores how knowledge memorization affects generation quality, with Zhang et al. (2024) and Chang et

Read the complete text