Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology
Minsang Kim, Seungjun Baek · 2024
Read Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology by Minsang Kim, Seungjun Baek (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08846.txt.
Introduction
Minsang Kim and Seungjun Baek quiz open chat models on world folk ways and tie the marks to how each model was raised. Chat tools live close to folks, so they need a sound grasp of folk ways. Their 2024 sweep asks the models to judge folk norms in a span of kinds and lands. Calls are set next to human calls.
Models match folks most on day norms and least on rule webs and growth faith. A West lean marks their calls as a flock. Fruit from many tongues in the train set trims that lean. Big size helps on social marks, yet small models gain from made-up train fare. Raise craft, not just scale, sets folk sight.
Catalog record
| CREATOR | Minsang Kim, Seungjun Baek |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08846
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology Minsang Kim, Seungjun Baek Abstract Large language models (LLMs) closely interact with humans, and thus need an intimate understanding of the cultural values of human society. In this paper, we explore how open-source LLMs make judgments on diverse categories of cultural values across countries, and its relation to training methodology such as model sizes, training corpus, alignment, etc. Our analysis shows that LLMs can judge socio-cultural norms similar to humans but less so on social systems and progress. In addition, LLMs tend to judge cultural values biased toward Western culture, which can be improved with training on the multilingual corpus. We also find that increasing model size helps a better understanding of social values, but smaller models can be enhanced by using synthetic data. Our analysis reveals valuable insights into the design methodology of LLMs in connection with their understanding of cultural values. 1 Introduction & Related Work Large Language Models (LLMs) have become a core technology in many real-world applications to closely interact with humans. Thus, it is important that LLMs understand the cultural values and diversities of human societies. In particular, since LLMs are widely used as general assistants Achiam et al. 2023 ; Team et al. 2023 , they should generate responses which capture the cultural context of target users. For example, people in some countries may think that divorce is largely unacceptable, but those in other countries may consider it as an unpleasant but acceptable practice. Recently, there have been studies on how language models judge moral norms. Schramowski et al. 2022 investigated whether English language models such as BERT have moral bias like humans in English culture. Ramezani and Xu 2023 proposed a methodology for probing moral norms of English language models across diverse cultures. In their experiments, models trained in English predict empirical moral norms across countries worse than the English moral norms. Thus, the authors argue that the moral norms inferred by LLMs have potential risks of bias, and the moral norms suggested by the models should not be regarded as definitive values of morality. Recently, Arora et al. 2023 examined bidirectional language models such as BERT Devlin 2018 and RoBERTa Liu 2019 on cross-cultural values in different countries. While the authors extensively analyzed the LLMs’ predictions, they did not examine the links between training/design choices for LLMs and the human-likeness of LLMs’ judgments. In this paper, we extensively study the progress of LLMs in cultural values across countries using World Value Survey Haerpfer et al. 2022 . We provide detailed observations on the capability and limitations of LLMs and their relation to the design choices for LLMs. We systemically study recent models such as LLaMA Touvron et al. 2023 ; Dubey et al. 2024 , Phi Abdin et al. 2024 , and Yi Young et al. 2024 on diverse and debatable categories of cultural values. In particular, we discuss how pre-training methodology Touvron et al. 2023 ; Team et al. 2024 ; Young et al. 2024 ; Abdin et al. 2024 and model attributes such as model sizes, training corpus, alignment, etc., affect the LLMs’ understanding of human cultures. We expect that our findings provide key insights on design methodology to improve the quality of cultural judgments by LLMs across countries. Our findings are summarized as follows: i) LLMs similarly judge Socio-Cultural Norms with humans, but are less similar in Social Systems and Progress. ii) LLMs’ assessment of cultural values tends to be biased toward Western cultures. iii) However, cultural diversity can be improved by training on the multilingual corpus. iv) Larger models have a stronger cultural-awareness. v) Synthetic data can distill cultural knowledge of larger language models to smaller models. vi) Alignment causes LLMs to resemble humans. 2 Methodology of probing LLMs 2.1 Dataset - World Value Survey We use the World Value Survey (WVS) Haerpfer et al. 2022 dataset to probe LLMs’s knowledge of cultural values across countries. WVS contains questions on diverse topics of social values. There are 209 questions from 12 categories in 55 countries. To probe LLM, we convert all questions and answers to multiple-choice tasks using the template in Fig. 5 in Appendix A.1 . Each question has K candidate answers a_{k} . A sample question: ‘‘Is the death penalty justifiable?’’ has K=3 candidate answers: {A) Never Justifiable, B) Neutral C) Justifiable} . In our experiment, K takes values from \{3,5,7,9\} . Table 2 in Appendix A.3 shows dataset statistics of WVS. 2.2 Method - Scoring Cultural Values Below we describe a method to measure the correlation between the answers provided by LLM and humans. We individually assign scores to candidate answers which enables quantifying correlations in the choice of answers. Let s_{k} denote the score assigned to k -th candidate answer for k\in[K] . We let s_{k} take values between -1 and 1. Since there are K answers, we regularly distribute s_{1},\dots,s_{K} over interval [-1,1] . For example, the question ‘‘Is the death penalty justifiable?’’ has candidate answers never justifiable , neutral , or justifiable . We assign the scores -1, 0, and 1 to the candidate answers. For each question, We will compute the mean score which is the expectation of score over the distribution of answers generated by LLMs and humans, and compare the correlations of their average scores. Firstly, we define the distribution of answer candidates chosen by LLMs. Let p_{k} denote the probability of generating k -th answer candidate for k\in[K] conditional on the question. p_{k} is defined to be the likelihood (softmax output) of answer k of LLM model P : p_{k}=\prod_{i=c+1}^{c+l_{k}}P({w}_{i}|{w}_{i-1},...,{w}_{1}) (1) where w_{i} denotes the i -th token, l_{k} denotes token length of k -th answer candidate, and c denotes the length of question contexts. We normalize p_{k} ’s to yield a probability distribution \hat{p}_{k}=p_{k}/(\sum_{i=1}^{K}p_{i}) Thus, the mean score of LLMs is given by \sum_{i=1}^{K}\hat{p}_{k}s_{k} . Secondly, we compute the distribution of answer choices by humans. The distribution denoted by r_{k} , k\in[K] , is simply set to the empirical distribution of choosing answer candidate k for the given question. The statistics of answer distribution by the survey participants are available in the WVS dataset. Thus, the mean score of humans is given by \sum_{i=1}^{K}r_{k}s_{k} . Finally, we compute the Pearson’s correlation Freedman et al. 2007 between mean scores over questions to measure the degree of agreement between LLM and humans. Figure 1: Comparison of human-ratings and LLM predictions across 55 countries in WVS Haerpfer et al. 2022 . Left: Boxplots of human ratings per category across countries. Right: Corresponding cultural judgment scores estimated by Llama-2-70b Chat Touvron et al. 2023 . Figure 2: Average Pearson’s correlation of small sizes model up to 8B across all countries grouped by continents. Figure 3: Average Pearson’s correlation of larger sizes model including 13B, 34B, and 70B across all countries grouped by continents. 3 Experiment & Analysis 3.1 Experimental settings Baselines. We evaluate various open-source LLMs such as phi-series Li et al. 2023 ; Gunasekar et al. 2023 ; Abdin et al. 2024 , Yi Young et al. 2024 , Llama-2 Touvron et al. 2023 , and Llama-3 Dubey et al. 2024 . We chose the open models because