Musical ethnocentrism in Large Language Models

Anna Kruspe · 2025

Read Musical ethnocentrism in Large Language Models by Anna Kruspe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13720.txt.

Introduction

Musical bias in large language models is measured by Anna Kruspe in a 2025 paper. Such models reflect biases in their training data, including geocultural bias from uneven coverage of regions and cultures. Music is a clear test case, since tastes and canons vary widely around the world, yet this angle had been little studied.

Kruspe ran two experiments on models including ChatGPT and Mixtral. First, the models listed the top musical contributors in varied categories, and the countries of origin were counted. Second, the models rated aspects of the musical cultures of distinct countries with numbers. Both tests showed a strong preference for Western music cultures. Readers get early evidence that global users may get a narrow view of music from these tools.

Catalog record

CREATORAnna Kruspe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13720

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Musical ethnocentrism in Large Language Models Anna Kruspe Abstract Large Language Models (LLMs) reflect the biases in their training data and, by extension, those of the people who created this training data. Detecting, analyzing, and mitigating such biases is becoming a focus of research. One type of bias that has been understudied so far are geocultural biases. Those can be caused by an imbalance in the representation of different geographic regions and cultures in the training data, but also by value judgments contained therein. In this paper, we make a first step towards analyzing musical biases in LLMs, particularly ChatGPT and Mixtral. We conduct two experiments. In the first, we prompt LLMs to provide lists of the “Top 100” musical contributors of various categories and analyze their countries of origin. In the second experiment, we ask the LLMs to numerically rate various aspects of the musical cultures of different countries. Our results indicate a strong preference of the LLMs for Western music cultures in both experiments. 1 Introduction It has long been known that machine learning models pick up and thus perpetuate human biases in various ways, most prominently by learning them from their training data. For text-based models, even early embedding approaches exhibited e.g. gender bias Bolukbasi et al. 2016 . With the recent rise of Large Language Models (LLMs), gender and race biases were quickly discovered and analyzed in various domains Kotek et al. 2023 ; Sun et al. 2023 ; Omiye et al. 2023 ; Warr et al. 2023 . Types of bias that have been considered somewhat less include those based on culture and geography. However, Manvi et al. 2024 recently showed that LLMs also exhibit those, both implicitly and explicitly. Their research demonstrated that when prompted to rate random locations on Earth on various characteristics, LLMs generally yielded lower ratings for certain regions, e.g. the global South. There appear to be correlations with the coverage of regions and their cultural and historical significance in the training data, statistics across a range of aspects around the world, and possibly structural biases of the institutions creating these models, which are mainly based in North America and Europe. We hypothesize that such biases are not only present for purely geographic topics, but also for cultural developments in different regions of the world. In this paper, we conduct first experiments to detect such biases with regards to music culture. Those are based on two different types of measurements, translated into prompts: a) Asking models to give an overview of top musical artists, and b) asking models to rate aspects of musical culture in different regions of the world. The first experiment elicits model bias on an open-ended question with regards to presence of different cultural regions in the models, while the second one employs a direct comparison to extract implicit judgments learned by the models. To gain insights into the influence of where and how the model was trained, we prompt two models from different regions of the world in four languages. The rest of the paper is structured as follows: Section 2 gives an overview of other work in the field of geocultural biases in LLMs. Section 3 provides details of our experimental design. The results are presented in section 4 . Finally, sections 5 and 6 discuss our findings and make suggestions for future work. 2 Related work Initial studies, such as those discussed in Tao et al. 2024 and Naous et al. 2023 , show LLMs often encode biases favoring Western, English-speaking norms, impacting their fairness and representation of non-Western cultures as well as performance on non-Western topics, like Traditional Chinese Medicine Zhu et al. 2024 . Further research seeks methodologies to measure and mitigate these biases more accurately. The interdisciplinary approach in Biedma et al. 2024 and the survey on modeling culture in LLMs Adilazuarda et al. 2024 propose new frameworks for understanding and adjusting the embedded cultural values in LLMs. The “CulturePark” Li et al. 2024b initiative and the “NormAd” Rao et al. 2024 benchmark are notable in their attempts to simulate cross-cultural communication scenarios and assess LLMs’ adaptability to cultural contexts through synthetic story generation, providing novel approaches to evaluating cultural sensitivity and adaptability in AI technologies. The “CDEval” Wang et al. 2023 benchmark specifically addresses the need to evaluate the cultural dimensions of LLMs, integrating automated generation and human verification to assess cultural traits across multiple domains. Similarly, the “CultureLLM” Li et al. 2024a project aims to fine-tune LLMs on culturally diverse data. In the music domain, Li et al. 2024c illustrates domain-specific biases, arguing for more comprehensive benchmarks in varied knowledge domains. 3 Methodology In this section, we will describe our experimental design, including used models, prompt design, prompted tasks, and postprocessing of the results. 3.1 Models We tested our bias prompts on two different models via their online interfaces: • ChatGPT-4 (paid version) via its online interface ( https://chatgpt.com/ ) • Mixtral-8x7B via the online interface under https://deepinfra.com/mistralai/Mixtral-8x7B-Instruct-v0.1 , maximum new token length set to 10,000 ChatGPT was created in the US by OpenAI, while Mixtral was released by the French company Mistral AI. We wanted to compare models from two different regions of the world on this geocentric task. A more geographically wide-ranging selection of LLMs would be of high interest for future comparisons. Currently, Chinese institutions are also intensifying their efforts in the LLM domain, but we were not able to obtain access to a freely available Chinese model. ⬇ You will be given a country randomly sampled from all human - populated locations on Earth . You give your rating keeping in mind that it is relative to all other human - populated locations on Earth ( from all continents , countries , etc .). You provide ONLY your answer in the exact format " My answer is X . X ." where ‘ X . X ’ represents your rating for the given topic . ... task : Agreeableness of music region : Denmark Listing 1: Example prompt for the rating experiment 3.2 Prompt design We conducted two experiments. In the first one, we asked open-ended questions about the “Top 100” musical performers of various types. Those included bands, solo musicians, singers, instrumentalists, and composers. Prompts were simply of the form " Name the Top 100 singers / instrumentalists / bands /..." . We then asked the model to extend this list with the performers’ countries of origin. In the second experiment, we asked the models to rate certain characteristics of the music of all countries in the world. We used the methodology from Manvi et al. 2024 to design our prompts. In essence, the LLMs are given an over-all task of providing ratings on a certain topic for a certain region of the world compared to all other inhabited areas. An example is shown in Listing 1 . Then, we focused on a specific characteristic and gave the model a list of all countries. For such subjective topics, it is usually not possible to ask for direct comparisons (e.g. “which country has the best music”) due to content filters, but ratings worked well. The aspects of music culture included agreeableness, successfulness, musical creativity, global influence,

Read the complete text