Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment?
2026
Read Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment? (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25654.txt.
Introduction
Branco and twelve colleagues review datasets for assessing cultural alignment in large language models, moving past artworks and tourist clichés. Cultural mismatch has drawn growing attention, yet little work has addressed dataset design. The team identifies the main limits of existing approaches and proposes guidelines for raters. They then build a dataset on those rules and run contrastive experiments with it. The design yields test sets with greater power to tell apart models specialised for a given culture from those that are not, all else equal.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42025 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment? Abstract Although the cultural (mis)alignment of Large Language Models (LLMs) has attracted increasing attention—often framed in terms of cultural bias—until recently there has been limited work on the design and development of datasets for cultural assessment. Here, we review existing approaches to such datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a dataset built according to these principles. We further present a series of contrastive experiments conducted with this dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus . Keywords: LLMs, cultural alignment, test datasets, language resources evaluation [chinese]rmNotoSansCJK-Regular.ttc [korean]rmNotoSansCJK-Regular.ttc [hindi]rmNotoSansDevanagari-Regular.ttf [farsi]rmNotoSansArabic-Regular.ttf Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment? António Branco, † João Silva, † Nuno Marques, † Luis Gomes, † Ricardo Campos, ‡ Raquel Sequeira, ‡ Sara Nerea, ‡ Rodrigo Silva, ‡ Miguel Marques, ‡ Rodrigo Duarte, ‡ Artur Putyato, ‡ Diogo Folques, ‡ Tiago Valente ‡ † University of Lisbon, ‡ University of Beira Interior [email protected] Abstract content 1. Introduction Since the advent of Large Language Models (LLMs), the question of how to undertake their assessment has become a central topic of research. Initially such evaluation efforts were supported mostly by the so-called instruct datasets, consisting of pairs of input questions and of the respective gold answers, thus focusing on the semantic aptitude of the models, that is, in their aptitude to basically handle knowledge about the world ( 20 ) . Soon, further efforts were directed to other aptitudes as well ( 17 ) , such as, among others, their aptitude in terms of civility, to have relations with humans or other agents within the limits of consensually established social constraints on courtesy and proper interaction ( 29 ) . Concomitantly, issues related to sovereignty became also a major topic of debate and research. As these have been articulated in a wide range of sectors, from governments to scientific researchers, they have been concerned with what has been termed “cultural”, “linguistic”, or, even more broadly, “digital” sovereignty ( 3 ; 4 ) . At the intersection of these two important lines of debate and inquiry, one finds the discussion on how to properly assess LLMs for cultural aptitude ( 2 ) , for their ability to know, acknowledge, handle, deliver, respect and/or support what any given group assumes it is or belongs to its own culture, which helps to individuate such group and socially bind their members among themselves—be it that it envisages itself as a nation, a people, a town, a group of fans, etc. Like for the assessment of other aptitudes of LLMs, a central role is played by the research on how to design and develop test datasets to undertake such assessment ( 12 ) . That is the topic of this paper. A first goal here is to present a critical analysis of the major examples or approaches in the literature to the design of test datasets for the purpose of evaluating cultural alignment. Our tenet is that they suffer from a number of limitations. Focusing mostly on historical events or literate culture highlights, that many times belong also to universal or global culture, or echoing stereotypical clichés from points of view that are external to the social group at stake, among others, are just a couple of such limitations. Accordingly, the leverage of such datasets to ultimately fulfill its intended role—of discriminating between models that may happen to be more or less aligned with the distinctiveness of the culture of a group—vanishes given the widespread presence in the web of such facets, from where the training data for those models have been extensively scraped. Taking that analysis into account, another major goal of this paper is to propose a design approach for these datasets that avoids such limitations. This approach is materialized in a set of development guidelines that we present and justify. Their appropriateness to provide discriminative leverage is empirically assessed by experimenting with the development of a dataset under such guidelines, concerning a given culture, and with its subsequent usage for evaluating the respective cultural alignment of a range of LLMs—large and small, with and without having been subject to dedicated finetuning for that culture. We discuss previous approaches to the design of datasets for cultural alignment in Section 2 . Taking into account the lessons learned from that analysis, in Section 3 we propose and discuss design guidelines for annotators, and in the following Section 4 the development of a dataset according to these guidelines is presented. This dataset was used to evaluate the cultural alignment of a range of LLMs and the results and implications of this experiment are reported in Section 5 . In the final Section 6 , conclusions are presented. 2. Background In this section we proceed with an overview of the literature related to the design of datasets aimed at assessing cultural alignment. Though the issues raised around this topic have received considerable highlight in the media and public discussion since the advent of ChatGPT, under the consideration of so-called cultural biases of models, somewhat surprisingly until a few years ago there were not many publications on cultural assessment datasets in the literature. Translating and getting generic clichés The Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (BLEnD), described in ( 19 ) , is a benchmark for assessing cultural alignment that seems to have gathered considerable traction given the attempts to translate it into other languages. This is a dataset with over 52k question-answer pairs, with some questions coupled with short-answers and some others with multiple-choice answers. It includes multiple subsets, with the respective entries written in 13 languages, that are meant to address multiple cultures, from 16 countries. The building process of BLEnD proceeds as follows: Native speakers from different countries are asked to write a few culturally-relevant question-answer pairs. In order to extend these pairs to other cultures, each question is translated into the target language and the result is edited such that the country mentioned in that question is now the country of interest; this transposed question is coupled with its answer, newly formed for this purpose. For instance, a Spanish annotator could have contributed the question ‘‘What is the most popular indoor sport in Spain?’’ 1 1 1 ¿Cuál es el deporte de interior más popular en España? , together with the respective answer. This question can then serve as a sort of template to generate similar questions for other cultures, by translating it into other languages and replacing “España” by another country name. These newly-generated questions are then answered by the annotators that are native to the culture specified in the question, and this new question-answer is entered into the benchmark. When empirically experimenting with actual LLMs on these datasets, these authors report that, as expected, while the LLMs tested tend to show good results for languages that are highly represented in the training data, performance drops