Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)
Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J. Gomez, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti · 2025
Read Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES) by Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J. Gomez, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24958.txt.
Introduction
Guido Ivetta and seven co-authors built LACES, a set of shared views on groups in Latin America, drawn from folks who live there. Bias tests for language tools lack data from this part of the world, so harms there go un gauged. Their set spans fifteen lands in the region.
Near five thousand group views were made and marked by hand, from more than eighty folks in joint work with local teams. Their new build flow folds fresh views and checks of old views into one pass, which yielded more sole views than past fixed flows. Tests show past bias fix tools work less well on this set. That marks the size of the geo gap.
Catalog record
| CREATOR | Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J. Gomez, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24958
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES) Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J. Gomez, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti Abstract The evaluation of societal biases in NLP models is critically hindered by a glaring geo-cultural gap, as existing benchmarks are overwhelmingly English-centric and focused on U.S. demographics. This leaves regions such as Latin America severely underserved, making it impossible to adequately assess or mitigate the perpetuation of harmful regional stereotypes by language technologies. To address this gap, we introduce a new, large-scale dataset of stereotypes developed through targeted community partnerships across Latin America. Additionally, we present a novel adaptive data collection methodology that uniquely integrates the sourcing of new stereotype entries and the validation of existing data within a single, unified workflow. This combined approach results in a resource with significantly broader coverage and greater regional nuance than static collection methods, enabling a more robust stereotype evaluation and significantly addressing the geo-cultural deficit in fairness resources for Latin America. Furthermore, we believe that our adaptive data collection method could facilitate a more effective, community-engaged approach to capturing a broader spectrum of sociocultural knowledge. 1 Introduction While mitigating the social biases that NLP models learn from web-scale training data continues to remain a challenge, the resources that we rely on to evaluate these harms are themselves critically biased. A growing body of research demonstrates that NLP models tend to perpetuate and sometimes amplify social stereotypes Bolukbasi et al. 2016 ; Garg et al. 2018 , underscoring the need for robust stereotype evaluation datasets; yet, the field is dominated by resources that are overwhelmingly English-centric and focused on U.S. demographics. While NLP researchers have brought attention to these gaps in recent years Prabhakaran et al. 2022 , glaring gaps continue to persist in the global coverage of evaluation resources, especially in Latin America. Figure 1: This dataset covers 4789 stereotypes, covering 120 identities and 842 attributes. It was built by 83 annotators from 15 distinct countries. The map illustrates the participating Latin American nations. The methodology of collection adapts to the participant identity by bringing examples relevant to their nation of origin. While recent research has begun to address these gaps in evaluation data by curating candidate stereotypes from large language models themselves Bhutani et al. 2024 ; Jha et al. 2023 , this approach anchors primarily on the data and sociocultural knowledge captured in the training set and fails to include more nuanced and broader community insights. In contrast, there is also recent complementary work that engages with a larger set of participants (e.g., Dev et al. 2023 ; Maina et al. 2024 ; Mitchell et al. 2025 ; Ivetta et al. 2025 ) to collect a broader set of stereotypes. Our work is in line with these efforts, with a focus on sourcing stereotypes across countries in Latin America. These community-sourced data collection approaches tend to be static and non-adaptive — i.e., they are agnostic to the existing state of knowledge, offering no mechanism to dynamically assess their own comprehensiveness or coverage. Consequently, when sampling data from a specific locale, we cannot systematically identify critical gaps or data sparsity. This leaves datasets often containing redundant information about the most common cultural knowledge, while the long-tail information is often missed. The resulting datasets will be incomplete in unknown ways and provides no guarantee that we have successfully captured the necessary spectrum of cultural knowledge. This “blind sampling” methodology creates a second, concurrent failure: profound resource inefficiency due to data redundancy. Lacking a mechanism to track known information, collection efforts repeatedly oversample the high-frequency “head” of the cultural knowledge distribution—typically the most salient, common-knowledge concepts. It not only wastes resources by re-annotating the known “head” but also fails to capture the diverse, nuanced knowledge in the “long tail.” Critically, this ad-hoc redundancy is currently treated as a mere annotation artifact and discarded, rather than as a rich sociolinguistic signal. Systematically captured repetition, for instance, serves as a powerful proxy for saliency (or prevalence) and pluralism (contested perspectives). In this paper, we describe a novel adaptive data collection approach that we utilized for collecting stereotype data across 15 countries in Latin America, collected over two separate initiatives. We demonstrate how our adaptive approach results in reduced redundancy, while improving coverage above and beyond what a traditional data collection approach would have yielded. Our resulting dataset has 4789 stereotypes, covering 120 identities and 842 attributes across a multitude of different sociodemographic axes, as annotated by 83 individuals from 15 distinct countries. We also demonstrate critical gaps in stereotype evaluations that our dataset reveals, and that the existing mitigation approaches crucially overlooks these stereotypes. 2 Related Work In this section we first review previous work on stereotype dataset creation and explore how datasets are not neutral repositories of linguistic knowledge but are shaped by the choices of stakeholders involved. Then we review how examples can shape the data collected particularly in the creation of stereotype datasets. Finally we describe methods that have been used to dynamically adapt the examples so that they are meaningful and relevant for the annotators. Existing stereotype datasets Several benchmark datasets have been proposed to quantify and analyze stereotypes encoded in language models. Early efforts such as CrowS-Pairs ( Nangia et al. 2020 ) and StereoSet ( Nadeem et al. 2021 ) focused on English and U.S.-centric contexts, measuring stereotypical bias across domains like race, religion, gender, and profession. BBQ ( Parrish et al. 2022 ) extended this line of work to question answering, showing that models tend to rely on stereotypes when contextual information is under-specified. More recent resources broaden geographic and linguistic scope. SeeGULL ( Jha et al. 2023 ) used language models to propose stereotypes from different regions and validated them with globally diverse annotators to construct a dataset covering stereotypes from 178 countries across six continents, while SHADES ( Mitchell et al. 2025 ) provided a multilingual parallel dataset spanning 16 languages and 20 regions. Complementary work has emphasized participatory and community-driven approaches: Dev et al. ( Dev et al. 2023 ) engaged with Indian communities to surface locally grounded stereotypes absent from Western-centric benchmarks. Ivetta et al. Ivetta et al. 2025 is a similar effort that engaged Latin American communities. These newer datasets highlight the need for socio-culturally inclusive stereotype evaluation. However, all of them use fixed examples to elicit stereotypes instead of adapting the examples to the social group of the annotator, as we do in this paper. Datasets as shaped artifacts Different annotators will not necessarily assign the same labels to the same texts, resulting in human label variation Plank 2022 . There is evidence that this variation depends on the demographic characteristics of annotators ( Binns et al. 2017 ; Al Kuwatly et al. 2020 ; Excell and Al Moubayed 2021 ; Shen and Rose 2021 ) . Variation is stronger for subjective tasks like toxic content detection ( Sap et al. 2019 ; Kumar et al. 2021 ; Sap et al. 2022 ; Goyal et al. 2022 ) and stereotype elicitation. Annotation guidelines are known to influence the resulting data obtained. In particular, for concepts such as safety and offensiveness Davani et al. 2024 ; Aroyo