Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

Lance Calvin Gamboa, Mark Lee · 2024

Read Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia by Lance Calvin Gamboa, Mark Lee (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07303.txt.

Introduction

Lance Gamboa and Mark Lee build Filipino bias benchmarks for language models in a 2024 paper. Bias tests exist for rich-resource tongues, but Filipino is low-resource and understudied. The pair adapt two English test sets to Filipino culture. Their sets cover sexist bias and anti-queer bias side by side.

The work yields over seven thousand challenge pairs, with the adaptation steps documented in full. Masked and causal multilingual models were put to the test. All showed considerable bias. Models fed more Filipino text in training carried distinct bias levels for the tongue. The sets give future work a base for fairer Southeast Asian models.

Catalog record

CREATORLance Calvin Gamboa, Mark Lee
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07303

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia Lance Calvin Gamboa, Mark Lee Abstract Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets—a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models. 1 Introduction Despite the rapid evolution of PLMs and efforts to minimize their social harms ( OpenAI et al. 2023 ; Meta 2024 ) , recent studies still confirm the presence of biases within them ( Liu et al. 2024 ; Felkner et al. 2023 ; Steinborn et al. 2022 ) . AI fairness, therefore, remains to be a critical area of focus for the research community, which bears an ethical responsibility to mitigate the potential negative impacts of the technologies it builds ( Talat et al. 2022 ; Amershi et al. 2020 ; Hovy and Spruit 2016 ) . Scholars have developed bias evaluation benchmarks to not only establish baselines quantifying biased behavior exhibited by off-the-shelf PLMs, but also to measure the effectiveness of bias mitigation techniques applied on these models ( Reusens et al. 2023 ; Blodgett et al. 2021 ; Nangia et al. 2020 ) . Most bias studies in the literature, however, use only English benchmarks to assess monolingual PLMs ( Goldfarb-Tarrant et al. 2023 ) . Only a few recent exceptions have emerged to examine bias in multilingual PLMs using datasets written in other languages—i.e., French ( Névéol et al. 2022 ; Reusens et al. 2023 ) , German ( Steinborn et al. 2022 ; Reusens et al. 2023 ) , Dutch ( Reusens et al. 2023 ) Finnish, Thai, and Indonesian ( Steinborn et al. 2022 ) . Among these multilingual studies of bias, the benchmarks used often treat gender as a binary construct and do not thoroughly investigate biases against non-heterosexual identities ( Goldfarb-Tarrant et al. 2023 ; Tomasev et al. 2021 ) . There is thus an absence of non-English homophobic bias evaluation benchmarks that can catalyze work in evaluating and mitigating anti-queer bias in PLMs deployed in non-English-speaking contexts. In this paper, we address this gap by adapting two bias benchmarks—Crowdsourced Stereotype Pairs or CrowS-Pairs ( Nangia et al. 2020 ) and WinoQueer ( Felkner et al. 2023 ) —for Filipino, a language that currently does not have high NLP resources ( Joshi et al. 2020 ) . CrowS-Pairs is a dataset widely used to probe PLMs for different stereotypes (e.g., race, gender, religion, age, etc.), while WinoQueer is a recently released benchmark designed to assess the extent of anti-LGBTQ+ bias encoded in PLMs. Designing Filipino versions of these English materials is valuable for two reasons. First, the English and Filipino languages do not share the same linguistic and grammatical gender mechanisms ( Santiago and Tiangco 2003 ; Santiago 1996 ; Demond 1935 ) , nor do concepts of queerness and non-heterosexuality in their corresponding cultures completely overlap ( Cardozo 2104 ; Garcia 1996 ) . Our method for culturally adapting CrowS-Pairs and WinoQueer into Filipino elucidates how generalizable these benchmarks are to low-resource languages and what considerations and challenges need to be accounted for in translating them. Our corpus development procedure can serve as a template or guide for future endeavors creating bias benchmarks in other languages. Second, the integration of AI into the Southeast Asian landscape is growing. Reports highlight both the rapid uptake of language-based AI technologies in the area ( Sarkar 2023 ; Navarro 2024 ) and local NLP practitioners’ deployment of PLMs trained with higher proportions of Southeast Asian textual data ( Zhang et al. 2024 ; AI Singapore 2023 ; Maria 2024 ) . Designing contextually appropriate bias benchmarks in Southeast Asian languages—especially Filipino, which has 83 million speakers ( Eberhard et al. 2023 ) —stands as a crucial first step in mitigating the societal harms of such PLMs used in the region. We demonstrate our Filipino benchmarks’ ability to contribute to this regard by evaluating both sexist and homophonic bias in off-the shelf multilingual PLMs, including causal ones specifically developed for the Southeast Asian context. To the best of our knowledge, we are the first to use non-English benchmarks in assessing causal and Southeast Asian models. Our work can thus serve as a baseline for future work aiming to reduce bias in such models. Our contributions are threefold: • We provide insights on the cultural generalizability of existing bias benchmarks and propose solutions to challenges in extending these datasets to a low-resource language like Filipino. • We release Filipino CrowS-Pairs and Filipino WinoQueer, adding 7,074 new Filipino entries to the pool of multilingual bias evaluation datasets existing in the literature. 1 1 1 Available at https://github.com/gamboalance/filipino_bias_benchmarks • We use Filipino CrowS-Pairs and Filipino WinoQueer to establish baseline bias evaluation results for off-the-shelf multilingual PLMs, including causal ones and those from Southeast Asia. The remainder of this paper is structured as follows. Section 2 first provides a background on the research areas to which we contribute: bias evaluation and its implementation in multilingual and Filipino contexts. Next, Section 3 describes our corpus development method, including a discussion of the issues we encountered in translating CrowS-Pairs and WinoQueer and our solutions for addressing these. Section 4 then discusses our use of the newly curated Filipino benchmarks to probe off-the-shelf PLMs for sexist and homophobic bias. Finally, Section 5 concludes the paper with a summary while Section 6 details our work’s limitations and ethical considerations. 2 Related Work 2.1 Bias Evaluation An extensive body of research explores the identification and quantification of bias in language models ( Talat et al. 2022 ; Goldfarb-Tarrant et al. 2023 ) . Initial work in the field relied on word sets to characterize bias in word embeddings. For example, ( Caliskan et al. 2017 ) found that in word2vec and GloVe, vectors of science-related words are more associated with male word vectors than female word vectors because these static models learned gender biases from their pretraining data. The rise of Transformer-based models, however, caused a shift from using word sets to relying on prompt and template sets to measure PLM bias ( Blodgett et al. 2021 ) . Kurita et al. 2019 were among the first to develop a prompt-based bias evaluation dataset for BERT. The benchmark consisted of artificially constructed templates like <MASK> is a programmer. These templates were given to BERT as inputs to

Read the complete text