Towards Fairness Assessment of Dutch Hate Speech Detection

Julie Bauer, Rishabh Kaushal, Thales Bertaglia, Adriana Iamnitchi · 2025

Read Towards Fairness Assessment of Dutch Hate Speech Detection by Julie Bauer, Rishabh Kaushal, Thales Bertaglia, Adriana Iamnitchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12502.txt.

Introduction

Julie Bauer, Rishabh Kaushal, Thales Bertaglia, and Adriana Iamnitchi wrote a 2025 paper on how fair hate speech detection tools are when the language is Dutch. Most such work is in English and chases raw accuracy. Fairness toward all groups is a side note at best. This team puts fairness first.

They built a list of Dutch social group terms that fit the local scene of the country. They then made paired texts, alike except for the group named, with the help of language models and set substitution rules. The paired sets were used to fine-tune the detection tools, and the tools were graded on accuracy and on fairness scores. The tuned tools detected hate speech well and scored as fairer than the untuned ones. The team also flags how hard it is to make paired texts that read as natural Dutch.

Catalog record

CREATORJulie Bauer, Rishabh Kaushal, Thales Bertaglia, Adriana Iamnitchi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12502

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Fairness Assessment of Dutch Hate Speech Detection Julie Bauer, Rishabh Kaushal, Thales Bertaglia, Adriana Iamnitchi Abstract Numerous studies have proposed computational methods to detect hate speech online, yet most focus on the English language and emphasize model development. In this study, we evaluate the counterfactual fairness of hate speech detection models in the Dutch language, specifically examining the performance and fairness of transformer-based models. We make the following key contributions. First, we curate a list of Dutch Social Group Terms that reflect social context. Second, we generate counterfactual data for Dutch hate speech using LLMs and established strategies like Manual Group Substitution (MGS) and Sentence Log-Likelihood (SLL). Through qualitative evaluation, we highlight the challenges of generating realistic counterfactuals, particularly with Dutch grammar and contextual coherence. Third, we fine-tune baseline transformer-based models with counterfactual data and evaluate their performance in detecting hate speech. Fourth, we assess the fairness of these models using Counterfactual Token Fairness (CTF) and group fairness metrics, including equality of odds and demographic parity. Our analysis shows that models perform better in terms of hate speech detection, average counterfactual fairness and group fairness. This work addresses a significant gap in the literature on counterfactual fairness for hate speech detection in Dutch and provides practical insights and recommendations for improving both model performance and fairness. 1 Introduction Figure 1: Proposed methodology outlining key steps. SGT list is curated. Training data forms input to the four counterfactual data generation methods, namely, LLMlist, LLMdef, SLL, and MGS. BERTje model is fine-tuned with counterfactual data. The model is tested on test data and Dutch counterfactual data generated using templates. Finally, performance evaluation and fairness computations are performed. While the ease of expressing oneself on social media platforms has led to creative and meaningful interactions, it has also amplified the spread of hate speech – particularly content targeting specific groups based on ethnicity, gender, sexual orientation, and similar characteristics. To address this issue, researchers have developed numerous computational methods for detecting hate speech Alkomah and Ma 2022 ; Yin and Zubiaga 2021 ; MacAvaney et al. 2019 . However, the majority of these efforts focus on the English language. Regardless of language, a critical question remains: Are these detection models fair? In other words, do they detect hate speech targeted at all social groups with similar accuracy? If not, the models are unfair Mehrabi et al. 2021 ; Pessach and Shmueli 2022 . Unfair models risk perpetuating biases, which can exacerbate existing issues and erode users’ trust in social media platforms. This work evaluates the fairness of hate speech detection models for the Dutch language. Consider the following two sentences: S1: “All Moroccans are troublemakers.” and S2: “All Dutch are troublemakers” . A fair hate speech detection model should classify both sentences as equally hateful since they share the same structure and level of negativity. However, if the model predicts S1 to be 98% likely to be hateful but S2 only 10%, this disparity indicates unfairness. This kind of bias often arises when sensitive identity attributes, such as nationality, ethnicity, sexuality, or religion, disproportionately influence model predictions Garg et al. 2019 . This example highlights the importance of counterfactual fairness : the principle that a model’s decision should remain consistent if sensitive attributes in the input data are changed. For instance, S2 is the counterfactual version of S1 (and vice versa). If a model evaluates these sentences differently, it fails to meet the standard of counterfactual fairness Kusner et al. 2017 . This challenge is particularly critical in hate speech detection, where sensitive attributes often appear in potentially biased contexts. More formally, a model M is considered counterfactually fair if it produces the same predictions for all possible values of a sensitive attribute A . Mathematically, this can be expressed as: Pr(\hat{Y}\mid A=a_{1})=\Pr(\hat{Y}\mid A=a_{2}) , where A\in\{a_{1},a_{2}\} represents different values of the sensitive attribute. Despite progress in counterfactual fairness research for the English language, not much work has been done in European languages such as Dutch. Several studies on Dutch hate speech detection have been conducted using pre-trained language models and mainly focused on creating datasets and models for hate speech detection Caselli et al. 2021 ; Caselli and Van Der Veen 2023 ; Lemmens et al. 2021 ; Markov and Daelemans 2022 ; Markov et al. 2022 ; Ruitenbeek et al. 2022 ; Theodoridis and Caselli 2022 . This work aims to bridge the gap in the existing literature by exploring the counterfactual fairness of pre-trained models in detecting hate speech in Dutch on social media platforms. We aim to answer the following research questions: What are the methods to generate counterfactual data for the Dutch language? Are hate speech detection models in the Dutch language counterfactually fair? What is the impact of generated counterfactual data on the performance and fairness of the hate detection model in the Dutch language? Figure 1 presents the key steps in our methodology. We begin by curating a list of Dutch Social Group Terms and generating counterfactual sentences. To do this, we use Large Language Models (LLMs) and other techniques like Manual Group Substitution (MGS) and Sentence Log-Likelihood (SLL) originally proposed for English. Through qualitative evaluation, we identify challenges in generating realistic counterfactual sentences that conform to the rules of Dutch grammar. Next, we fine-tune transformer-based hate speech detection models using the generated counterfactual data. We then evaluate the performance of these models in detecting hate speech in Dutch. Finally, we assess counterfactual fairness using Counterfactual Token Fairness (CTF). We also compute group fairness metrics, namely, equality of odds and demographic parity. 2 Related Work Hate speech detection has been extensively studied MacAvaney et al. 2019 ; Bertaglia et al. 2021 ; Mullah and Zainon 2021 ; Yin and Zubiaga 2021 ; Alkomah and Ma 2022 ; Subramanian et al. 2023 ; Rawat et al. 2024 ; Gandhi et al. 2024 . Defining hate speech is inherently challenging, as it is a complex phenomenon influenced by interpretation Hietanen and Eddebo 2023 . Fortuna et al. Fortuna and Nunes 2018 proposed a comprehensive definition of hate speech: “Hate speech is language that attacks or diminishes, incites violence or hate against groups based on specific characteristics such as physical appearance, religion, descent, national or ethnic origin, sexual orientation, gender identity, or others. It can manifest in various linguistic styles, including subtle forms or even through humour.” This definition captures the diverse ways in which hateful language can be expressed. Although most of the research in this domain has focused primarily on the English language, some studies have explored hate speech detection in multiple languages Corazza et al. 2020 . Notable examples include investigations into Italian Del Vigna et al. 2017 , Danish Sigurbergsson and Derczynski 2020 , and Spanish Plaza-del Arco et al. 2021 , among others. We focus on hate speech detection in the Dutch language. Among earlier works, Tulkens et al. Tulkens et al. 2016a ; Tulkens et al. 2016b performed a dictionary-based approach for the detection of racist discourse in Dutch using automated means. Markov et al. Markov et al. 2021 explored features based on emotions and style for cross-domain hate speech detection in multiple languages including Dutch. Caselli et al. Caselli et al.

Read the complete text