An Annotated Corpus of Arabic Tweets for Hate Speech Analysis

Wajdi Zaghouani, Md. Rafiul Biswas · 2025

Read An Annotated Corpus of Arabic Tweets for Hate Speech Analysis by Wajdi Zaghouani, Md. Rafiul Biswas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11969.txt.

Introduction

Hate speech in Arabic hides inside rich dialect variety, which defeats tools trained on one standard form. Wajdi Zaghouani and Rafiul Biswas built a labeled data set to fight back in this 2025 paper. Ten thousand Arabic tweets were collected and read by several annotators. Each tweet was judged offensive or not.

Offensive tweets were further tagged by target: religion, gender, politics, ethnicity, origin, or others. One tweet can carry several target tags at once. Agreement between annotators scored high for offensiveness and solidly for targets. Standard language models were then benchmarked on the set, with an Arabic-tuned model on top. Shared data like this anchors fair comparisons of moderation tools.

Catalog record

CREATORWajdi Zaghouani, Md. Rafiul Biswas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11969

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Annotated Corpus of Arabic Tweets for Hate Speech Analysis Wajdi Zaghouani, Md. Rafiul Biswas Abstract Identifying hate speech content in the Arabic language is challenging due to the rich quality of dialectal variations. This study introduces a multilabel hate speech dataset in the Arabic language. We have collected 10000 Arabic tweets and annotated each tweet, whether it contains offensive content or not. If a text contains offensive content, we further classify it into different hate speech targets such as religion, gender, politics, ethnicity, origin, and others. A text can contain either single or multiple targets. Multiple annotators are involved in the data annotation task. We calculated the inter-annotator agreement, which was reported to be 0.86 for offensive content and 0.71 for multiple hate speech targets. Finally, we evaluated the data annotation task by employing a different transformers-based model in which AraBERTv2 outperformed with a micro-F1 score of 0.7865 and an accuracy of 0.786. Introduction Identifying hate speech has become a critical area of research due to the rise of online platforms where offensive and harmful content can spread rapidly. The engagement of communities in digital platforms is increasing day by day, and so it increases the risk of individuals being affected by the negativity of hate speech content ( Costello and Hawdon 2020 ) . Therefore, the development of an effective computational tool is required to monitor and mitigate such content. While significant advancements have been made in hate speech detection for English and other widely spoken languages, addressing this issue in the Arabic language remains particularly challenging due to the language’s complexity and diversity ( Alhazmi et al. 2024a ) . Arabic is a rich language with numerous dialectal variations across different regions, making natural language processing (NLP) tasks, including hate speech detection, more difficult than in languages with more standard forms ( Alhazmi et al. 2024b ) . Moreover, Arabic dialects are highly informal in nature, and they vary significantly from Modern Standard Arabic (MSA), adding layers of complexity when developing NLP models ( Abdelsamie, Azab, and Hefny 2024 ; Al-Ibrahim, Ali, and Najadat 2023 ) . Many of the available Arabic datasets focus on formal Arabic or a specific dialect, which limits the generalizability of hate speech detection models. In addition to linguistic diversity, the low-resource nature of Arabic dialects also poses a significant challenge ( Abdul-Nabi, Obeidat, and Bsoul 2024 ) . Many Arabic NLP tasks suffer from a scarcity of annotated datasets, especially tasks like hate speech detection. While hate speech detection in English has benefited from large-scale datasets and well-developed models, similar efforts in Arabic are comparatively limited ( Alkomah and Ma 2022 ; Kansok-Dusche et al. 2023 ) . The multilabel nature of hate speech—where a text can target multiple identities such as religion, gender, or ethnicity—adds further complexity, making it essential to develop datasets that capture the breadth of hate speech in different forms ( Zaghouani, Mubarak, and Biswas 2024 ) . This study addresses these challenges by introducing a novel multilabel hate speech dataset in Arabic, specifically designed to encompass multiple hate speech targets such as religion, gender, politics, ethnicity, and origin. By annotating a large corpus of Arabic tweets, we aim to provide a resource that can enable more accurate and nuanced detection of hate speech across different target categories. Our dataset is a significant contribution to the Arabic NLP field, providing an essential resource for future research on hate speech detection in low-resource and dialectally diverse languages. 1 Novelty: The dataset represents a unique, large-scale, annotated corpus of Arabic tweets specifically designed for multilabel hate speech analysis. It addresses the paucity of resources in this domain and provides a foundational dataset for the training and evaluation of machine learning models. 2 Availability: The annotated dataset and guidelines are made freely accessible to the research community upon request. The dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license, which allows for free use, distribution, and adaptation, provided that the original work is properly attributed. This open access ensures that the majority of academic and industry researchers can access and utilize the resource. 3. Utility: The dataset and guidelines are well-documented, with clear instructions for accessing and using the resource. The annotation process, which involves diverse annotators from various Arab countries, is described in detail, ensuring the representativeness and accuracy of the dataset. The corpus can be employed to train and evaluate various machine learning models of hate speech detection in Arabic social media content. ( Kakulapati, Reddy, and Paramasivam 2023 ; Xian-Yi et al. 2010 ; Gaind, Syal, and Padgalwar 2019 ; Pereira-Kohatsu et al. 2019 ) . Related Works Numerous research efforts have been conducted to develop and examining Arabic corpora for various NLP applications. However, a few of them focused on the collection of hate speech datasets in Arabic. We will categorize the related works into two categories: i) collection of Arabic hate speech corpus and ii) techniques to detect Arabic hate speech. Collection of Arabic Hate Speech Corpus : Table 1 describes a recent dataset on Arabic hate speech collections. Mulki et al. 2019 collected Arabic Twitter data between March 2018 and February 2019. They collected 5846 tweets containing three labels: normal, abusive, and hate. The search keywords contain refugee, female, Arabs, Druze, tweets belonging to the post, and replies from politicians, feminists, and activists. ( Albadi, Kurdi, and Mishra 2018 ) collected 9,316 tweets classified into three labels: hateful, abusive, or normal. The search keywords contain racist, religious, and ideological hate speech, and the search spans from March 2018 to August 2018. ( Zaghouani, Mubarak, and Biswas 2024 ) collected tweets between between August 12, 2020, and October 4, 2020. It covers diverse contents focusing on emotion, emotion intensity, offensive content, fact-checing and so on. ( Alshaalan and Al-Khalifa 2020 ) collected 3075 tweets in 2018 on religion, ethnicity, nationality, and gender. The tweet Id belongs to gulf countries. Khezzar, Moursi, and Al Aghbari 2023 proposed the arHateDataset dataset, which comprises 34,107 tweets obtained from different Arabic standards and dialectals. However, this dataset is not a unique corpus from the author. Rather, they combined seven Arabic datasets together to form this dataset. Dataset Keywords Multilabel Size Platforms Dialect Model ( Mulki et al. 2019 ) Refugee, female, Arabs No 5,846 Twitter Syrian,Lebanese SVM, NB ( Albadi, Kurdi, and Mishra 2018 ) Racist, religious, tribes No 9,316 Twitter Saudi SVM,CNN, RNN, BERT ( Zaghouani, Mubarak, and Biswas 2024 ) Variety of topics Yes 15,965 Twitter Across Arab countries LR, SVM, AraBERT ( Alsafari, Sadaoui, and Mouhoub 2020 ) Religion, ethnicity, gender No 3,075 Twitter Gulf countries LR, CNN, mBERT ( Khezzar, Moursi, and Al Aghbari 2023 ) Varieties No 34,107 Twitter Multi NB, CNN, AraBERT Table 1: Comparison of Arabic Hate Speech Datasets Techniques to Detect Arabic Hate Speech: Table 1 demonstrates the uses of different model to detect offensive content. ( Mulki et al. 2019 ) used traditional machine learning techiniques Support Vector Machine (SVM) and Naïve Bayes (NB) to identify tweets wheather a tweet is hateful or not. ( Albadi, Kurdi, and Mishra 2018 ) applied SVM and Logistic Regression (LR) techqniue as traditional machine learning model. For deep

Read the complete text