Advancing Hate Speech Detection with Transformers: Insights from the MetaHate

Santosh Chapagain, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi · 2025

Read Advancing Hate Speech Detection with Transformers: Insights from the MetaHate by Santosh Chapagain, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04913.txt.

Introduction

Santosh Chapagain, Shah Muhammad Hamdi, and Soukaina Filali Boubrahimi explore in 2025 transformer models for detecting hate speech online. Hate posts can cause real social and physical harm, and social platforms host huge volumes of them. Older deep learning models face limits with long text and parallel training.

They use a meta-collection of thirty-six data sets with more than a million social media samples. Several transformer models were tested, including BERT and GPT-2 variants. A fine-tuned ELECTRA model scored best. Error analysis shows ongoing trouble with sarcasm, coded language, and noisy labels in the data.

Catalog record

CREATORSantosh Chapagain, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Advancing Hate Speech Detection with Transformers: Insights from the MetaHate Santosh Chapagain, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi Abstract Hate speech is a widespread and harmful form of online discourse, encompassing slurs and defamatory posts that can have serious social, psychological, and sometimes physical impacts on targeted individuals and communities. As social media platforms such as X (formerly Twitter), Facebook, Instagram, Reddit, and others continue to facilitate widespread communication, they also become breeding grounds for hate speech, which has increasingly been linked to real-world hate crimes. Addressing this issue requires the development of robust automated methods to detect hate speech in diverse social media environments. Deep learning approaches, such as vanilla recurrent neural networks (RNNs), long short-term memory (LSTM), and convolutional neural networks (CNNs), have achieved good results, but are often limited by issues such as long-term dependencies and inefficient parallelization. This study represents the comprehensive exploration of transformer-based models for hate speech detection using the MetaHate dataset—a meta-collection of 36 datasets with 1.2 million social media samples. We evaluate multiple state-of-the-art transformer models, including BERT, RoBERTa, GPT-2, and ELECTRA, with fine-tuned ELECTRA achieving the highest performance (F1 score: 0.8980). We also analyze classification errors, revealing challenges with sarcasm, coded language, and label noise. This paper includes examples of harmful and offensive language used for research and illustrative purposes only. 1 Introduction Hate speech is a particularly troubling type of offensive language that has been linked to real-world hate crimes on social media platforms [ 15 ] . It can be a damaging form of online abuse, contributing to emotional distress, fostering social discrimination, and potentially inciting violence. Despite variations in definitions, there is consensus that hate speech targets individuals or groups with harmful intent, often based on race, religion, skin color, sexual identity, gender identity, ethnicity, disability, or national origin [ 1 ] . The rise of social networks and online platforms has increased global connectivity but also highlighted a troubling surge in hate speech across geographical and cultural boundaries. In recent studies, approximately 30% of the adolescents surveyed reported experiencing cyberbullying at some point in their lives. Furthermore, around 13% indicated that they had been cyberbullied within the 30 days before the survey [ 9 ] . Despite ongoing efforts by social media platforms to improve policies and detection systems, effectively addressing hate speech remains challenging due to complex and varying contexts, coded language, indirect expressions, and evolving slang [ 7 ] . The need for effective hate speech detection is critical. Advances in artificial intelligence (AI) and machine learning (ML) have shown promise in this area [ 7 ] , emphasizing the importance of capturing the context of hate speech. LLMs like BERT [ 6 ] , based on the transformer architecture [ 17 ] , represent significant progress in NLP by utilizing attention mechanisms to capture longer dependencies. These models have excelled in tasks such as text classification and sentiment analysis [ 8 ] . However, they require large datasets. In this paper, we conduct a large-scale investigation into the contextual detection of hate speech using LLMs, using the extensive MetaHate [ 16 ] dataset, which explored over 60 hate speech detection datasets and integrated 36 of them into the meta-collection. Our study presents a comprehensive evaluation of the transformer-based large language model for online hate detection. We evaluated several state-of-the-art models, including BART, ELECTRA, BERT, RoBERTa, and GPT-2 using the MetaHate dataset. Among the evaluated models, ELECTRA achieved the highest F1 score, outperforming all other baselines in hate speech classification. 2 Related Work 2.1 Social Media as a Key Resource for Hate Speech Data Social media platforms like Reddit generate vast user content and have become key venues for hate speech [ 14 ] . Their anonymity enables the spread of hate, often linked to real-world crimes [ 15 ] , and hate groups use coded symbols (e.g., "88" for "Heil Hitler") to organize [ 11 ] . Social media platforms facilitate the spread of online hate by making it easier to connect with people who share similar opinions, ideas, or interests, and by identifying potential targets based on searchable traits such as religion, gender, and race [ 18 ] . For example, Twitter allows users to easily identify people’s religion, gender, race, and political views through hashtags, profiles, and images. Social networks provide feedback mechanisms such as comments, likes, and upvotes that reinforce hateful behavior [ 18 ] . These interactions can increase the frequency and intensity of hate messages, as well as strengthen the beliefs of those who spread them [ 18 ] . Although platforms like Twitter and Facebook use human annotators to remove hateful content and encourage users to report it, these methods are labor-intensive and prone to bias. This has led to the development of automated hate speech detection using classical and deep learning algorithms. However, there is still an urgent need to develop robust models for real-time hate speech detection. Thus, the insights gained from the hate speech data on social networks are invaluable to developing more effective detection algorithms. 2.2 Machine Learning Approaches in Hate Speech Detection Research in hate speech detection on social media has evolved from traditional machine learning methods like Support Vector Machines (SVM) and Logistic Regression to recent deep learning approaches such as LSTMs, CNNs, and more recently, transformer-based models like BERT [ 6 ] . Warner and Hirschberg were among the first to apply supervised machine learning techniques to hate speech detection using SVM [ 19 ] . Later, Davidson et al. [ 5 ] expanded the use of algorithms in this domain with Logistic Regression, Naïve Bayes, Decision Trees, Random Forests, and linear SVMs. Naïve Bayes and K-NN classifiers have also proven effective, as demonstrated by Yan et al. [ 20 ] and Yin et al. [ 21 ] , who combined these classifiers with techniques like bag-of-words and term frequency-inverse document frequency (TF-IDF). Kumari et al. [ 10 ] proposed an LSTM-based approach for detecting hate speech on Twitter, demonstrating superior performance compared to shallow learning methods. Although deep learning and word embeddings-based models have shown promising performance, they could not match the capabilities of modern transformer-based models like BERT [ 6 ] and RoBERTa [ 12 ] . Mittal et al. [ 13 ] demonstrated that BERT-based models outperform traditional techniques by effectively capturing complex semantics and adapting to various linguistic contexts. LLMs has achieved remarkable performance in hate speech detection and other NLP tasks [ 8 , 3 , 2 , 4 ] In this paper, we explore various transformer-based large language models, comparing the performance of fine-tuned transformers on the MetaHate [ 16 ] dataset. Our study finds that ELECTRA outperforms other transformers on the MetaHate [ 16 ] dataset. 3 Big Data and Label The creators of the MetaHate [ 16 ] dataset conducted a thorough review of over 60 existing hate speech detection datasets and ultimately selected 36 of them to include in their meta-collection. This process resulted in 1,667,496 entries, with 1,226,202 being unique comments sourced from social media platforms like Twitter, Facebook, and Reddit. The dataset excludes synthetic data and comments from non-social media sources. The dataset is binary-classified into hate (label=1) and non-hate

Read the complete text