Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
1 st Amel Muminovic, 2 nd Amela Kadric Muminovic · 2025
Read Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages by 1 st Amel Muminovic, 2 nd Amela Kadric Muminovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09992.txt.
Introduction
Amel Muminovic and Amela Kadric Muminovic test language models on toxic comments in Balkan languages in a 2025 paper. Serbian, Croatian, and Bosnian have few labeled data sets, so moderation tools for them are weak. The team built and hand-labeled forty-five hundred comments from YouTube and TikTok, across topics from music and sport to politics.
Four large models were tested in two modes: cold, with no help, and with a short context snippet added. Context raised recall by about a tenth on average and lifted overall scores. It also raised false alarms at times. One model in context mode gave the best balance of scores. A rival led on precision when used cold. The authors suggest prompt design and threshold tuning as cheap gains for underserved language communities.
Catalog record
| CREATOR | 1 st Amel Muminovic, 2 nd Amela Kadric Muminovic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09992
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages 1 st Amel Muminovic, 2 nd Amela Kadric Muminovic Abstract Online toxic language causes real harm, especially in regions with limited moderation tools. In this study, we evaluate how large language models handle toxic comments in Serbian, Croatian, and Bosnian, languages with limited labeled data. We built and manually labeled a dataset of 4,500 YouTube and TikTok comments drawn from videos across diverse categories, including music, politics, sports, modeling, influencer content, discussions of sexism, and general topics. We demonstrate how adding minimal context can improve toxic language detection in low-resource settings and suggest practical strategies such as improved prompt design and threshold calibration. Four models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro, and Claude 3 Opus) were tested in two modes: zero-shot and context-augmented. We measured precision, recall, F1 score, accuracy and false positive rates. Including a short context snippet raised recall by about 0.12 on average and improved F1 score by up to 0.10, though it sometimes increased false positives. The best balance came from Gemini in context-augmented mode, reaching an F1 score of 0.82 and accuracy of 0.82, while zero-shot GPT-4.1 led on precision and had the lowest false alarms. These results show that prompt design alone can yield meaningful gains in toxicity detection for underserved Balkan language communities. I Introduction Any communication that uses insults, threats, harassment or other demeaning remarks aimed at an individual or group qualifies as toxic language. This category includes crude slurs and explicit threats as well as sarcastic jabs and demeaning jokes. We adopt the definition of toxicity from Davidson et al. (2017) [ 1 ] which defines toxic language as behavior that causes emotional or psychological harm regardless of whether it targets personal traits opinions or social identities. I-A Background & Problem Statement Toxic language remains one of the most pressing challenges for social platforms, with significant psychological and societal impacts, particularly in politically sensitive or post-conflict regions [ 2 ] . The problem is amplified in online spaces due to anonymity, lack of accountability, and the speed at which harmful content can spread. These factors create environments where users feel emboldened to express hate or harassment they might avoid in offline settings [ 3 ] . While tech companies have invested in automated moderation systems, these systems often perform well only in high-resource languages like English [ 4 ] . Most rely on keyword filters or classifiers trained on large datasets, but they fall short in low-resource languages due to limited data, cultural nuance, and evolving online language patterns [ 5 ] . For languages spoken across the Western Balkans, especially Serbian, Bosnian, and Croatian, moderation tools are less reliable, allowing toxic language and harassment to spread with limited oversight [ 6 ] . A 2021 investigation by the Balkan Investigative Reporting Network (BIRN) found that nearly half of the toxic language posts reported in Balkan languages remained online, even after Facebook and Twitter confirmed that the content violated their rules [ 7 ] . Other studies have similarly highlighted systemic moderation shortcomings across smaller linguistic communities, particularly emphasizing the limited training data and cultural context necessary for effective AI moderation of toxic language [ 8 ] . These moderation challenges in the Balkans are further complicated by historical tensions, entrenched political polarization, and specific cultural sensitivities, which significantly impact how toxicity manifests and is perceived within these communities [ 9 ] . Platforms like YouTube and TikTok play an important role in shaping online conversations in these regions [ 10 , 11 ] . However, the informal tone of comment sections, frequent code-switching between Latin and Cyrillic scripts, and the use of slang or sarcasm make harmful content harder to detect automatically [ 12 ] . Large language models (LLMs) such as GPT, Claude, and Gemini offer new moderation possibilities thanks to their generalization capabilities and multilingual support. Yet, their performance in low-resource, context-heavy environments remains underexplored [ 13 ] . This gap raises an important question about whether simple improvements, such as adding brief contextual information, could make these models more effective in practice. This paper examines how the performance of large language models changes when detecting toxic language in Balkan language comments, first using zero-shot prompts and then by adding minimal context. We test whether a short description of the video a comment responds to can help the model better interpret intent and tone. Our hypothesis is that even a small amount of context can help models distinguish between ordinary criticism and toxic remarks, especially when comments involve cultural references, sarcasm, or platform-specific slang. To our knowledge, no prior study has systematically evaluated the performance of state-of-the-art large language models for toxic language detection in Serbian, Croatian, and Bosnian, making this a notable gap in the current research landscape. I-B Objectives and Contributions The goal of this paper is to evaluate how the presence of contextual information affects the ability of large language models to detect toxic language in comments written in Balkan languages. We focus on three languages: Serbian, Croatian, and Bosnian, where reliable labeled datasets are limited and where linguistic variation makes moderation difficult. Specifically, we compare zero-shot prompts with and without additional context, such as a brief description of the video being commented on. Our main contributions are: • Introducing a manually labeled dataset of 4 500 YouTube and TikTok comments (1 500 per language), spanning a range of categories including music, politics, sports, modeling, influencers, sexism, and general topics. • Evaluating multiple versions of state-of-the-art language models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro and Claude 3 Opus), comparing their performance with and without added context, and analyzing how contextual prompts affect false negatives across languages. • Providing qualitative insights into common failure cases, including misinterpretation of slang, sarcasm, and mixed script comments, which highlight the limits of zero-shot moderation in low resource settings. II Related work II-A Toxic Language Detection Prior work on toxic language detection has focused on various machine learning approaches and challenges, especially in high-resource languages. A variety of classifiers such as Support Vector Machines (SVM), Random Forests, and deep learning architectures like Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) have been successfully applied to this problem [ 14 , 15 ] . Recent advancements in deep learning and transformer-based architectures have significantly improved toxic language detection. Transformer models, especially multilingual variants such as XLM-RoBERTa and multilingual BERT, have exhibited notable improvements due to their ability to capture contextual nuances more effectively than traditional methods [ 16 ] , [ 17 ] . However, despite their strong performance, transformer models still struggle with low-resource languages due to limited training data and the complexity of informal, regionally diverse, and culturally specific language use [ 18 ] . II-B Large Language Model-based Methods In recent years, large language models, such as GPT‑3, GPT‑4, Claude, and Gemini, have increasingly been leveraged for text classification tasks, including the identification of toxic language. These models have been particularly effective in zero‑shot and few‑shot scenarios, enabling them to generalize and perform well even with limited task‑specific annotated data [ 19 ] . Studies involving GPT‑based models have demonstrated their versatility in handling various