The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation
Yuhang Zhou Yimin Xiao Wei Ai Ge Gao · 2025
Read The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation by Yuhang Zhou Yimin Xiao Wei Ai Ge Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00583.txt.
Introduction
Yuhang Zhou and co-authors study how emojis carry harm in online posts, in this 2025 paper. Emojis alone rarely offend, but context, sarcasm, and coded meanings can turn them into slurs or threats. The team mapped which emojis appear in which kinds of offensive posts on a major platform, and how users exploit their ambiguity.
They then built a moderation pipeline, powered by a large language model, that swaps harmful emojis for safer ones while keeping the post's meaning. Human judges found the edited posts less offensive and still faithful. Effects differed by offense type, which the authors say should guide platform rules.
Catalog record
| CREATOR | Yuhang Zhou Yimin Xiao Wei Ai Ge Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00583
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation Yuhang Zhou Yimin Xiao Wei Ai Ge Gao Abstract Social media platforms have become central to modern communication, yet they also harbor offensive content that challenges platform safety and inclusivity. While prior research has primarily focused on textual indicators of offense, the role of emojis, ubiquitous visual elements in online discourse, remains underexplored. Emojis, despite being rarely offensive in isolation, can acquire harmful meanings through symbolic associations, sarcasm, and contextual misuse. In this work, we systematically examine emoji contributions to offensive Twitter messages, analyzing their distribution across offense categories and how users exploit emoji ambiguity. To address this, we propose an LLM-powered, multi-step moderation pipeline that selectively replaces harmful emojis while preserving the tweet’s semantic intent. Human evaluations confirm our approach effectively reduces perceived offensiveness without sacrificing meaning. Our analysis also reveals heterogeneous effects across offense types, offering nuanced insights for online communication and emoji moderation. 1 Introduction Social media platforms host an incredibly diverse range of content, which is central to how people communicate online. However, due to varying degrees of content moderation and censorship policies, platforms like Twitter often become repositories for offensive language, threatening the cohesion and safety of online communities Davidson et al. 2019 . When analyzing offensive tweets, most research has focused on textual elements—explicit slurs, abusive phrases, or implicit language that reflects social biases Caselli et al. 2020 ; Zampieri et al. 2019 . In response, scholars have developed various approaches, many leveraging Large Language Models (LLMs), to detect offensive content across cultural and linguistic contexts Zhou et al. 2023a ; Deng et al. 2022a . Yet, despite these efforts, one critical aspect of online communication has been largely overlooked: the role of emojis in conveying or intensifying offensive messages. Emojis, as visual symbols, are embedded in the context of communication and carry more complex semantics than individual words. On the one hand, unlike word tokens, very few emojis directly convey offensive meanings: exceptions include emojis like (middle finger) and (pile of poop), as emojis are generally not designed with the intent to offend. On the other hand, the widespread use of emojis leads to varying interpretations based on their visual appearance. Emojis, with their symbolic representation of objects or ideas through similar shapes, can convey offensive meanings. For example, users often use the (peach) emoji to symbolize buttocks and the (droplets) emoji to symbolize sperm. Moreover, for sentiment-related emojis, one of the key characteristics of emojis is their ability to express irony or sarcasm Hu et al. 2017 . Emojis such as (upside-down face) and (rolling on the floor laughing) are often used to intensify offense by conveying a sarcastic tone. Even emojis typically associated with positive sentiment, such as (smiling face with heart-eyes), can take on an offensive meaning when used in inappropriate contexts, such as sexual harassment. Given the subtle yet potent ways in which emojis contribute to offensive communication, it is essential to systematically examine their roles and functions within online discourse. In this work, we take a first step toward that goal. We begin by identifying emojis frequently found in offensive tweets and analyzing how they relate to different types of offensive content. To deepen our understanding, we classify offensive tweets by category (e.g., personal attacks, racial slurs) and investigate which emojis are commonly used within each category. While content moderation has traditionally focused on text Zampieri et al. 2019 ; Pitsilis et al. 2018 ; Husain and Uzuner 2021 , we argue that emojis present a common jailbreaking way that users exploit, either deliberately or unintentionally, to convey offensive meaning through stereotypical associations. To address this, we design a multi-step moderation pipeline powered by large language models. Rather than rewriting entire tweets, a method that risks eliminating linguistic nuance and cultural expression, we target only the emojis likely to cause harm. The pipeline is implemented to identify emojis that are either directly related to offensive content or have the potential to evoke offense, and recommend emoji surrogates that preserve the tweet’s semantics. Human evaluations show that this pipeline effectively reduces offensiveness while maintaining the tweet’s meaning. We also analyze its heterogeneous effects across different tweet types and examine the relationship between emoji functionality and offensiveness. We summarize our contributions as follows: • We explore the relationship between emojis and offensive content in online communication, examining the roles emojis play under different offensive types. • We design and implement a multi-step LLM pipeline to better moderate offensive emojis in tweets and recommend emoji surrogates. • We conduct a human evaluation to demonstrate the effectiveness of our pipeline and analyze its heterogeneous effects across various offensive types. 2 Related Work Our work is based on two lines of existing work: emoji functionality and offensive content detection. Emoji Functionality and Interpretation Emojis, as prevalent visual elements, have attracted the interest of researchers. The semantics embedded in emojis extend beyond a single word token, giving their various functionalities such as expressing sentiments and irony, softening tones, and enhancing communication Ai et al. 2017 ; Ge 2019 ; Hu et al. 2017 ; Miller et al. 2016 ; Cramer et al. 2016 . The rich meanings and diverse functionalities of emojis make them useful in various tasks, including sentiment analysis, predicting user behavior, and increasing communication on social media Felbo et al. 2017 ; Chen et al. 2018 ; Chen et al. 2019 ; Zhou et al. 2023b ; Zhou and Ai 2022 . However, aside from emojis’ use in expressing irony or sarcasm, most studies focus on the positive effects of emojis, overlooking the fact that negative sentiments or visual symbols in emojis can also be used to offend others. In this work, we focus on the patterns of emoji usage in offensive tweets and explore how users employ emojis to convey offensive content. Offensive Content Detection The prevalence of social networks has encouraged users to develop more flexible forms of offensive behavior. In response, researchers have examined patterns of offense in online communication and developed various methods to detect and mitigate offensive content in text Davidson et al. 2019 ; Davidson et al. 2017 ; Pitsilis et al. 2018 ; Poletto et al. 2021 . With the advancement of LLMs, there is increasing interest in leveraging them for the effective detection of hate speech and other hidden performance bias Huang et al. 2023 ; Li et al. 2023 ; Zhu et al. 2023 ; Zhou et al. 2025 . Furthermore, due to their text generation capabilities, some studies have used LLMs to augment collected datasets, thus enhancing the robustness of hate speech detection models Xiao et al. 2024 ; Nghiem and Daumé III 2024 . Beyond general offense, researchers have also explored offenses of different types, including sexual Vandenbosch et al. 2015 , racial Davidson et al. 2019 , and violent offenses Zhong et al. 2019 , as each offense type tends to target different groups and exhibit specific linguistic patterns. Given the strong link between offense and culture, researchers have also explored offensive content across multiple languages, including English Pitsilis et al. 2018 , Chinese Deng et al. 2022b , Arabic Husain and