Silencing Empowerment, Allowing Bigotry : Auditing the Moderation of Hate Speech on Twitch

Prarabdh Shukla, Wei Yin Chong, Yash Patel, Brennan Schaffner, Danish Pruthi, Arjun Bhagoji · 2025

Read Silencing Empowerment, Allowing Bigotry : Auditing the Moderation of Hate Speech on Twitch by Prarabdh Shukla, Wei Yin Chong, Yash Patel, Brennan Schaffner, Danish Pruthi, Arjun Bhagoji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07667.txt.

Introduction

Prarabdh Shukla and five co-authors audit Twitch's automated chat filter and find it misses most hate while blocking teaching content, in a study from 2025. Live chat moves too fast for human moderators, so platforms lean on automation whose real accuracy is rarely tested. The team built isolated test streams to probe it.

Over one hundred seven thousand comments from four datasets were sent through the filter. Up to ninety-four percent of plainly hateful messages slipped through on some sets. Adding slurs to those messages caused total removal, exposing a slur crutch. Meanwhile up to about ninety percent of benign educational uses of sensitive words were blocked.

Catalog record

CREATORPrarabdh Shukla, Wei Yin Chong, Yash Patel, Brennan Schaffner, Danish Pruthi, Arjun Bhagoji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Silencing Empowerment, Allowing Bigotry : Auditing the Moderation of Hate Speech on Twitch Prarabdh Shukla, Wei Yin Chong, Yash Patel, Brennan Schaffner, Danish Pruthi, Arjun Bhagoji Abstract Warning : This paper contains content that may be offensive or upsetting. To meet the demands of content moderation, online platforms have resorted to automated systems. Newer forms of real-time engagement ( e.g. , users commenting on live streams) on platforms like Twitch exert additional pressures on the latency expected of such moderation systems. Despite their prevalence, relatively little is known about the effectiveness of these systems. In this paper, we conduct an audit of Twitch’s automated moderation tool ( AutoMod ) to investigate its effectiveness in flagging hateful content. For our audit, we create streaming accounts to act as siloed test beds, and interface with the live chat using Twitch’s APIs to send over 107,000 107,000 comments collated from 4 4 datasets. We measure AutoMod ’s accuracy in flagging blatantly hateful content containing misogyny, racism, ableism and homophobia. Our experiments reveal that a large fraction of hateful messages, up to 94 % 94\% on some datasets, bypass moderation . Contextual addition of slurs to these messages results in 100 % 100\% removal, revealing AutoMod ’s reliance on slurs as a moderation signal. We also find that contrary to Twitch’s community guidelines, AutoMod blocks up to 89.5 % 89.5\% of benign examples that use sensitive words in pedagogical or empowering contexts. Overall, our audit points to large gaps in AutoMod ’s capabilities and underscores the importance for such systems to understand context effectively. 1 1 1 Our code and data can be found at https://github.com/weiyinc11/HateSpeechModerationTwitch 1 Introduction Figure 1: The audit pipeline : ( 1 ) Setting up bots to interact with AutoMod and data collation, ( 2 ) Recording moderation decisions at scale, ( 3 ) Analysis of AutoMod moderation decisions. ( 2.1 ) shows actual instances of moderated and non-moderated messages from our experiments in the Twitch interface. The moderation decisions clearly show the limitations of AutoMod . For any online platform to exist without being overrun by hateful, pornographic, abusive, misogynistic and violent content, it must moderate what its users post Gillespie 2018 . Barring certain regions Bundesamt für Justiz 2022 , there are few or no legal regulations dictating what is acceptable content on platforms Schaffner et al. 2024 . Moreover, in the Global North, platforms often enjoy legal protections from liability for hosting user-generated content 47 U.S.C. § 230 1996 ; EU 2000 . This favorable regulatory framework bestows upon platforms the discretion to moderate content as they wish. The discourse stands divided on the benefits of such freedom, with some crediting it for aiding the growth of online platforms Kosseff 2019 , while others critique it for enabling the proliferation of harmful content Wakabayashi 2019 . Platforms codify the behavior expected of their users through terms of service and community guidelines. While policies and guidelines may considerably vary across platforms, most promise their users a safe space, free from online harm. For instance, the community guidelines of Twitch, a video streaming platform, state: “Twitch does not permit behavior that is motivated by hatred, prejudice or intolerance, including behavior that promotes or encourages discrimination, denigration, harassment, or violence based on the following protected characteristics: race, ethnicity, color,… We also provide certain protections for age.” The rhetoric notwithstanding, the pressures exerted on moderation systems have never been higher. Platforms have increasingly begun to integrate automated, usually machine learning-based, systems in their moderation pipelines Gorwa et al. 2020 due to the scale and velocity of content posted as well as the demanding latency expectations. 2 2 2 From April 2022 to April 2023, social media platforms saw nearly 150 million new users ( 4.7 /second) Nyst 2023 . The practice of content moderation is further challenged by balancing the competing objectives of blocking broadly undesirable content and upholding users’ freedom of expression. In response to these pressures, several platforms release aggregate data about the state of harmful content and corresponding actions taken Twitch 2024b ; X 2024 ; Meta 2024 . Despite such transparency efforts, little is known about the underlying algorithms used for moderation and the biases they may introduce. In this work, we conduct an audit of Twitch’s content moderation system. Twitch is a digital platform designed primarily for live streaming content which is created in channels or “streams" that visitors can watch and interact with through a text-based live chat. We focus on hate speech content as it is socially important and easier to define than other harmful content such as misinformation Schaffner et al. 2024 . We identify Twitch as a fertile platform for auditing due to three key advantages it offers: (i) Twitch is widely used TwitchTracker 2025 ; (ii) streams on Twitch can be “siloed,” allowing us to set up controlled experiments where only the research team can view the content to be tested (iii) the platform provides streamers with a suite of machine learning-based moderation tools collectively called AutoMod Twitch 2024a , which provides streamers control over different categories of harmful content. Within each category, it offers options to block harmful content targeted at different identity groups, with knobs to vary the extent of moderation ( Section 3.1 ). Through this audit, we seek to answer the following key research questions ( Section 4 ): (i) How effective is AutoMod at flagging commentary rife with hate?; (ii) How specific and effective are individual filters at blocking hate of different kinds and intents?; and (iii) Are moderation rates consistent across different target groups, or are certain groups disproportionately affected? We use case-studies ( Section 5 ) to nuance these by asking (i) how reliant are the moderation filters on explicit slurs; (ii) how do they respond to sensitive but pedagogical content about marginalized groups?; and (ii) how far can a filter-aware malicious actor bring down their performance? To answer such questions, we develop a framework that allows us to stress test AutoMod at scale by launching chatbots in a siloed sandbox. For our study, we use four datasets – a real world comment dataset (SBIC), a real world implicit hate dataset (IHC), a synthetic dataset on implicit hate (ToxiGen) and a synthetic dataset designed to fool hate classifiers (DynaHate). We discuss the hate speech datasets used in Section 3.2 and our experimental setup in Section 4.1 . Our audit reveals that Twitch’s automated moderation is far from adequate ( Section 4.2 ), flagging only 22\% of hateful content even at its most stringent setting. We observe that hateful samples concerning Race, Ethnicity and Religion are least flagged, with only 12.3\% of hateful examples being caught. Further, on some datasets, we find that an overwhelming 98\% of hate targeted at Mentally Disabled folks escapes moderation. On the two implicit hate datasets we use, AutoMod is only able to flag 6.8\% of hateful examples, implying that the filters rely heavily on slur-based moderation and miss out on implicit hate. Worryingly, empowering or positive phrases about communities ( e.g. Figure 1 ) are flagged, and we find AutoMod to be quite brittle to semantic-preserving perturbations (See Section 5 ). While it is undeniable that Twitch provides powerful,

Read the complete text