How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models
Subhojit Ghimire · 2025
Read How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models by Subhojit Ghimire (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06676.txt.
Introduction
Subhojit Ghimire measures dialect bias in a popular toxicity filter and builds a teaching tool that makes the bias tangible. Automated moderation now scans everyday speech, and a post flagged as toxic may simply be written in a dialect the model misunderstands. The study compares scores for African-American English against Standard American English.
On average the model rated the African-American English text about one point eight times more toxic and nearly nine times higher for identity hate, a systematic skew. The interactive tool lets users drag a sensitivity threshold and see the real lever of harm. The harm is not just biased scores, but the neutral-looking policy cutoff a human sets, which turns skew into silencing.
Catalog record
| CREATOR | Subhojit Ghimire |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How AI Fails: An Interactive Pedagogical Tool for Demonstrating Dialectal Bias in Automated Toxicity Models Subhojit Ghimire Abstract Now that AI-driven moderation has become pervasive in everyday life, we often hear claims that “the AI is biased.” While this is often said jokingly, the light-hearted remark reflects a deeper concern. How can we be certain that an online post flagged as “inappropriate” was not simply the victim of a biased algorithm? This paper investigates this problem using a dual approach. First, I conduct a quantitative benchmark of a widely used toxicity model (unitary/toxic-bert) to measure performance disparity between text in African-American English (AAE) and Standard American English (SAE). The benchmark reveals a clear, systematic bias: on average, the model scores AAE text as 1.8 times more toxic and 8.8 times higher for “identity hate.” Second, I introduce an interactive pedagogical tool that makes these abstract biases tangible. The tool’s core mechanic, a user-controlled sensitivity threshold, demonstrates that the biased score itself is not the only harm; instead, the more concerning harm is the human-set, seemingly neutral policy that ultimately operationalises discrimination. This work provides both statistical evidence of disparate impact and a public-facing tool designed to foster critical AI literacy. I Introduction Online content moderation tools extensively use automated toxicity models to evaluate content and decide whether to flag it as inappropriate [ 1 ] . This practice is not new, but it is becoming more prevalent today with the introduction of AI, where automated tools are now publicised as AI systems. Even though there is nothing particularly intelligent about these systems, which is why I prefer calling them just “algorithms”, the AI namesake has managed to pique the public’s interest. Even the general population has begun using and interacting with these “AI”-moderation tools. In one sense, this is a good thing. If more people are aware of a certain tool, more people will try to understand its intricacies and begin to question and dissect it. This public auditing, in itself, can ensure that any hidden malpractice is brought to light and that appropriate actions can be taken to correct it, preventing further harm. This is especially crucial in systems that operate as “black boxes,” raising concerns about their fairness and real-world impact. When an AI model’s decision-making is opaque, it becomes difficult to challenge or even question its outcomes [ 2 ] . The arbitrary “black box” problem should not be dismissed as a mere technical inconvenience, especially when the stakes in a critical society are high. If a model tasked with detecting “toxic” language contains hidden, systematic biases, it will not simply produce random errors but may systematically silence the voices of minority groups [ 3 ] . An AI model is trained on a vast dataset, so it learns to associate linguistic features and dialects with the demographic groups represented in that dataset [ 4 ] . But is the dataset used to train the model fair? Does it contain equal representation of dialects across demographics? Or is it biased towards one majority group? If so, the result is a disparate impact, wherein a system, under the guise of safety and moderation, disproportionately flags benign text from one community while ignoring harmful text from another [ 5 ] . Many thorough studies have been conducted on this topic, and while some expert tools exist to visualise this problem, they are neither readily available nor designed as pedagogical instruments for a non-expert audience. This has created a significant gap between expert findings and public understanding. It is one thing to read a statistical report on bias; it is another to see and interact with that bias in real time. This paper aims to bridge that gap. I present a dual-pronged contribution to investigate and demonstrate this problem. First, I conduct a quantitative audit of a foundational, widely used toxicity model unitary/toxic-bert [ 6 ] to scientifically measure its performance disparity on African-American English (AAE) versus Standard American English (SAE). Then, I introduce a novel interactive pedagogical tool that I developed, designed explicitly to make the bias tangible. This tool displays scores for various sentiment categories after analysing the sentiment of an input text and allows users to adjust a sensitivity threshold slider. This interaction is designed to provide a key insight, as users realise that this slider represents not a neutral, model-determined boundary, but a human-set policy, which ultimately leads to discriminatory outcomes. II Related Work My research is situated at the intersection of two fields: 1. The quantitative study of algorithmic bias in NLP models and, 2. the Human-Computer Interaction (HCI) challenge of making AI understandable to the public. This section reviews work in both areas to establish the context for my contribution. Blodgett et al., in their foundational study, demonstrated that language identification models often misidentify AAE [ 4 ] . They provided a large-scale Twitter corpus, the same one I use in this paper, to help researchers investigate these disparities [ 7 ] . Their work showed that demographic dialectal variation is a significant confounding variable for many NLP tasks. Building on this, Sap et al. analysed several large-scale datasets used for hate speech detection [ 8 ] . They presented a critical finding that all of them contained a strong, spurious correlation between AAE linguistic features and “toxicity”, meaning that models trained on this data learned to associate AAE with abusive language, even when the context was benign. Parikh et al. further established that this systematic, data-driven bias is not a minor flaw but a significant, measurable problem in widely-deployed systems, such as in the medical field, where it can perpetuate health disparities [ 9 ] . Amidst the challenge of bridging the gap between expert and public understanding, a series of research has emerged focused on building tools for AI literacy. These tools can be broadly categorised into two parts: diagnostic tools for experts and pedagogical tools for the public. Tenney et al.’s Language Interpretability Tool (LIT) is a powerful example of a diagnostic tool for experts, allowing for deep analysis of model internals [ 10 ] . Similarly, Biaslyze: The NLP Bias Identification Toolkit is a powerful Python package that offers a concrete entry point for impact assessment within NLP models, along with mitigation measures [ 11 ] . Kabir et al.’s STILE and Viswanath et al.’s FairPy are two other noteworthy toolkits aimed at measuring and mitigating biases in language models [ 12 , 13 ] . While these tools serve as invaluable analysers for experts, they are not designed for a non-technical audience. They are “expert-facing” and require a high degree of technical knowledge to interpret. Raz et al., on the other hand, developed Face Mis-ID , a computer vision platform targeted at a non-specialist audience [ 14 ] . Their goal was to demonstrate disparate failure of facial recognition using a simple interaction mechanic: a “match threshold” slider, similar to the one I use in my platform. This was a powerful way to demonstrate how human policy choices lead to discriminatory outcomes. In the computer vision domain, there is yet another non-specialist tool, How Normal Am I , which is a playful platform that manages to demonstrate algorithmic judgment