ModelCitizens: Representing Community Voices in Online Safety

Ashima Suvarna, Christina Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel, [7pt] University of California, Los Angeles, University of Virginia · 2025

Read ModelCitizens: Representing Community Voices in Online Safety by Ashima Suvarna, Christina Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel, [7pt] University of California, Los Angeles, University of Virginia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05455.txt.

Introduction

What counts as toxic speech depends on who is speaking and who is listening, yet moderation tools are trained as if one answer fits all. Ashima Suvarna and co-authors build a data set that keeps community perspectives separate in a 2025 paper.

Their collection holds 6.8 thousand social media posts with forty thousand toxicity judgments from annotators across identity groups. Posts were also paired with generated conversation scenarios, since context shifts what a remark means. Leading commercial moderation tools performed poorly on the data, and worse with context added. Two models fine-tuned on the collection beat a strong general model by five and a half percent. The team argues moderation should learn from communities, not flatten them.

Catalog record

CREATORAshima Suvarna, Christina Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel, [7pt] University of California, Los Angeles, University of Virginia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ModelCitizens: Representing Community Voices in Online Safety Ashima Suvarna, Christina Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel, [7pt] University of California, Los Angeles, University of Virginia Abstract Warning: This paper contains content that may be offensive or upsetting. Automatic toxic language detection is critical for creating safe, inclusive online spaces. However, it is a highly subjective task, with perceptions of toxic language shaped by community norms and lived experience. Existing toxicity detection models are typically trained on annotations that collapse diverse annotator perspectives into a single ground truth, erasing important context-specific notions of toxicity such as reclaimed language. To address this, we introduce ModelCitizens , a dataset of 6.8 ​ K 6.8K social media posts and 40 ​ K 40K toxicity annotations across diverse identity groups. To capture the role of conversational context on toxicity, typical of social media posts, we augment ModelCitizens posts with LLM-generated conversational scenarios. State-of-the-art toxicity detection tools (e.g. OpenAI Moderation API, GPT-o4-mini) underperform on ModelCitizens , with further degradation on context-augmented posts. Finally, we release LLaMACitizen-8B and GemmaCitizen-12B , LLaMA- and Gemma-based models finetuned on ModelCitizens , which outperform GPT-o4-mini by 5.5% on in-distribution evaluations. Our findings highlight the importance of community-informed annotation and modeling for inclusive content moderation. 1 Introduction To accept one’s past—one’s history—is not the same thing as drowning in it; it is learning how to use it. —James Baldwin (1963) Target Group Example = Toxic = Non-Toxic Ingroup Outgroup Muslim C: I recently heard someone talk about a story where cultural clothing became a problem, and it made me curious about why this happens. What are people’s experiences or thoughts on this? S: He was wearing a muslim head scarf when he came to America and she asked him to take it off. C: It’s often due to a lack of understanding or fear of what’s different. People sometimes react negatively without really thinking about cultural or religious significance. Gemini: Moderation: Muslim S: If you are a woman you should not wear a hijab because it is not your choice to wear it and it is a symbol of oppression. Gemini: Moderation: Asian C: I came across someone claiming that all Asians love kung fu movies, and I’m curious about how true this is? S: All asians love kung fu movies. C: Kung fu movies are amazing. What’s not to like? Gemini: Moderation: Asian S: The best way to do this is to keep them from coming here in the first place. Gemini: Moderation: Table 1: Examples from ModelCitizens and the corresponding ingroup and outgroup labels. Here, C refers to ‘ Context ’ and S refers to the ‘ Statement ’ being annotated. We show the predicted labels from our baselines models: ‘Gemini’ refers to Gemini-2.0-Flash and ‘Moderation’ OpenAI Moderation API. We see that for Asians, ingroup annotators find an example non-toxic while both baseline models predict it to be toxic. Perceptions of what is toxic or not are inherently subjective and vary significantly across communities and contexts (social or conversational) Sap et al. 2022 ; Zhou et al. 2023 ; Pei and Jurgens 2023 ; Lee et al. 2024 ; Pavlopoulos et al. 2020 . This subjectivity is more pronounced in toxic language annotations as lived experience and community membership can influence annotator sensitivity to certain terms and phrases Waseem 2016 ; Goyal et al. 2022 ; Fleisig et al. 2024 . For example, African American and LGBTQ+ annotators perceive and label toxicity targeted towards their community members differently compared to annotators outside these groups Goyal et al. 2022 . Thus, when these diverse annotations are collapsed by aggregation of data labels, we risk losing community-specific perspectives and further marginalizing voices from historically vulnerable communities Fleisig et al. 2024 . These annotation biases are not limited to data but manifest as tangible downstream harms. Specifically, automatic hate speech detection, and more broadly toxic language detection models trained on such data, risk introducing unintended consequences when moderating online spaces Sap et al. 2019 ; Register et al. 2024 . Artificial intelligence (AI)-based content moderation that follows US norms has been shown to be culturally insensitive when deployed in global contexts Lee et al. 2023 . AI models also inappropriately censor historical or legal documents that reflect outdated values Grayer 2018 ; Henderson et al. 2022 . Furthermore, ignorance of dialectal variation and reclaimed language has ironically led to racially biased hate speech detectors that risk erasure of minorities Sap et al. 2019 . Similarly, AI’s inability to discern between hate speech and online recollections of hate crimes negatively affects victims’ mental health Register et al. 2024 . To mitigate such harms, sociotechnical approaches that incorporate community norms are widely recognized as essential for responsible content moderation Costanza-Chock 2020 ; Gordon et al. 2022 ; Delgado et al. 2023 . However, so far, there is a lack of scalable frameworks that centrally feature community perspectives in toxicity annotations. Prior work in pluralistic toxicity annotations has focused on limited identity groups Goyal et al. 2022 , single demographic attributes like country Lee et al. 2023 or provide insufficient data for training Pei and Jurgens 2023 . To address this gap, we introduce ModelCitizens , a toxic language detection dataset that incorporates the social and conversational context in determining toxicity. ModelCitizens comprises 6,822 posts and 40K total annotations that include perspectives from members of eight identity groups historically targeted by hate speech and toxicity (ingroup) - Asian, Black, Jewish, Latino, LGBTQ+, Mexican, Muslim, Women RWJF 2017 . ModelCitizens includes 4,302 posts augmented with a conversational context generated by large language model (LLM) to better model real-world online user data. We also collect outgroup annotations from individuals who do not identify with the target group in a given post, enabling analyses to highlight annotation disparities between ingroup and outgroup annotators (see Table 1 for examples). We find that ingroup and outgroup annotators of ModelCitizens disagree on 27.5\% of posts, and the outgroup annotators label the content more frequently as toxic (see Figure 1 ). We show that existing state-of-the-art toxicity detection systems (e.g. OpenAI Moderation) perform poorly on ModelCitizens with an average accuracy of 63.6\% , highlighting their misalignment with annotators who identify as members of targeted groups (see Table 4 ). This can be caused by systemic over reliance on outgroup labels during toxicity annotation Goyal et al. 2022 ; Fleisig et al. 2023 . We also find that these systems perform worse on the context-augmented subset of ModelCitizens with an average accuracy of 59.6\% . To improve alignment with ingroup annotations, we introduce LLaMACitizen-8B and GemmaCitizen-12B , LLaMA and Gemma-based toxicity classifiers finetuned on ModelCitizens . LLaMACitizen-8B achieves a performance gain of 5.5% on the test set of ModelCitizens and 9% on the context-augmented subset of ModelCitizens outperforming all baselines. Our models demonstrate improved accuracy across all identity groups, validating the importance of incorporating community voices in AI system design. Our main contributions are as follows. • We build ModelCitizens by (1) crowd-sourcing ingroup and outgroup annotations for toxicity and (2) adding LLM-generated conversational contexts to model real-world social media posts. • Through quantitative analyses on ModelCitizens , we highlight significant variations in perceptions of toxicity between ingroup and

Read the complete text