How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape DOI: XXXXXXX.XXXXXXX

Patrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby, Kurt Thomas, Allison Woodruff · 2026

Read How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape DOI: XXXXXXX.XXXXXXX by Patrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby, Kurt Thomas, Allison Woodruff (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06033.txt.

Introduction

Generative AI helps both attackers and defenders online, report Patrick Gage Kelley and 4 co-authors in a 2026 study. Their work covers Trust and Safety, the field that fights harmful content and abuse on platforms. Misuse worries are growing, while defensive uses remain less studied.

The team interviewed 43 experts across child safety, election integrity, hate and harassment, scams, and violent extremism. Attackers gain scale and speed, with lower barriers to propaganda and deepfakes. Defenders imagine using the same tools to detect harm at scale, run investigations, answer with counternarratives, support moderators, and help users. The paper offers a strategic framework for using generative AI responsibly to make online spaces safer.

Catalog record

CREATORPatrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby, Kurt Thomas, Allison Woodruff
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape DOI: XXXXXXX.XXXXXXX Patrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby, Kurt Thomas, Allison Woodruff Abstract . Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of attacks, lowering the barrier to entry for creating harmful content, including sophisticated propaganda and deepfakes. Conversely, defenders envision leveraging GenAI to detect and mitigate harmful content at scale, conduct investigations, deploy persuasive counternarratives, improve moderator wellbeing, and offer user support. This work provides a strategic framework for understanding GenAI’s impact on Trust & Safety and charts a path for its responsible use in creating safer online environments. Content Warning : This paper discusses topics that some readers may find distressing, including child sexual abuse material (CSAM), hate speech, harassment, and violent extremism. 1. Introduction Recent advances in artificial intelligence have substantially raised expectations that it will transform work and society in areas such as economic productivity, scientific progress, education, and quality of life, among others ( 76 ; 78 ; 125 ) . Many of these expectations stem from the newest GenAI systems, and their general-purpose nature, which allow them to be used for a wide variety of tasks across nearly every domain. As with many areas, this technological shift is poised to broadly reshape the landscape of Trust & Safety, a field that has gained increased visibility in the CHI community in recent years ( 73 ; 20 ) . Trust & Safety encompasses the organizations and disciplines focused on protecting the internet from myriad harmful activities and content, ranging from child safety to misinformation, through content moderation, platform policies, signal gathering, and investigations. The Trust & Safety field is characterized by an ongoing interplay between attackers (e.g., scammers, harassers) and defenders (e.g., content moderation teams, child safety hotlines, law enforcement). GenAI may fundamentally change the threat landscape across Trust & Safety—who attackers are, who they target, how they operate, and the nature of their attacks. Concurrently, GenAI offers significant, underdeveloped opportunities for defenders to restructure their operations to counter both novel GenAI-enabled attacks and longstanding ones. While prior research has focused on technical safeguards to minimize misuse of AI models and systems ( 102 ; 122 ; 2 ; 85 ; 110 ; 120 ) , these studies fall short of detailing how the uneasy sociotechnical balance between attackers and defenders will evolve alongside GenAI. We advance HCI’s contribution to Trust & Safety by exploring how GenAI impacts the field, centering the perspectives of Trust & Safety experts on the front lines of their domain. Drawing on participatory methods from HCI, we conducted six-hour participatory research workshops for five different Trust & Safety domains with a total of 43 expert participants from Asia, Europe, and North America. Our contributions include: • We present a novel qualitative study of experts in five Trust & Safety domains —child safety, election integrity, hate and harassment, scams, and violent extremism—exploring how GenAI currently affects their domains, and how they expect it to affect them in the future. • We describe a dominant narrative that GenAI empowers both attackers and defenders , a view shared by participants across all Trust & Safety domains studied. • We identify structural differences across domains , such as the nature of harm and the scale of attacker operations, that lead to unique transformative effects. • Building on our participants’ insights, we identify opportunities for making the internet safer . In the remainder of the paper, we review relevant background, describe our methodology, present our findings, and discuss tangible paths for creating safer online environments. 2. Background To situate our study of GenAI’s impact on Trust & Safety, we review the technical foundations of generative models, Responsible AI and Trust & Safety, Trust & Safety domains, and emerging literature on GenAI’s effects in this area. 2.1. GenAI Models The most recent leap forward in artificial intelligence systems has been a set of technologies broadly referred to as GenAI, which includes large language models (LLMs) and other frontier models. These GenAI systems generate realistic or high-quality synthetic text, images, code, audio, and video, typically in response to a set of instructions called a prompt. Technologically, these systems are driven by an advancement in system architecture, the attention-based transformer model ( 119 ) , which allows for training extremely large models on vast datasets using highly parallelized modern chip architectures. GenAI systems are marked by three key characteristics: (1) applicability to generalized rather than specialized use cases; (2) production of original content that is often indistinguishable from human-created content; and (3) intuitive and accessible interfaces ( 18 ) . These systems are deployed as both proprietary models, typically equipped with safety guardrails by their developers, and as open-source models, which offer greater customizability and can be run with fewer inherent restrictions. 2.2. Responsible AI and Trust & Safety Responsible AI (RAI) is an umbrella term for research and practices aimed at developing safe, reliable, and ethical AI systems in a manner that aligns with societal values. Covering the entire system lifecycle, RAI is often structured around key principles ( 52 ) , such as fairness , to mitigate algorithmic bias; accountability , to establish clear lines of responsibility; and transparency , to ensure model decisions are understandable ( 31 ) . This upstream approach to technology development is enacted through governance structures and best practices for AI design, development, and deployment ( 103 ; 86 ) . This applied work translates high-level RAI principles into concrete technical safeguards, focusing on preventing sociotechnical harms ( 102 ; 122 ) by ensuring the robustness, reliability, and security of AI models and systems ( 2 ; 85 ) . It addresses vulnerabilities to prevent both direct misuse and harm from benign use ( 110 ) . This work involves a range of activities, including identifying social and ethical risks ( 90 ) ; implementing pre-training interventions, such as improving the data the model learns from ( 110 ) ; applying post-training mitigations to prevent undesirable behaviors like sycophancy or goal misinterpretation ( 84 ) ; and ensuring the fairness and robustness of tools used for content moderation, such as multimodal classifiers that detect hate speech ( 95 ) or other policy-violating content ( 39 ) . The field also focuses on mitigating risks such as “jailbreaking,” where users circumvent safety controls with adversarial prompts, and improving scaled evaluation techniques ( 120 ) . Complementing these safeguards are the operational practices of Trust & Safety, which strives “to promote a safer and more trustworthy internet” ( 26 ) . Historically, the field operated under a reactive model of incident response, such as removing user-flagged content or responding to legal requests after harm occurred ( 22 ; 67 ) ; however, Trust & Safety is shifting toward a proactive strategy to

Read the complete text