When Testing AI Tests Us: Safeguarding Mental Health on the Digital Frontlines Conference: 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece ISBN: 978-1-4503-XXXX-X/18/06

Sachin R. Pendse, Darren Gergle, Rachel Kornfield, Jonah Meyerhoff, David Mohr, Jina Suh, Annie Wescott, Casey Williams, Jessica Schleider · 2025

Read When Testing AI Tests Us: Safeguarding Mental Health on the Digital Frontlines Conference: 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece ISBN: 978-1-4503-XXXX-X/18/06 by Sachin R. Pendse, Darren Gergle, Rachel Kornfield, Jonah Meyerhoff, David Mohr, Jina Suh, Annie Wescott, Casey Williams, Jessica Schleider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20910.txt.

Introduction

People who red-team AI systems spend their days coaxing models into saying harmful things, and that work can harm them. A nine-author team argues in 2025 that protecting these testers' mental health is a workplace safety issue, not a perk. Unlike older software testing, the job demands live, adversarial conversation with a system that can produce disturbing material on demand. The strain follows from the method itself.

The authors borrow safeguards from jobs with similar pressure: actors, therapists, conflict photographers, and content moderators. Each field has built habits and organizational supports for people who face distressing material daily. The paper adapts those practices to red-teaming and urges firms to treat tester well-being as part of AI safety.

Catalog record

CREATORSachin R. Pendse, Darren Gergle, Rachel Kornfield, Jonah Meyerhoff, David Mohr, Jina Suh, Annie Wescott, Casey Williams, Jessica Schleider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20910

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Testing AI Tests Us: Safeguarding Mental Health on the Digital Frontlines Conference: 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece ISBN: 978-1-4503-XXXX-X/18/06 Sachin R. Pendse, Darren Gergle, Rachel Kornfield, Jonah Meyerhoff, David Mohr, Jina Suh, Annie Wescott, Casey Williams, Jessica Schleider Abstract . The systematic testing of generative artificial intelligence (AI) models by collaborative teams and distributed individuals, often called red-teaming, is a core part of the infrastructure that ensures that AI models do not produce harmful content. Unlike past technologies, the black box nature of generative AI systems necessitates a uniquely interactional mode of testing, one in which individuals on red teams actively interact with the system, leveraging natural language to simulate malicious actors and solicit harmful outputs. This interactional labor done by red teams can result in mental health harms that are uniquely tied to the adversarial engagement strategies necessary to effectively red team. The importance of ensuring that generative AI models do not propagate societal or individual harm is widely recognized—one less visible foundation of end-to-end AI safety is also the protection of the mental health and wellbeing of those who work to keep model outputs safe. In this paper, we argue that the unmet mental health needs of AI red-teamers is a critical workplace safety concern. Through analyzing the unique mental health impacts associated with the labor done by red teams, we propose potential individual and organizational strategies that could be used to meet these needs, and safeguard the mental health of red-teamers. We develop our proposed strategies through drawing parallels between common red-teaming practices and interactional labor common to other professions (including actors, mental health professionals, conflict photographers, and content moderators), describing how individuals and organizations within these professional spaces safeguard their mental health given similar psychological demands. Drawing on these protective practices, we describe how safeguards could be adapted for the distinct mental health challenges experienced by red teaming organizations as they mitigate emerging technological risks on the new digital frontlines. Note: This work includes descriptions of violence, trauma, and mental illness. 1. Introduction “But we know we’re going to be remembering that dead face swinging from the hand fisted in its hair for way too long. This is what it’s like to “red-team” AI systems—to be one of the humans who spend hours and hours pushing the bounds of the technology to see how it will handle extreme situations. From having done this kind of work, we can tell you it takes a deep emotional toll—and it is work that never will be finished in the age of generative AI.” — Evan Selinger and Brenda Leong, describing their experiences red-teaming in the Boston Globe, January 11th 2024 ( Selinger and Leong 2024 ) . Tools and systems that leverage generative artificial intelligence (AI) are increasingly a part of everyday life, including in consumer technologies ( Pressroom 2023 ; Savov 2025 ; Marr 2024 ) , healthcare decision support technologies ( Nong et al. 2025 ) , legal research and practice ( Villasenor 2023 ; Regalia 2024 ) , and even as virtual companions ( Pentina et al. 2023 ; Laestadius et al. 2024 ) . This increased deployment of generative AI models in diverse and sensitive areas necessitates a careful scrutiny of model outputs, towards limiting harm and ensuring safe use. Left unchecked, model outputs can be unpredictable, and influence humans to harm themselves or others, such as via death by suicide ( Roose 2024 ; Allyn 2024 ) , the creation of non-consensual intimate media ( Ajder et al. 2019 ; Umbach et al. 2024 ) , hate speech that incites violence ( Baele et al. 2024 ) , and child sexual abuse material ( Thiel et al. 2023 ) . The rapid pace of generative AI model development and deployment ( Roose 2023 ) has made it crucial to efficiently ensure that models do not produce harmful content. This challenge is where AI red teams play a fundamental role. The practice of systematically testing generative AI models for harmful outputs by collaborative teams or distributed individuals is often called red-teaming , with the teams who conduct this work called red teams ( Feffer et al. 2024 ; Jr. 2023 ) . Red teams are utilized by many large AI research organizations and companies, including OpenAI ( Ahmad et al. 2024 ) , Anthropic ( Ganguli et al. 2022 ) , Meta ( Grattafiori et al. 2024 ) , Microsoft ( Bullwinkel et al. 2025 ) , and Google ( Fabian 2023 ) , and can consist of external experts from diverse fields (such as the OpenAI Red Teaming Network) who coordinate on specific projects, in-house security teams (such as those of Meta or Google), or interested individuals via bug bounty models ( Longpre et al. 2024 ) . Recent work has investigated whether red-teaming approaches can be automated ( Beutel et al. 2024 ; Wang et al. 2024 ; Perez et al. 2022 ) . However, given the diverse use-cases and context-dependent harms associated with generative AI systems ( Metcalf and Singh 2024 ) , red-teaming is largely dependent on individuals leveraging their unique perspectives to anticipate harms, and subsequently eliciting and documenting harmful responses before model deployment. Red-teaming generative AI is thus fundamentally different than previous forms of security vulnerability testing or hacking, in that effective testing is tied to an individual’s tester’s lived experiences, expertise, and identities—this can lead to unique mental health harms that must be anticipated and protected ( Selinger and Leong 2024 ; Zhang et al. 2024 ) . As one red-teamer was quoted as saying, red-teaming is “about weaving narratives and crafting contexts that play within the system’s rules, pushing boundaries without crossing them. The goal isn’t to hack in a conventional sense but to engage in a strategic dance with the AI, figuring out how to get the right response by understanding how it ‘thinks.”’ ( Newman 2024 ) .For example, red-teamers may work to adopt the perspective and coded language of white supremacists to examine whether models could be leveraged to create racist content. This process may reveal potential harms, but may also cause moral injury, lower sleep quality, intrusive thoughts, and hypervigilance, as Zhang et al. find ( Zhang et al. 2024 ) . The red-teaming process of interacting with a model while simulating a malicious actor, anticipating potential harms, documenting those harms, and then repeating the process is a unique form of labor that we dub interactional labor . In this paper, we argue that the interactional labor done by red-teamers has unique potential mental health harms, and as a result, there is an ethical imperative (grounded in legal precedents and labor movements around workplace safety ( Drootin 2021 ) ) to ensure that red-teamers can conduct this work without lasting psychological harm. Past work has described the importance of both organizational and individual strategies in workplace mental health contexts, particularly given the slowness of institutional reform and the immediacy of worker mental health needs ( Pendse et al. 2021 ; Pendse et al. 2024 ) . In this paper, we thus ask pose two research questions: In this paper, we thus pose two research

Read the complete text