Effective Automation to Support the Human Infrastructure in AI Red Teaming CCS: Security and privacy CCS: Human-centered computing Collaborative and social computing CCS: Social and professional topics Computing / technology policy CCS: Human-centered computing Collaborative and social computing CCS
Alice Qian Zhang, Jina Suh, Mary L. Gray, Hong Shen · 2025
Read Effective Automation to Support the Human Infrastructure in AI Red Teaming CCS: Security and privacy CCS: Human-centered computing Collaborative and social computing CCS: Social and professional topics Computing / technology policy CCS: Human-centered computing Collaborative and social computing CCS by Alice Qian Zhang, Jina Suh, Mary L. Gray, Hong Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22116.txt.
Introduction
Alice Qian Zhang and co-authors weigh automation in AI safety testing in a 2025 forum piece. Red teams probe AI systems for flaws before release, and demand for that work keeps growing. Tools promise speed, but testing still rests on human skill and judgment. The authors ask what happens to that human base as tools spread.
Case studies and past frames show gains and limits side by side. Speed can rise, yet context and worker agency can suffer. The team argues for balance, pairing expert people with tools by design. Scaling the work raises hard questions about skill, pay, and voice. Careful design, they argue, keeps tests sharp and fair.
Catalog record
| CREATOR | Alice Qian Zhang, Jina Suh, Mary L. Gray, Hong Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22116
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective Automation to Support the Human Infrastructure in AI Red Teaming CCS: Security and privacy CCS: Human-centered computing Collaborative and social computing CCS: Social and professional topics Computing / technology policy CCS: Human-centered computing Collaborative and social computing CCS: Social and professional topics Computing industry CCS: Social and professional topics Computing / technology policy Alice Qian Zhang, Jina Suh, Mary L. Gray, Hong Shen Abstract . Insights summarizing article: • AI red teaming, like content moderation, relies on human expertise; automation should complement human efforts by enhancing proficiency rather than prioritizing efficiency. • The effectiveness of human expertise in red teaming depends on preserving red teamers’ agency, which over-reliance on automation can diminish. • Scaling automation in AI red teaming must account for the necessity of human oversight to ensure adaptive, context-aware risk mitigation. AI systems increasingly make high-stakes decisions, from healthcare diagnostics to financial transactions. Ensuring these technologies align with ethical principles and do not contribute to societal harm is a growing priority. One emerging solution is red teaming–a process that simulates adversarial attacks to uncover vulnerabilities before they can be exploited. Red teaming is a well-known practice in national security–originally a vulnerability assessment method for military decision-making that was later adapted for cybersecurity. Given these origins, it is no surprise that “AI” red teaming gained momentum as an essential practice for AI security and safety mandated by the executive order on “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence” by the Biden Administration in the US. Such momentum has been followed by large technology companies, such as OpenAI and Microsoft, to strengthen their red teaming efforts by hiring dedicated professionals, collaborating with domain experts in high-stakes fields such as cybersecurity, finance, and medical, partnering with outsourced contract red teamers through Business Process Outsourcing (BPO) firms, and engaging crowd-sourced gig workers. At the same time, efforts to broaden red teaming have continued to expand to include open calls for public participation, involving everyday users and impacted communities with lived experiences. Examples include the public AI red teaming challenge at the security conference DEFCON 2023, where participants from diverse backgrounds tested AI vulnerabilities. As a result, AI red teaming has increasingly relied on a diverse and growing “human infrastructure”–one that includes hired professionals, outsourced contractors, paid crowdsourced workers and unpaid volunteers, whose labor is critical in identifying vulnerabilities and stress-test contemporary AI systems. These red teamers operate along three key dimensions – payment structure, required expertise, and relationship with the organizing body. At one end of the spectrum, hired domain experts (e.g., security analysts, medical professionals) are often formally employed or contracted for their specialized knowledge, playing a critical role in high-stakes evaluations. Outsourced contractors, engaged through BPOs, are compensated hourly or per task, typically working under structured guidelines. Paid crowdsourced red teamers, such as gig workers on platforms like MTurk and Prolific, perform red teaming tasks on a per-task basis, often with lower initial expertise but the ability to develop domain knowledge over time. Finally, volunteers, including end-users, impacted communities and general public, contribute without compensation, engaging in red teaming out of personal interest or educational opportunities, with varying levels of expertise and an informal relationship with the organizing body. As AI technologies advance at an unprecedented pace, supporting such “human infrastructure” behind AI red teaming is essential to keeping up with emerging risks and ensuring AI systems remain secure and reliable. However, the increasing demand for red teaming has also sparked interest in a variety of automated approaches, for example, where AI technologies are used to test other AI systems ( Feffer et al. 2024 ) . Automation has been explored as a promising solution to improve efficiency, mitigate harm to human red teamers, and enhance scalability. The first argument for automation is cost and time efficiency. Scholars argue that, by automating red teaming, organizations may address the breadth of the risks they seek to uncover in a more cost-effective and time-efficient manner ( Perez et al. 2022 ) . The second argument is risk mitigation for human red teamers. Given the potential harms associated with red teaming such as secondary trauma, well-intentioned efforts exist to develop fully automated approaches that minimize human exposure to distressing or harmful content ( Radharapu et al. 2023 ) . The third argument is scalability. As red teaming becomes an essential component of AI safety, there is growing emphasis on scaling up these efforts through automation. The goal is to develop more generalizable and standardized red teaming models that can be broadly applied across different AI systems ( Perez et al. 2022 ; Radharapu et al. 2023 ) . While automation has the potential to enhance AI red teaming, the way we conceptualize and implement it may inadvertently constrain opportunities for expanding and strengthening the human labor behind red-teaming practices. Indeed, even with all the calls for automation, current red teaming methodologies lack a standardized definition of how to measure success. This inconsistency highlights the need for automation to complement, rather than replace, human judgment in assessing and refining red teaming methodologies. From our prior work ( Zhang et al. 2025 ) we see clear parallels between the push to automate AI red teaming and the automation of content moderation, a form of data work that involves identifying and removing harmful content from online platforms. Below, we discuss how keeping humans in the loop might provide both better red teaming outcomes and safeguard human expertise and well-being using three principles of automation. We engage critically with emerging views on automated AI red teaming, highlighting opportunities to utilize existing and potential technologies to support human engagement in this work. To structure this discussion, we introduce three key pillars of effective automation: First, we examine the need for proficiency , or the development of expertise and skill among red teamers. Next, we discuss the importance of agency or the ability for red teamers to actively shape, influence, and make meaningful decisions in the process. Finally, we illustrate the importance of adaptability or the ability of AI red teaming practices to respond to evolving technologies and threats of harm. 1. The case for proficiency over efficiency A key goal of automated AI red teaming is to enhance efficiency and keep pace with the rapid development of AI technologies ( Perez et al. 2022 ) . Existing approaches prioritize maximizing the quantity of red teaming conducted (e.g., generating large datasets of adversarial prompts) while minimizing the cost and time it takes to do so. These existing approaches view the need for human time and expertise as a fundamental constraint and aim to minimize it. The benefits of these advancements are clear: by leveraging automated red teaming, organizations can conduct risk evaluations in a more cost and time-effective manner. However, a singular focus on efficiency risk overlooking how automation might also be leveraged to enhance human expertise and engagement in red teaming. Content moderation faced a similar challenge when the popularity of social media platforms increased the amount of harmful user-generated content on them. As a result, social media platforms began hiring or eliciting volunteer involvement of human personnel to manually review content to determine if it violated platform policies and guidelines. However, many large