Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
Alberto Purpura *, Sahil Wadhwa *, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad · 2025
Read Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models by Alberto Purpura *, Sahil Wadhwa *, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01742.txt.
Introduction
Alberto Purpura and a team of co-authors wrote this 2025 survey of red teaming for large language models. Red teaming means attacking a model on purpose to find its weaknesses before bad actors do. The survey targets builders who need the whole picture fast.
It opens with the safety demands placed on well-known models, then walks through a red teaming setup piece by piece. Attack methods get a full tour, from hand-written prompts to automated fuzzing. So do the ways of judging whether an attack worked and of scoring a campaign. Software packages that implement each piece are named along the way. The authors frame red teaming as a core stage in shipping safe generative AI products.
Catalog record
| CREATOR | Alberto Purpura *, Sahil Wadhwa *, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01742
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models Alberto Purpura *, Sahil Wadhwa *, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad Abstract The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems against misuse by malicious actors, researchers have recently complemented these efforts with an offensive approach that involves red teaming , i.e., proactively attacking LLMs with the purpose of identifying their vulnerabilities. This paper provides a concise and practical overview of the LLM red teaming literature, structured so as to describe a multi-component system end-to-end. To motivate red teaming we survey the initial safety needs of some high-profile LLMs, and then dive into the different components of a red teaming system as well as software packages for implementing them. We cover various attack methods, strategies for attack-success evaluation, metrics for assessing experiment outcomes, as well as a host of other considerations. Our survey will be useful for any reader who wants to rapidly obtain a grasp of the major red teaming concepts for their own use in practical applications. 1 Introduction The popularity and widespread adoption of Large Language Models (LLMs) has been transformative across many industries, ushering in new possibilities for enhancing productivity, decision-making, and user engagement. LLMs are contributing significantly to fields such as finance, healthcare, and legal services where they are being leveraged for tasks such as customer servicing support, clinical notes and contract analysis. However, the increasing reliance on LLMs brings with it a critical and challenging ethical-moral responsibility: ensuring that the deployed system responds to any possible input in safe or otherwise desirable ways. While LLMs offer remarkable capabilities, they are also vulnerable to various forms of misuse. Such attacks could provoke LLMs to generate misinformative, biased, or toxic content Abid et al. (2021) ; Lin et al. (2022) or expose private information Carlini et al. (2021) . Microsoft’s Tay, in a high-profile case, was successfully provoked by attackers to send racist or sexually-charged tweets to a large audience Lee (2016) . A great deal of research on improving LLM safety has been conducted from a defensive standpoint, with investigators developing methods for guardrailing LLMs against potential attacks Dong et al. (2024) . These attacks, however, must be identified beforehand, which has proven to be challenging – e.g., GPT-4 was vulnerable to attacks absent from its safety training that were written in low-resource languages Yong et al. (2024) . Investigators have hence turned to complementing defensive efforts with an offensive approach to LLM safety, proposing strategies for red teaming LLMs, i.e., proactively attacking or testing LLMs with the purpose of identifying their vulnerabilities. Red teaming is useful for any organization that aims not only to productionize some LLM-supported system, but to effectively anticipate threats to their system and safeguard against them before production. While prior reviews of LLM red teaming focused on serving as an encyclopedic taxonomic resource, e.g., of attack methodologies Lin et al. (2024) , we anticipate a wide need for a concise and practical overview geared toward readers who want to rapidly grasp the major concepts and components of a red teaming system and available software tools that have emerged, for example to devise and implement a system of their own. The purpose of this paper is to provide such an overview: one that balances comprehensive treatment of research with conciseness, and structures the exposition to describe a multi-component system end-to-end. Figure 1 provides an illustration of the framework and its components, the latter of which are covered in Sections 4 , 5 and 6 . Figure 1: Core components of a red teaming system. After covering related work, we survey a few case studies from the tech industry, highlighting problems that motivate the field of red teaming. We then dive into the central components that make up a complete red teaming system, reviewing popular methods, software packages, and other resources that have emerged to support these components. We cover various attack strategies, with attention paid to categorizing particular methods and distinguishing single-turn from multi-turn attacking, manual from automated attacking, and different varieties of automated attacking. We then dive into popular approaches to attack success evaluation, as well as safety metrics for assessing overall experimental outcomes. We discuss a number of publicly available resources for red teaming, including software packages and datasets. We also touch briefly upon guardrailing steps commonly taken after red teaming. Finally, we close with future directions that we judge to constitute some of the most impactful opportunities for progress, including strategies for adapting automated attackers to generate more relevant and diverse sets of attacks, including multi-turn ones. 2 Related work Recent literature has explored various facets of LLM red teaming, offering valuable insights into the rapidly evolving field. Some organizations aim to provide up-to-date informational materials geared toward helping developers and web-security practitioners secure their particular applications MITRE (2024) ; Commons () ; Commons (2024) . For example, the OWASP Foundation published the OWASP Top 10 OWASP Foundation (2025) , a document that describes, as deemed by common consensus, some of the most major threats to the security of LLM-supported applications, and provides mitigation strategies. On the academic side, Feffer et al. (2024) provide a high-level overview of and stance on red teaming practices, indexing on particular aspects of the literature to argue that the red teaming community lacks consensus around scope, structure, and evaluation of red-teaming. Verma et al. (2024) operationalize a threat model for red teaming, providing a taxonomy based on entry points in the LLM lifecycle. Rawat et al. (2024) provide a practitioner’s viewpoint of challenges within LLM red teaming and emphasizes the context-dependent nature of vulnerabilities, and introduces a taxonomy of single-turn, prompt-based attacks. Mo et al. (2024) develop a taxonomy of attacks against language agents in particular – i.e., systems equipped with additional capacities for reasoning, planning, and task completion. Shi et al. (2024) offer a comprehensive survey of LLM safety more broadly, encompassing various risks beyond attacks, including value misalignment and autonomous AI risks. But perhaps the most extensive treatment to date specifically of LLM red teaming is given in Lin et al. (2024) , which provides a fine-grained taxonomy of attack strategies grounded in LLM capabilities as well as several mitigating strategies, an overview of attack success evaluation strategies, and a framework that unifies attack-search strategies for automated red teaming. While insightful, the latter two articles’ extensive lengths would be prohibitive for readers who seek a more concise overview of major red teaming concepts and trends. We see these papers as valuable in their own right, but anticipate the need for a resource that balances broad representation of the literature with concise exposition. 3 Policies on LLM Safety Policies and risk mitigation strategies devised for ensuring the proper use of LLM-driven products have been crucial to their safety and success. This section serves to motivate LLM red teaming and LLM safety, providing a brief survey of some major safety considerations and policies from different leading LLM providers within industry along with risk mitigation strategies they have taken. These policies play a key role in shaping the goals