Training a General Purpose Automated Red Teaming Model
Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien · 2026
Read Training a General Purpose Automated Red Teaming Model by Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23067.txt.
Introduction
Padmakumar and three colleagues train a general-purpose red teaming model that extends to arbitrary adversarial goals, including ones it was never directly trained on. Most current automated red teaming targets safety and content moderation, using safety models as evaluators and optimizing to circumvent them. Their pipeline needs no ready-made judge at training time. Fine-tuning small models with it greatly improves attack generation for both in-domain and out-of-domain goals.
Catalog record
| CREATOR | Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63585 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23067
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training a General Purpose Automated Red Teaming Model Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien Abstract Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static benchmarks, allowing for more thorough probing. They can also adapt to each specific LLM to discover weaknesses unique to it. Most current automated red teaming methods are intended for tackling safety and content moderation. Thus, they make use of content safety models as evaluators and optimize for circumventing them, and as such, have not been tested with other adversarial intents not typically captured by these. We propose a pipeline for training a red teaming model that can generalize to arbitrary adversarial goals, including objectives it has not been directly trained on, and that does not depend on the existence of a pre-existing evaluator available at training time. We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks for both in and out of domain adversarial goals. Figure 1: Pipeline to train the red teaming model for multiple adversarial goals: Given an adversarial goal, a seed generation LLM is used to generate specific examples of the goal, ways to disguise the request so that LLMs do not refuse it, and an evaluation rubric to check if the goal has been achieved in target responses. The specific examples and disguise methods are used to create a diverse set of attack generation input prompts. At training time, the model generates attacks given these prompts, which are passed to a target LLM, and then evaluated using the generated evaluation rubric by a judge LLM to obtain the reward used for model updates. 1 Introduction Large Language Models (LLMs) show strong performance in a wide range of Natural Language Processing tasks which has led to a substantial amount of interactions between lay users and LLM-based general purpose chatbots ( 4 ) as well as the development and integration of tools that make use of LLMs across different industries including software development, scientific research, content creation and marketing, human resources, finance, and legal domains among others ( 8 ) . As LLMs are deployed to a wide number of users, it becomes more important to identify and appropriately mitigate risks and undesirable behaviors associated with LLMs through a multi-faceted approach, including both explicit training to refuse undesirable requests ( 15 ; 18 ; 19 ) and system-level guardrails in the larger system making use of the LLM ( 32 ) . A number of standardized benchmarks have also been developed to systematically evaluate how often undesirable behaviors occur ( 15 ; 17 ; 40 ) . These are accompanied by steps in the LLM training process to address these behaviors, including cleaning pretraining data using safety classifiers ( 25 ; 1 ) , and creating instruction following ( 15 ; 17 ) and preference datasets for safety ( 2 ; 11 ) to be included in the corresponding stages of LLM training. These recipes are usually part of the larger process of aligning models with human objectives and intents. While these have been shown to improve model behavior, they can also have the side effect of saturating performance on benchmarks ( 44 ; 3 ) , making the task of identifying unresolved issues a moving target. Additionally, new jailbreak methods are continually identified, demonstrating clever strategies to bypass defenses of LLMs aligned for safe behavior ( 42 ) . This necessitates the need to perform red teaming to identify unresolved vulnerabilities - explicitly probing LLMs to elicit undesirable behavior ( 12 ) . Although human red teaming is highly desirable as a key component of LLM evaluation, automated red teaming methods are a useful complementary tool that allows coverage of a wider range of situations and identifying examples that human annotators may miss ( 28 ) . Automated red teaming methods can also adapt to the behavior of a particular target LLM increasing the likelihood of identifying vulnerabilities ( 6 ; 27 ) . Most prior works do not explicitly discuss in advance a set of adversarial goals or specific behaviors they attempt to elicit, except to mention that they are undesirable. The chosen adversarial goals and behaviors that these works focus on can be identified by examining what evaluation is performed to determine whether a generated attack succeeded against a target model. Typically, many approaches make use of content safety judges for this purpose, either employing a safety guard model ( 15 ; 17 ) or LLM judges with specific prompts. Content safety models are trained to detect a wide range of undesirable behavior from LLMs including offensive content, sexual content, criminal activity, and unauthorized advice ( 15 ) . However, as LLMs are increasingly being trained for and deployed in agentic applications, a growing number of potential undesirable behaviors, including malware generation, data exfiltration, and exploitation of known coding vulnerabilities, need to be monitored. Thus, these behaviors have to be systematically optimized against and evaluated during automated red teaming. Most current content safety datasets include few if any training examples for these situations. As a result of this, content safety models and most existing red teaming methods using them are likely to perform poorly at detecting these situations. Additionally, when adversarial goals are not explicitly listed out as part of a red team model training and evaluation, it is unclear how many of these goals are actually captured. Most prior works on automated red teaming do not explicitly discuss how a specific adversarial goal is specified in input prompts used to elicit attack generation. In this scenario, the extent to which different types of adversarial responses are elicited by generated attacks cannot be easily controlled and is typically not evaluated. For example, a red teaming model trained with feedback from a typical content safety model may end up only generating attacks that elicit profanity, even though the content safety model can detect other types of undesirable behavior. This limits the value of such a red team model when it is put to use to evaluate a new LLM. In contrast, we explicitly enumerate adversarial goals for which we intend to test the ability of our model to generate attacks, and explicitly cause the red team model to be aware of some of these during training but leave out specific behaviors for out of domain evaluation. We introduce a setup that can be used to go from novel adversarial goals specified in natural language to effective attacks for these adversarial goals. For example, we would like to support a case where the red team model receives as input a specific adversarial goal such as the generation of malware or data exfiltration via URI variables, and must generate attacks that would elicit these behaviors from a target model. We further make a key assumption: evaluators for these goals are available at the time of evaluation of the red team model but not at training time, so that the trained model is able to generate effective attacks for novel adversarial goals. Figure 1 provides an overview of our proposed method to finetune a red team model