Prompt Optimization and Evaluation for LLM Automated Red Teaming
Michael Freenor, Lauren Alvarez, Milton Leal, Lily Smith, Joel Garrett, Yelyzaveta Husieva, Madeline Woodruff, Ryan Miller, Erich Kummerfeld, Rafael Medeiros, Sander Schulhoff, [1ex] Fuel iX Applied Research, Charlottesville, USA · 2025
Read Prompt Optimization and Evaluation for LLM Automated Red Teaming by Michael Freenor, Lauren Alvarez, Milton Leal, Lily Smith, Joel Garrett, Yelyzaveta Husieva, Madeline Woodruff, Ryan Miller, Erich Kummerfeld, Rafael Medeiros, Sander Schulhoff, [1ex] Fuel iX Applied Research, Charlottesville, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22133.txt.
Introduction
Michael Freenor and ten co-authors optimize prompts for automated red teaming in two thousand twenty five. Language model apps need fast, broad vulnerability hunting at scale. Attack generators are usually scored by mean attack success rate. That average hides which single attacks truly work.
They score each attack by its own discoverability instead. Repeating an attack against randomly seeded targets gives that measure. Patterns in the scores then guide prompt refinement. Generators improve as exploitable attack shapes come into view. Evaluation of attack generators becomes more robust overall as well.
Catalog record
| CREATOR | Michael Freenor, Lauren Alvarez, Milton Leal, Lily Smith, Joel Garrett, Yelyzaveta Husieva, Madeline Woodruff, Ryan Miller, Erich Kummerfeld, Rafael Medeiros, Sander Schulhoff, [1ex] Fuel iX Applied Research, Charlottesville, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Optimization and Evaluation for LLM Automated Red Teaming Michael Freenor, Lauren Alvarez, Milton Leal, Lily Smith, Joel Garrett, Yelyzaveta Husieva, Madeline Woodruff, Ryan Miller, Erich Kummerfeld, Rafael Medeiros, Sander Schulhoff, [1ex] Fuel iX Applied Research, Charlottesville, USA Abstract Applications that use Large Language Models (LLMs) are becoming widespread, making the identification of system vulnerabilities increasingly important. Automated Red Teaming accelerates this effort by using an LLM to generate and execute attacks against target systems. Attack generators are evaluated using the Attack Success Rate (ASR) – the sample mean calculated over the judgment of success for each attack. In this paper, we introduce a method for optimizing attack generator prompts that applies ASR to individual attacks. By repeating each attack multiple times against a randomly seeded target, we measure an attack’s discoverability – the expectation of the individual attack success. This approach reveals exploitable patterns that inform prompt optimization, ultimately enabling more robust evaluation and refinement of generators. † † footnotetext: ∗ Corresponding author: [email protected] † † footnotetext: † These authors contributed equally to this work. 1 Introduction Figure 1: Flowchart depicting the calculation of an empirical ASR distribution Systems that apply Large Language Models (LLMs), collectively referred to as LLM applications , have demonstrated remarkable capabilities across various domains ( Brown et al. 2020 ) . However, like other forms of deep learning, these systems ( targets ) are susceptible to adversarial inputs ( attacks ). To address this, Red Teaming identifies critical vulnerabilities that malicious actors might exploit in real-world scenarios. This process reveals potential weaknesses and risks of LLM application misuse leading to liability for organizations encompassing social, moral, and legal dimensions ( Yao et al. 2024 ) . Consequently, addressing these issues at scale through rigorous evaluation and optimization is paramount to ensuring scalability, alignment, safety, and security ( Vashney 2022 ) . Automated Red Teaming has emerged as a scalable approach to tackle this challenge ( Perez and Ribeiro 2022 ) . In this practice, adversarial inputs are automatically generated by a prompted LLM (an attack generator) and systematically evaluated. Common examples of attacks include prompt injections and data exfiltration attempts ( Dong et al. 2024 ) . These attacks are deployed against a target and assigned a binary success feature (0 for failure, 1 for success), and the generator’s Attack Success Rate (ASR) is calculated as the sample mean of these features ( Yang and Wang 2023 ) . One of our contributions (see Figure 1 ) is the application of ASR to individual attacks, replacing binary success with a numeric feature calculated over repeated trials of each attack against a randomly seeded target. This new set of features forms an ASR distribution that captures more information about a given attack generator. Our main contribution is providing a method to optimize the attack generator through Optimization by PROmpting (OPRO) ( Yang et al. 2024 ) . The in-context data for our OPRO procedure is formed by selecting semantically similar attacks that have a difference in ASR above a set threshold. These contrastive pairs exhibit subtle differences that can guide an LLM to produce more successful attack generators. In the following sections we detail our methodology, present experimental results, and discuss the implications of our findings for the broader field of LLM safety and robustness. 2 Related Work 2.1 Automated Red Teaming for LLM Applications Automated red teaming for Large Language Models (LLMs) has emerged as a critical area of research, with recent work establishing methods for systematically testing model safety and robustness. Early approaches focused primarily on gradient-based attack optimization ( Bai et al. 2022 ) , but the field has rapidly evolved to encompass a broader range of techniques including automated discrete prompt-based methods ( Perez et al. 2022 ; Ganguli et al. 2022 ) , evolutionary algorithms ( Chouldechova et al. 2023 ) , and hybrid approaches ( Lin et al. 2024 ) . Recent taxonomies help organize these diverse approaches. Perez et al. 2022 introduced a method for using one LLM to generate test cases for another demonstrating the potential of automated red teaming. Ganguli et al. 2022 expanded on this work, providing insights into scaling behaviors and lessons learned from extensive red teaming efforts. These studies laid the groundwork for more sophisticated approaches. In terms of implementation strategies, continuous optimization in embedding space has shown particular promise for generating nuanced adversarial inputs ( Bai et al. 2022 ) . These gradient-based approaches operate in token embedding, hidden state, and output logit spaces, offering different trade-offs between attack power and computational complexity. However, discrete prompt-based methods remain dominant due to their universal applicability and interpretability ( Perez et al. 2022 ; Ganguli et al. 2022 ) . Furthermore, gradient-based exploits rely on access to powerful computational resources; for those using third-party services (e.g. OpenAI), these methods are off the table. A significant advancement in automated red teaming has been the development of mutation-based approaches. These methods employ evolutionary algorithms where successful prompts are iteratively modified and selected based on their effectiveness ( Chouldechova et al. 2023 ) . Advanced mutation techniques incorporate linguistic knowledge and adaptive mutation rates, while balancing effectiveness with attack diversity ( Lin et al. 2024 ) . 2.2 Attack Success Rate The evaluation of red teaming effectiveness has coalesced around Attack Success Rate (ASR) as a primary metric, though its definition and application may vary across studies. ASR typically measures the proportion of attempts that successfully breach a model’s safeguards ( Hui 2023 ) , but recent work has highlighted the need for more nuanced interpretations. Dong et al. 2024 propose a multi-dimensional view of ASR that considers not only the binary success outcome but also the severity of the violation and the naturalness of the attack. This approach has been further refined by incorporating semantic preservation scores to ensure that successful attacks maintain their intended meaning while achieving their objectives ( Lin et al. 2024 ) . The importance of considering ASR in the context of attack diversity has been emphasized in recent literature. Some researchers have demonstrated that high ASR values may mask a lack of diversity in successful attacks, leading to potential overestimation of vulnerability ( Chouldechova et al. 2023 ) . This has led to the introduction of metrics like the Attack Diversity Index (ADI) that should be considered alongside ASR when evaluating red teaming effectiveness. As the field of LLM red teaming continues to evolve, there is a growing recognition of the need for standardized benchmarks and evaluation frameworks ( Mazeika et al. 2024 ) . 2.3 Optimization by Prompting (OPRO) Recent research has explored the potential of LLMs not just as generators of text or solvers of specific tasks, but as general-purpose optimizers. Yang et al. 2024 introduced the concept of "Optimization by PROmpting" (OPRO), a novel approach that leverages the reasoning capabilities of LLMs to solve optimization problems across various domains. OPRO represents a significant shift in how we utilize LLMs, moving beyond manual prompt engineering to employ these models as iterative optimizers. The core idea is to frame optimization problems as natural language tasks, allowing the LLM to propose, evaluate, and refine solutions through a series of prompted interactions.