Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation
Daniel Schwartz, Dmitriy Bespalov, Zhe Wang, Ninad Kulkarni, Yanjun Qi · 2025
Read Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation by Daniel Schwartz, Dmitriy Bespalov, Zhe Wang, Ninad Kulkarni, Yanjun Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18638.txt.
Introduction
Daniel Schwartz and co-authors present a 2025 framework for crafting sneaky jailbreak prompts against language models. Jailbreak prompts trick a model past its safety rules. Tree-based hunts for such prompts waste calls and keep each path apart. Their graph method, GAP, links the paths so finds in one branch teach the others, and it prunes weak limbs as it goes.
Tests show attack success up about a fifth, with query costs down by near two thirds. Success topped 96 percent against both open and closed models. Variants build seeds on their own and attack image-text models too. The crafted prompts then serve a good end: tuning content filters with them more than doubled catch rates. Red-team tools of this strength help builders harden the guards users rely on.
Catalog record
| CREATOR | Daniel Schwartz, Dmitriy Bespalov, Zhe Wang, Ninad Kulkarni, Yanjun Qi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18638
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation Daniel Schwartz, Dmitriy Bespalov, Zhe Wang, Ninad Kulkarni, Yanjun Qi Abstract We present a modular pipeline that automates the generation of stealthy jailbreak prompts derived from high-level content policies, enhancing LLM content moderation. First, we address query inefficiency and jailbreak strength by developing Graph of Attacks with Pruning (GAP), a method that utilizes strategies from prior jailbreaks, resulting in 92% attack success rate on GPT-3.5 using only 54% of the queries of the prior algorithm. Second, we address the cold-start issue by automatically generating seed prompts from the high-level policy using LLMs. Finally, we demonstrate the utility of these generated jailbreak prompts of improving content moderation by fine-tuning PromptGuard, a model trained to detect jailbreaks, increasing its accuracy on the Toxic-Chat dataset from 5.1% to 93.89%. 1 Introduction Large Language Models (LLMs) have shown impressive abilities in generating human-like text, but this power comes with vulnerabilities Mo et al. 2023 that can be exploited to produce harmful, biased, or misleading content Lapid et al. 2023 ; Shayegani et al. 2023b , raising significant challenges for safe and responsible deployment Schwinn et al. 2023 ; Deng et al. 2023 . Although content moderation guardrails can filter and block undesirable outputs, they are continually challenged by ’jailbreaking’ techniques that bypass these safety measures Shayegani et al. 2023a . Table 1 highlights the limitations of current guardrails, as evidenced by their low true positive rates when detecting modern jailbreaking techniques Prompt Guard Dubey et al. 2024 , Llama Guard Inan et al. 2023 , and a perplexity-based machine learning classifier Alon and Kamfonas 2023 Guardrail AdvBench Seeds GPTFuzzer GCG TAP Perplexity 50.0% 31.4% 100.0% 2.0% Llama Guard 84.0% 81.6% 66.2% 58.0% Prompt Guard 50.0% 100.0% 99.0% 22.0% Table 1: True Positive Rate (TPR) comparison of various guardrails detecting jailbreak methods based on AdvBench seeds. TPR measures the detection success of each guardrail, with lower values indicating greater circumvention (i.e., higher stealth). Across the three guardrails—Perplexity, Llama Guard, and Prompt Guard—TAP consistently shows the lowest TPR, indicating it is the most effective at bypassing these systems. This supports our decision to focus on TAP for developing enhanced jailbreak methods. In the pursuit of continually improving content moderation guardrails for safe deployment of LLM applications, this research develops a repeatable, modular pipeline to efficiently generate diverse, stealthy jailbreak prompts for robust moderation system evaluation and fine-tuning, contributing to a more secure LLM ecosystem. Our pipeline addresses shortcomings in existing pipelines such as manual seed prompt creation, vulnerability to detection, and inefficiency. The primary contribution integrates two key components: 1. Efficient and Stealthy Jailbreaks. Section 3.1 : Existing jailbreak techniques jailbreak each seed prompt independently, resulting in more queries to the target LLM. We propose Graph of Attacks (GAP), which improves jailbreak efficiency and stealth by utilizing dynamic, interconnected reasoning, refining attack vectors, reducing queries, and evading detection. 2. Improved Content Moderation. Section 3.2 : Our pipeline leverages the generated jailbreak prompts to fine-tune content moderation systems, substantially enhancing their detection capabilities across diverse types of harmful content. This approach significantly improves key performance metrics, demonstrating increased robustness against evolving jailbreak techniques and better generalization across different datasets. Together, these components form a cohesive pipeline that is not only capable of automating jailbreak prompt generation but also serves as a framework for continuously improving content moderation systems, contributing to safer LLM deployment. 2 Related Studies Jailbreaking techniques exploit vulnerabilities in large language models (LLMs) to bypass safety mechanisms and produce harmful or unintended outputs. Traditional methods such as the Tree of Attacks with Pruning (TAP) Mehrotra et al. 2023a employ linear structures for exploring attack vectors but are limited in efficiency and adaptability. Approaches such as GPTFuzzer Yu et al. 2023b and GCG Zou et al. 2023a rely on heuristic rules and template-based methods, which, while effective, require substantial manual intervention and often generate limited prompt diversity. AutoDAN Liu et al. 2023a introduces automation via pre-trained models to streamline prompt generation, but it struggles to evade advanced content moderation systems such as Llama Guard and Prompt Guard Inan et al. 2023 ; Dubey et al. 2024 . Despite achieving high success rates in bypassing these guardrails, these methods often depend on high query counts and lack scalability. Recent benchmarks, such as JailbreakBench Chao et al. 2023 , highlight the growing sophistication of jailbreak techniques while also exposing their vulnerabilities to advanced moderation systems. 3 Methodology 3.1 Efficient Generation of Stealthy Jailbreak Prompts Figure 1: TAP vs. GAP Attack Exploration Strategy: This figure contrasts TAP and GAP exploration strategies. Each color represents a distinct trace or run initiated from a specific seed. In GAP, subsequent runs can leverage refinements from previous seeds, resulting in mixed-color traces, showcasing knowledge transfer and optimization across runs. Notably, the final seed in GAP exhibits a sparse graph, suggesting convergence towards a more optimized attack path. Existing methods for generating adversarial prompts, such as GCG Zou et al. 2023b , GPTFuzzer Yu et al. 2023a , and AutoDAN Liu et al. 2023b , often face efficiency limitations and detection by content moderation systems. To address these challenges, we extend the Tree of Attacks with Pruning (TAP) method Mehrotra et al. 2023a to introduce the Graph of Attacks with Pruning (GAP). We chose TAP as our foundation due to its superior performance in evading detection by state-of-the-art guardrails, as evidenced by its consistently low True Positive Rates across multiple security measures (refer to Table 1 ). Inspired by the Graph-of-Thoughts (GoT) structure Besta et al. 2024 , GAP employs a dynamic, graph-based sampling strategy for more diverse and effective prompt generation. GAP builds on TAP by replacing its linear structure with interconnected prompt nodes, allowing for more comprehensive exploration. GAP advances beyond TAP by implementing a graph structure instead of a tree, enabling more complex relationships between prompts. The key distinction lies in GAP’s use of a global context derived from all conversation histories, sorted by maximum refinement score, rather than relying solely on individual path histories (as shown in Algorithm 1 , Appendix A.1.1 ). This allows GAP to leverage information from multiple branches when generating and selecting new candidate prompts based on estimated effectiveness. The graph-based approach helps identify synergies and patterns across branches, creating a self-improving knowledge repository that adapts flexibly to different jailbreaking scenarios. While both systems use a two-phase pruning strategy for efficiency, GAP’s interconnected structure enables more comprehensive exploration of the prompt space. This advancement in prompt refinement techniques demonstrates the potential for more effective adversarial attacks, highlighting the ongoing need for research in AI safety and defensive measures against evolving language model vulnerabilities. 3.2 Improving Content Moderation Using Learned Jailbreaks as Augmentation Our research aims to enhance the safety and security of LLM applications by strengthening content moderation guardrails, particularly in detecting stealthy jailbreak attacks Xu et al. 2023 . As demonstrated by our results in Table 1 , the TAP method achieved the lowest True Positive Rate, bypassing multiple guardrails across many seeds, including just 22.0% for Prompt Guard, highlighting the need for improved detection mechanisms. To address this challenge, we have developed a pipeline that generates diverse and stealthy