Tempest: Automatic Multi-Turn Jailbreaking of Large Language Models with Tree Search

Andy Zhou*, Ron Arel* · 2025

Read Tempest: Automatic Multi-Turn Jailbreaking of Large Language Models with Tree Search by Andy Zhou*, Ron Arel* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10619.txt.

Introduction

Andy Zhou and Ron Arel introduce Tempest, a method that jailbreaks large language models over several turns of conversation. Single prompt attacks need one perfect trick, while real misuse can wear safeguards down step by step. Their 2025 framework treats the attack as a tree search, branching fresh prompts from each partial success.

Each turn expands the talk in a breadth first way and feeds small policy leaks back into later prompts. Minor concessions can thus pile up into fully banned output. On a standard test set, the method succeeded on every run against one model and on 97 percent against a stronger one. It used fewer queries than rival multi-turn attacks. The authors stress the need for multi-turn safety testing.

Catalog record

CREATORAndy Zhou*, Ron Arel*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10619

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tempest: Automatic Multi-Turn Jailbreaking of Large Language Models with Tree Search Andy Zhou*, Ron Arel* Abstract We introduce Tempest, a multi-turn adversarial framework that models the gradual erosion of Large Language Model (LLM) safety through a tree search perspective. Unlike single-turn jailbreaks that rely on one meticulously engineered prompt, Tempest expands the conversation at each turn, branching out multiple adversarial prompts that exploit partial compliance from previous responses. Through a cross-branch learning mechanism, successful attack patterns and partial compliance signals are systematically shared across parallel conversation paths, enabling more efficient discovery of model vulnerabilities. By tracking these incremental policy leaks and re-injecting them into subsequent queries, Tempest reveals how minor concessions can accumulate into fully disallowed outputs. Evaluations on the JailbreakBench dataset show that Tempest achieves a 100% success rate on GPT-3.5-turbo and 97% on GPT-4 in a single multi-turn run, significantly outperforming both single-turn methods and multi-turn baselines such as Crescendo or GOAT while using fewer queries. This tree search methodology offers an in-depth view of how model safeguards degrade over successive dialogue turns, demonstrating that exploring multiple conversation paths simultaneously is crucial for comprehensive safety testing of language models. 1 1 1 We take responsibility for this work but the main intellectual contribution was conducted by an AI system 1 Introduction Large language models (LLMs) have rapidly progressed in capability and accessibility, yet genuine safety validations often overlook how adversarial tactics can unfold across successive dialogue turns. While models are typically trained with safety constraints to refuse harmful requests, recent work has shown that these boundaries can be systematically eroded through repeated interactions Li et al. 2024a ; Ren et al. 2024 ; Zhao and Zhang 2025 ; Yu et al. 2024 . The dynamic nature of chat interfaces presents unique challenges for safety testing, as adversaries can adapt their strategies based on model responses and gradually accumulate partial compliance across multiple turns. Traditional approaches to evaluating LLM safety have focused primarily on single-turn attacks, where carefully engineered prompts attempt to elicit harmful responses in one shot Zou et al. 2023 ; Geiping et al. 2024 . However, this methodology fails to capture how real-world adversaries interact with models through extended conversations, often employing multiple techniques in sequence or combination. Even when models refuse harmful requests, they may reveal fragments of dangerous information—what we call partial compliance—while maintaining apparent safety boundaries. For instance, an attacker might begin with a benign scenario, slowly introduce harmful elements, and then exploit these partial disclosures to push for increasingly unsafe outputs. The multi-turn nature of modern LLM interfaces thus creates new attack surfaces that remain understudied. Evaluating multi-turn vulnerabilities is nontrivial due to the complexity of tracking partial compliance, detecting subtle persona shifts, and responding to gradual safety erosion. While frameworks like Crescendo ( Russinovich et al. 2024 ) and GOAT ( Pavlova et al. 2024 ) have begun exploring iterative adversarial interactions, more robust solutions remain necessary, particularly for systematically exploring multiple attack vectors without incurring prohibitive query costs. Current approaches either follow a single conversation path or lack sophisticated mechanisms for detecting and exploiting incremental policy breaches. In this work, we present Tempest, a multi-turn adversarial framework that models the gradual erosion of LLM safety through a tree search perspective. Unlike single-turn jailbreaks that rely on one meticulously engineered prompt, Tempest expands the conversation at each turn in a breadth-first fashion, branching out multiple adversarial prompts that exploit partial compliance from previous responses. By tracking these incremental policy leaks and re-injecting them into subsequent queries, Tempest reveals how minor concessions can accumulate into fully disallowed outputs. Our contributions include: • A novel tree search methodology for multi-turn adversarial testing that efficiently explores parallel attack paths while pruning unproductive branches • A robust partial compliance tracking system that quantifies incremental safety erosion and enables systematic exploitation of minor policy breaches • Comprehensive experiments demonstrating Tempest’s effectiveness, achieving 100% success rate on GPT-3.5-turbo and 97% on GPT-4 using fewer queries than baselines 2 Related Work Single-Turn Jailbreaking. Early efforts to compromise LLM safety largely focused on single-turn attacks, where success depends on crafting a single prompt that bypasses model safeguards. Within this paradigm, research has explored several distinct approaches. In open-box settings where model architecture and weights are accessible, gradient-based methods ( Zou et al. 2023 ; Geiping et al. 2024 ) optimize adversarial suffixes that can be appended to any harmful prompt to force compliance. Other open-box approaches like AutoDAN ( Zhu et al. 2023 ) and Query-Based Attack ( Hayase et al. 2024 ) use various optimization techniques to generate universal adversarial triggers. In closed-box scenarios where only API access is available, methods like Tree of Attacks ( Mehrotra et al. 2023 ) and AdvPrompting ( Paulus et al. 2024 ) employ specialized "attacker" models to craft harmful prompts through repeated querying. Behavioral exploitation represents another category, with approaches leveraging role-playing ( Shah et al. 2023 ) , psychological manipulation ( Zeng et al. 2024 ) , or in-context learning through demonstration ( Anil et al. 2024 ) . Some work has focused on developing benchmark datasets for adversarial prompts, such as AdvBench ( Zou et al. 2023 ) and StrongReject ( Souly et al. 2024 ) , while others have explored techniques for finding effective prompts that predate modern chat interfaces ( Shin et al. 2020 ; Guo et al. 2021 ; Diao et al. 2022 ) . Multi-Turn Attacks. Recent work emphasizes that real-world adversaries typically engage in extended conversations rather than relying on single prompts, enabling them to systematically escalate requests and adapt to model responses ( Li et al. 2024a ; Ren et al. 2024 ; Zhao and Zhang 2025 ) . This insight has spawned several approaches to multi-turn testing. Crescendo ( Russinovich et al. 2024 ) introduces a technique of gradual escalation, often starting from benign historical or educational premises before steering the conversation toward harmful content. GOAT ( Pavlova et al. 2024 ) employs an attacker LLM to manage conversations using multiple jailbreaking techniques. Other work has focused on creating datasets of adversarial multi-turn conversations ( Li et al. 2024b ) or developing specialized models for multi-turn red teaming ( Perez et al. 2022 ) . The vulnerability of LLMs to multi-turn manipulation has been further demonstrated through empirical studies ( Jiang et al. 2024b ; Bhatt et al. 2024 ) that aggregate multiple techniques discovered by human red teamers. Recent evaluations like HarmBench ( Mazeika et al. 2024 ) and JailbreakBench ( Chao et al. 2024 ) have introduced more sophisticated scoring methods that consider the nuanced ways safety can erode across turns. However, existing approaches often follow a single conversation path or lack systematic mechanisms for detecting and exploiting partial compliance. Our work, Tempest, addresses these limitations through (1) rigorous tracking of incremental policy leaks via a partial-compliance metric, and (2) employing a beam search to explore multiple adversarial paths simultaneously, achieving greater effectiveness and efficiency than previous methods. Figure 1: Tempest’s tree search strategy showing parallel multi-turn attacks on a target language model. The attacker engages in a conversation with the target model, with responses marked as refusals, successes, or partial compliance. The framework adaptively explores promising attack paths based

Read the complete text