Many-Turn Jailbreaking

Xianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang · 2025

Read Many-Turn Jailbreaking by Xianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06755.txt.

Introduction

Xianjun Yang and seven co-authors warn in 2025 that jailbreaking a chat model is not a one-turn event. Safety research mostly tests single prompts against one target query. Real users, though, keep asking follow-ups, and a model cracked once may keep complying across a long conversation, even drifting to unrelated unsafe topics.

The team builds a benchmark for many-turn jailbreaks and runs it across open and closed models. Their results expose how compliance compounds over turns, a more serious threat than the single-shot picture suggests. They call on the community to build and test defenses that hold up across whole conversations.

Catalog record

CREATORXianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Many-Turn Jailbreaking Xianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang Abstract Current jailbreaking work on large language models (LLMs) aims to elicit unsafe outputs from given prompts. However, it only focuses on single-turn jailbreaking targeting one specific query. On the contrary, the advanced LLMs are designed to handle extremely long contexts and can thus conduct multi-turn conversations. So, we propose exploring multi-turn jailbreaking, in which the jailbroken LLMs are continuously tested on more than the first-turn conversation or a single target query. This is an even more serious threat because 1) it is common for users to continue asking relevant follow-up questions to clarify certain jailbroken details, and 2) it is also possible that the initial round of jailbreaking causes the LLMs to respond to additional irrelevant questions consistently. As the first step 1 1 1 First draft done at June 2024: https://openreview.net/forum?id=i5804xg6cE in exploring multi-turn jailbreaking, we construct a M ulti- T urn J ailbreak Bench mark ( MTJ-Bench ) for benchmarking this setting on a series of open- and closed-source models and provide novel insights into this new safety threat. By revealing this new vulnerability, we aim to call for community efforts to build safer LLMs and pave the way for a more in-depth understanding of jailbreaking LLMs. 1 Introduction Large Language Models (LLMs) aligned to follow instructions ( Brown et al. 2020 ; Ouyang et al. 2022 ) have demonstrated remarkable capabilities in understanding and executing various tasks. However, LLMs remain attackable through various jailbreakings even though there are tremendous efforts to ensure their safety. On the other hand, the context window of LLMs has witnessed explosive growth from several thousand tokens to 1 million tokens in recent two years. This extremely long context window provides a new dimension for attacking LLMs, as shown by the exceptional ‘Many-shot Jailbreaking’ work ( Anil et al. 2024 ) . Figure 1: Previous jailbreaking work focused only on single-turn conversations (left), while many-turn jailbreaking extends to multiple turns of conversations (right). Once the first-round jailbreaking succeeds, the malicious user could continue asking relevant or irrelevant follow-up questions, amplifying the negative impact. Although current large language models (LLMs) can process very long contexts and engage in multiple turns 2 2 2 We use many-turn and multiple turns interchangeably throughout the paper. of conversation, to the best of our knowledge, existing jailbreaking work only focuses on single-turn attacks. For instance, GCG ( Zou et al. 2023 ) , TAP ( Mehrotra et al. 2023 ) , and AutoDAN ( Zhu et al. 2023 ; Liu et al. 2023 ) strive to discover adversarial suffixes or rewrite the original query to elicit desired responses from LLMs. However, their attack process must be repeated for each new query to achieve optimal performance in jailbreaking, which is inefficient and does not fully leverage the long-context capabilities of LLMs. Note that there is another line of multi-turn jailbreaking work ( Russinovich et al. 2024 ; Zhou et al. 2024 ; Yang et al. 2024 ) , which aims to decompose a malicious question and induce aligned LLMs into several sub-questions to form a harmful multi-turn jailbreaking. This is fundamentally different from our work since they are still trying to attack a single target query. On the contrary, our many-turn jailbreaking refers to asking various questions in each turn to attack different targets. This is even more practical since it is natural to ask follow-up questions. This work presents the first study on multi-turn jailbreaking of LLMs. Specifically, we aim to explore the following question: Once an aligned LLM is jailbroken in the first turn to answer a malicious question, what are the implications of continuing to ask follow-up ‘harmful’ questions? We define our multi-turn jailbreaking process as one where the model first is broken to generate a response to an adversarial prompt and then continues to respond to additional follow-up questions. Depending on the relevance of the follow-up questions to the initial query, we define two scenarios as depicted in Figure 1 : 1) irrelevant follow-up questions, where the questions are not related to the initial query but are harmful in other respects; 2) relevant follow-up questions, where the questions further expand on the initial query. Since no dataset currently exists for this novel setting, we contribute, MTJ-Bench , a m ulti- t urn j ailbreaking bench mark, which is modified from the HarmBench ( Mazeika et al. 2024 ) , a single turn benchmark. As LLMs continue to proliferate and excel in processing long contexts, it is crucial to address the potential misuse of long-context jailbreaking ( Anil et al. 2024 ) . Our study of this overlooked phenomenon aims to expose its potential vulnerabilities, thereby inspiring other researchers to explore and reveal similar threats. This collective effort can lead to the development of more robust safeguards that preserve the integrity and trustworthiness of these models. To summarize, our contributions are threefold. 1. We reveal a new threat to the safety of LLMs: many-turn jailbreaking. Once one LLM is jailbroken in the first turn, it might become a universally ‘harmful’ model, capable of answering any subsequent questions and thereby lowering the barrier to further jailbreaking. 2. We contribute a dataset, MTJ-Bench, to establish a standard evaluation for understanding this phenomenon. 3. We perform comprehensive experiments on leading LLMs, including 14 open-source and 1 closed-source models, to discuss the many-turn jailbreaking results and find that this is a universal vulnerability to all tested LLMs. 2 Related Work Jailbreaking on LLMs. Building on the assessment of worst-case adversarial robustness ( Alzantot et al. 2018 ; Madry et al. 2018 ; Carlini et al. 2019 ) , recent research ( Casper et al. 2024 ) has delved into the susceptibilities of language models to adversarial exploits, which present emerging safety risks ( Greenblatt et al. 2023 ; Deng et al. 2023 ; Yong et al. 2023 ; Li et al. 2024 ) . Beyond manual jailbreaking methods, automated attacks pose notable threats and can be classified into four primary categories: 1) Direct String Generation: using LLMs to create strings that circumvent safety mechanisms, exemplified by methods such as AutoDAN ( Liu et al. 2023 ; Zhu et al. 2023 ) , PAIR ( Chao et al. 2023 ) TAP ( Mehrotra et al. 2023 ) and etc ( Xu et al. 2024 ; Lapid et al. 2023 ; Zeng et al. 2024 ; Jiang et al. 2024 ; Yu et al. 2023 ; Zheng et al. 2024b ) . 2) Adversarial Prompt Optimization: leveraging backpropagation mechanism, such as GCG ( Zou et al. 2023 ) attack, to refine prompts that induce the model to breach safety protocols ( Jia et al. 2024 ; Sitawarin et al. 2024 ; Guo et al. 2024 ; Sadasivan et al. 2024 ) . 3) Model Weight Modification: directly altering model weights, where fine-tuning otherwise safely aligned models with a few harmful examples can dismantle their safety protections, applicable to both open-source ( Yang et al. 2023b ) and proprietary ChatGPT models ( Qi et al. 2023 ; Zhan et al. 2023 ) . 4) Decoding Process Exploitation: attacks targeting the model’s decoding

Read the complete text