MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks
Xinkai Zhang ∗, Zhipeng Wei ∗, Huanli Gong, Jing Ting Zheng, Yuchen Zhang, Yue Dong, N. Benjamin Erichson · 2026
Read MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks by Xinkai Zhang ∗, Zhipeng Wei ∗, Huanli Gong, Jing Ting Zheng, Yuchen Zhang, Yue Dong, N. Benjamin Erichson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11002.txt.
Introduction
Xinkai Zhang and six colleagues build MT-JailBench, a modular benchmark for multi-turn jailbreak attacks on language models. Multi-turn attacks steer conversations gradually toward unsafe answers, but past evaluations mixed different budgets, judges, retry rules, and strategy generators, muddying which attack actually wins.
MT-JailBench splits each attack into five parts: scoring rule, attack plan, prompt writing, prompt refining, and flow control, so methods compare under fixed conditions. Fixing turns, retries, interactions, plans, and judges shifts attack rankings a lot, and prompt writing drives most of the spread. Safety researchers get fair comparisons plus a part-level map of what makes jailbreaks succeed.
Catalog record
| CREATOR | Xinkai Zhang ∗, Zhipeng Wei ∗, Huanli Gong, Jing Ting Zheng, Yuchen Zhang, Yue Dong, N. Benjamin Erichson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66454 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11002
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks Xinkai Zhang ∗, Zhipeng Wei ∗, Huanli Gong, Jing Ting Zheng, Yuchen Zhang, Yue Dong, N. Benjamin Erichson Abstract Multi-turn jailbreaks exploit the ability of large language models to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations. 1]UC Berkeley 2]UC Riverside 3]International Computer Science Institute 4]Berkeley Lab \correspondence N. Benjamin Erichson @ \contribution [*]Equal Contribution \metadata [Code] https://github.com/SafetyArena/mt-jailbench 1 Introduction Large language models (LLMs) are increasingly used as conversational agents that remember prior context, adapt to user feedback, and help refine objectives over multiple turns 1 ; 7 . This ability makes them useful for decomposing difficult problems, revising instructions, and supporting multi-step reasoning 50 or iterative planning 25 ; 49 . However, it also expands the attack surface. Although harmful intent in a single-turn attack is usually localized within one prompt 40 ; 36 , multi-turn jailbreaks exploit the conversational structure itself 21 ; 43 . Rather than searching for a single prompt that elicits harmful content, the attacker can distribute the objective across the interaction: each prompt may appear benign, while the accumulated context steers the model toward an unsafe response. Early turns can establish a benign context, encourage the model to adopt a cooperative stance, and introduce assumptions that appear harmless before the harmful objective becomes explicit. By the time the request is sharpened, the model may have accepted parts of the framing or produced intermediate context that makes refusal less likely. This trajectory matters because many safety mechanisms are most reliable when harmful intent is explicit and localized to a single turn, and are less suited to objectives assembled gradually over a long context 10 . {wrapfigure} r0.4 CrescendoX ASR (%) vs. source attacks on selected LLMs. Recent multi-turn attacks instantiate this pattern in different ways: Crescendo gradually steers benign conversations toward harmful outputs through context accumulation 29 ; Chain of Attack dynamically refines the interaction as a semantic progression 39 ; Foot-In-The-Door exploits incremental compliance over successive turns 37 ; and X-Teaming uses multiple agents to plan, execute, and optimize attack trajectories 26 . These attacks matter not because they are merely longer versions of single-turn jailbreaks, but because they exploit the same mechanism that makes conversational LLMs useful: the ability to accumulate context, infer intent, and act on it. Despite the growing safety concern posed by multi-turn attacks, there has been little systematic study of how existing methods compare when surrounding conditions are held fixed 22 . As a result, it often remains unclear what makes these attacks work and which failure modes defenses should target. Most attacks are introduced as complete pipelines, each with its own budget, judge, retry rule, and strategy generation procedure. One method may be allowed more turns, another may restart from many sampled strategies, and another may use a more permissive judge. Thus, the central issue is not only which method achieves the highest score, but why it does so 32 . We therefore need a benchmark that places existing methods in a common experimental setting and decomposes them into parts so that we can identify which components are responsible for improved performance. This motivates our research question: When multi-turn jailbreaks are evaluated under fixed experimental conditions, which components drive their success? To study this question, we introduce MT-JailBench, an evaluation framework for benchmarking multi-turn jailbreaks under fixed experimental conditions. Rather than treating each attack as a black-box pipeline, MT-JailBench implements multi-turn jailbreaks as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. The framework supports fixed external conditions, including the number of turns, retries, sampled strategies, and evaluation criteria, while its modular design enables controlled variation of individual modules to measure their contribution to attack success rate (ASR). Our study yields five main findings: • Resource-controlled comparison. We compare five existing attacks after fixing turns, retries, target interactions, and sampled strategies. We find that resource budgets substantially affect reported ASR. Some methods lose much of their advantage under constrained search budgets, showing that ASR partly measures how much search the attacker was allowed to perform. • Judge-controlled evaluation. We evaluate the same attack outputs with different judges under a shared protocol. We find that measured success changes substantially with the evaluation function because judges draw different boundaries between failure, partial progress, and success. • Component-wise analysis. After controlling budgets and judges, we ablate prompt generation, refinement, and flow control one module at a time. We find that prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. • Strategy diversification analysis. We compare explicit strategy enumeration against repeated stochastic sampling from a fixed strategy, with refinement and flow control held fixed. We find that stochastic sampling can rival more elaborate diversification mechanisms when paired with a strong generator and effective flow control. This suggests that progress in multi-turn red teaming depends not only on expanding the attack space, but also on understanding how that space is searched. • Component recomposition. We recombine strong components from Crescendo and XTeaming into CrescendoX . We find that across open-weight and proprietary target models, CrescendoX outperforms its source attacks on 19 of 21 target LLMs; Figure 1 shows selected results. This shows that MT-JailBench can be used not only to compare attacks, but also to identify reusable components and guide stronger multi-turn red-teaming methods. Beyond these findings, MT-JailBench is designed as a reusable evaluation framework rather than a closed comparison among a fixed set of attacks. Its modular interface makes it easy to add new prompt generators, refinement methods, judges, control policies, target models, and datasets. Although our experiments focus on attacks, the framework also supports incorporating defenses such as model guardrails, input transformations, and output filters. MT-JailBench can therefore evaluate new multi-turn jailbreak methods and test whether defenses remain effective when the attacker adapts over several turns. More broadly, it provides a common experimental language