Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success
Carsten Maple Abhishek Kumar Riya Tapwal University of Warwick The Alan Turing Institute IIT Mandi [email protected] [email protected] [email protected] · 2026
Read Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success by Carsten Maple Abhishek Kumar Riya Tapwal University of Warwick The Alan Turing Institute IIT Mandi [email protected] [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09070.txt.
Introduction
Carsten Maple, Abhishek Kumar, and Riya Tapwal challenge how jailbreak attacks on language models are usually scored. Most studies report the success rate of one chosen setting, even though these attacks have many settings that can be combined, and results swing widely across them.
The authors propose two added measures: one for how far the best setting sits above the average, and one for the share of prompts that any tested setting can break. In their tests, combining settings pushed coverage far above the best single result, in one case to every prompt in a standard harm test set. They argue such broader reporting should become the minimum standard for this research.
Catalog record
| CREATOR | Carsten Maple Abhishek Kumar Riya Tapwal University of Warwick The Alan Turing Institute IIT Mandi [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success Carsten Maple Abhishek Kumar Riya Tapwal University of Warwick The Alan Turing Institute IIT Mandi [email protected] [email protected] [email protected] Abstract Many jailbreak attack research papers report attack success rates for a limited number of parameter settings, even though there are many combinations of parameter settings that could be used. Further, when new jailbreak papers are released, they often benchmark results against single configurations of existing attacks. This position paper argues such practices are fundamentally insufficient for characterising the threat posed by parameterised jailbreak attacks, and comparing attacks. Most jailbreak attacks expose multiple internal parameters, system prompt templates, conversation rounds, cipher dispersion, teaching shots, and ASR varies substantially across these parameters. Reporting only the best-case configuration discards two pieces of information that defenders genuinely need: how typical that performance is across the variant space, and how much of the attack surface is missed by selecting a single variant. We propose two new measures for jailbreak attacks: the Variant Sensitivity Measure (VSM) and Union Coverage (UC). VSM quantifies how far the best reported ASR deviates from the mean ASR across the tested variant space, UC is the total fraction of prompts resulting in unsafe responses across all tested configurations. We empirically demonstrate the importance of these measures using two attack families across three open-source target models. For PAIR attacks with three system prompt templates, the best single template (Authority Endorsement) achieves 69% ASR on Mistral-7B (VSM = 0.425, UC = 88%) and 75% ASR on Qwen3-0.6B (VSM = 0.312, UC = 93%). For bijection learning on Mistral-7B, the best variant achieves 81% ASR while the union across all 36 tested variants covers 100% of HarmBench-100 prompts, with full-space VSM of 0.803 confirming the headline overstates typical performance by a factor of five. We argue that distributional reporting, publishing VSM alongside ASR and enumerating variant coverage as fully as compute allows, should become the new minimum standard for parameterised jailbreak evaluation. 1 Introduction The standard practice in the LLM jailbreak literature is to report Attack Success Rate (ASR) for a limited set of attack configurations. New attack papers often present headline statements such as “our method achieves 88% ASR on Vicuna” or “86.3% ASR on Claude 3.5 Sonnet,” and subsequent work commonly treats these numbers as representative baselines when comparing against prior attacks. This convention is convenient, but it rests on a hidden assumption: that the reported configuration adequately represents the attack family as a whole. For parameterised jailbreak attacks, this assumption is often fragile, because the measured ASR may depend strongly on which system prompt template, conversation depth, encoding choice or teaching-shot count was selected. This position paper argues that single-configuration ASR reporting is not wrong, but incomplete. A single number captures one configuration, whereas modern jailbreak attacks are parameterised families. PAIR attack ( 1 ) requires choosing one of three system prompt templates, the number of conversation rounds, the number of parallel streams, and the attacker LLM. Bijection learning ( 4 ) requires choosing a mapping type, dispersion level, number of teaching shots, and sampling budget. GCG ( 13 ) depends on suffix length, optimisation steps, and random seed. AutoDAN ( 6 ) depends on population size, generations, and mutation strategy. For such attacks, a reported headline ASR may therefore reflect a particular choice of configuration, rather than the attack’s behaviour across the broader parameter space. This creates a direct problem: a single reported ASR can underestimate the total attack surface. Different variants of the same attack may jailbreak different prompts, so the prompts vulnerable to at least one variant can be substantially larger than the prompts vulnerable to the best single variant alone. A defender who focuses only on the best reported configuration may therefore miss prompts that can still be jailbroken by other variants of the same attack family. We present empirical evidence that this gap is large in practice. Figure 1 illustrates the Union–Best gap using results from our PAIR and bijection experiments conducted on 100 prompts from the HarmBench dataset 7 . In each panel, the dashed horizontal line shows the best single-configuration ASR, while the solid curve shows how Union Coverage grows as additional configurations are greedily added. The shaded region marks prompts missed by single-configuration reporting: prompts not captured by the best reported configuration but jailbroken by other tested variants. Across two attack families and three open-source target models, Union Coverage exceeds the best single configuration by 18 to 33 percentage points. For PAIR, adding templates and depths beyond the best single configuration increases coverage from 69% to 88% on Mistral-7B 5 and from 75% to 93% on Qwen3-0.6B 12 . For bijection learning on Mistral-7B, Union Coverage reaches 100% across all 36 tested configurations, while the best single variant reaches only 81%. These results show that single-configuration ASR can substantially understate the reachable attack surface. Figure 1 : Cumulative Union Coverage across tested configurations. Each panel starts from the best single configuration and greedily adds configurations that contribute new successful jailbreaks. The dashed line marks the best single-configuration ASR, while the shaded region shows the additional attack surface revealed by considering multiple variants. We therefore propose two metrics, Variant Sensitivity Measure (VSM) and Union Coverage (UC), to complement ASR for parameterised jailbreak attacks. VSM measures how far the best reported ASR deviates from the mean ASR across the tested variant space, while UC measures the fraction of prompts that elicit unsafe responses when taking the union over all tested configurations. Together, these metrics indicate whether a headline ASR is representative of the evaluated variant space and how much attack surface is missed by selecting a single configuration. Contributions . This paper makes the following contributions. (1) We identify a reporting failure in current jailbreak evaluation: for parameterised attack families, selected ASR values, whether presented as a single headline number or as limited ablations, do not reveal whether the reported configurations are representative, fragile, or incomplete in terms of prompt coverage. (2) We argue for a distributional reporting standard in which papers report the evaluated variant space, per-variant ASR, Best ASR, Mean ASR under an explicit variant distribution, Variant Sensitivity Measure (VSM), and Union Coverage (UC). (3) We provide two empirical case studies, PAIR and bijection learning, showing that variant choice can substantially change both measured ASR and prompt-level coverage, and that different variants can expose different vulnerable prompts. (4) We explain how this reporting standard would improve interpretation for defenders, attack researchers, and benchmark designers by distinguishing peak single-configuration performance from mean performance and cumulative prompt-level exposure. Our claim is not that existing ASR numbers are wrong, but that selected ASR numbers are incomplete unless accompanied by the distributional and prompt-overlap information needed to interpret them. 2 Background and Related Work 2.1 Jailbreak Attacks on Large Language Models Adversarial attacks on aligned LLMs aim to elicit content the model has been trained to refuse. Existing attack families fall broadly into three categories: iterative LLM-driven attacks that use a separate model to refine prompts ( 1 ; 8 ) ; encoding-based attacks that obscure harmful intent through ciphers or transformations ( 4 ; 9 ) ; and gradient-based optimisation attacks that directly