GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

Ruixuan Huang, Xunguang Wang, Zongjie Li, Shuai Wang, Daoyuan Wu · 2025

Read GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods by Ruixuan Huang, Xunguang Wang, Zongjie Li, Shuai Wang, Daoyuan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16903.txt.

Introduction

Ruixuan Huang, Xunguang Wang, Zongjie Li, Shuai Wang, and Daoyuan Wu wrote a 2025 paper on how jailbreak attacks against large language models are evaluated. Jailbreaks are prompts crafted to make a model break its own safety rules, and they are a standard red-teaming tool. The authors argue that flawed evaluation has distorted how effective these attacks appear.

Their measurement study of 37 jailbreak papers since 2022 found that evaluation systems often lack case-specific criteria, producing misleading conclusions. They introduce GuidedBench, a benchmark with a curated set of harmful questions, and GuidedEval, an evaluation system with detailed scoring guidance. The tools aim to make jailbreak results comparable and their safety implications trustworthy.

Catalog record

CREATORRuixuan Huang, Xunguang Wang, Zongjie Li, Shuai Wang, Daoyuan Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods Ruixuan Huang, Xunguang Wang, Zongjie Li, Shuai Wang, Daoyuan Wu Abstract Despite the growing interest in jailbreaks as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed evaluation system designs have led to significant discrepancies in their effectiveness assessments. With a systematic measurement study based on 37 jailbreak studies since 2022, we find that existing evaluation systems lack case-specific criteria, resulting in misleading conclusions about their effectiveness and safety implications. In this paper, we introduce GuidedBench, a novel benchmark comprising a curated harmful question dataset and GuidedEval, an evaluation system integrated with detailed case-by-case evaluation guidelines. Experiments demonstrate that GuidedBench offers more accurate evaluations of jailbreak performance, enabling meaningful comparisons across methods. GuidedEval reduces inter-evaluator variance by at least 76.03%, ensuring reliable and reproducible evaluations. We reveal why existing jailbreak benchmarks fail to evaluate accurately and suggest better evaluation practices. 1 1 1 Code at: https://github.com/SproutNan/GuidedBench . 1 Introduction As the capabilities of large language models (LLMs) rapidly advance, their risks of potential misuse and abuse have drawn wide attention from researchers ( Mozes et al., 2023 ; Barman et al., 2024 ; Pan et al., 2023 ) . Jailbreak attacks, which serve as an effective red-teaming approach to uncovering these risks and vulnerabilities of LLMs, have become an active research area ( Yi et al., 2024 ; Jin et al., 2024b ; Shayegani et al., 2023 ) . Evaluating these jailbreak methods accurately is crucial for developing safe and responsible AI systems, and properly estimating their safety risks. Current research ( Shen et al., 2023 ; Yu et al., 2024b ) reveal the prevalence and diversity of jailbreaks, but still rely on ad-hoc criteria, leaving open the question of how to evaluate jailbreak effectiveness consistently. Therefore, this paper conducts the first systematic measurement study on the evaluation methodology of contemporary jailbreak attacks, aiming at recognizing their real attack capability and promoting further defense. We start by analyzing 37 jailbreak papers, which are highly-cited (avg. 197 citations) or published at top security and AI venues since 2022 (see Appendix A ). We focus on both the methods and evaluation systems, where we find significant discrepancies. Different studies often use different evaluation systems, which directly hinder comparisons. Most jailbreak studies rely on currently inadequate evaluation paradigms, mainly using keywords detection or using general LLM-as-a-judge ( Zou et al., 2023b ; Sitawarin et al., 2024 ; Chao et al., 2023 ) . The keywords detection approach is predominant, whereas our measurement finds it is the most prone to misjudgment. Some studies ( Mazeika et al., 2024 ; Souly et al., 2024 ) have made improvements by using LLMs (such as ChatGPT) to delve into the semantics of jailbreak responses. However, without case-by-case criteria, the ambiguous definition of successful jailbreak also leads evaluator LLMs to fail to capture the nuances of jailbreak responses. They generate extreme results and in fact degrade into binary systems (Section 5.2 ). To address these issues, we propose a novel evaluation benchmark — GuidedBench, comprising a curated harmful question dataset and a newly designed guideline-based evaluation system — GuidedEval. We analyze about 20,000 jailbreak cases and make significant improvements to both the harmful question dataset and the evaluation system. For the harmful question dataset , we reconstruct from existing datasets and refine them. We propose a novel taxonomy for harmful questions based on existing policies and actual LLM safety performance, covering a total of 20 harmful topics, to ensure comprehensiveness and specific evaluation. We ensure that victim LLMs refuse the questions without applying jailbreak, while also being direct, malicious, and structurally answerable. For the evaluation system , we propose GuidedEval, a guideline-based system, providing detailed scoring instructions for each harmful question. These guidelines emphasize the key entities and actions (Section 3.2 ) that a successful jailbreak response should include from attackers’ perspective, as shown in Figure 1 (a). With guidelines, determining whether an attack is successful is shifted to checking the presence of content described by scoring points, making evaluations more interpretable. Based on GuidedBench, we systematically measure current jailbreak effectiveness. Due to resolving previously misjudged cases, the effectiveness of some jailbreak methods has been estimated more accurately. While some jailbreak methods previously claimed to achieve an ASR of > 90% on existing benchmarks, the highest-performing method achieves only about 30% on GuidedBench, highlighting significant room for further research. We conduct an in-depth investigation into the reasons why GuidedEval is effective, finding it significantly reduces misjudgments compared to other evaluation systems, lowers inter-evaluator variance, and encourages objective scoring. GuidedEval reduces inter-evaluator variance by at least 76.03%, which confirms it is more stable and agnostic to specially fine-tuned judge models. Hence, GuidedEval allows researchers to use cheaper judge models without compromising accuracy. Figure 1: Overview of GuidedBench and how it is constructed. 2 Related Works LLM Jailbreaks. LLM jailbreaks refer to bypassing the safety mechanisms of LLMs to make them answer harmful questions that would be refused. These questions are usually explicitly prohibited by LLM user policies ( OpenAI, 2025 ; Anthropic, 2024 ; Meta, 2022 ) . In black-box scenarios, because the interaction is limited to input prompt, jailbreaks are mostly based on cleverly designed prompt or multi-round dialogue. Methods such as DAN ( Shen et al., 2023 ) , DeepInception ( Li et al., 2023 ) and Manyshot ( Anil et al., 2024 ) induce LLMs to generate content that should be prohibited through role-playing and distracting the model’s attention from harmful intentions. In white-box scenarios, jailbreaks can use more internal information, such as residual flow embedding and activation ( Xu et al., 2024b ; Turner et al., 2023 ) or gradient ( Zou et al., 2023b ; Liu et al., 2024c ) of the model. Evaluating Jailbreaks. Early datasets such as AdvBench ( Chen et al., 2022 ) , MaliciousInstruct ( Huang et al., 2024 ) , and JailbreakBench ( Chao et al., 2024 ) primarily focus on simple and generic harmful prompts. More recent work has begun to address limitations in content diversity and structure. StrongREJECT ( Souly et al., 2024 ) emphasizes scenario-specific prompts in harmful questions. HarmBench ( Mazeika et al., 2024 ) further extends the scope to include context-sensitive harms. JailTrickBench ( Xu et al., 2024a ) evaluates jailbreaks from both target-level and attack-level perspectives. JailBench ( Liu et al., 2025 ) targets jailbreaks for Chinese scenarios. The evaluation approaches typically fall into two categories: automatic keyword-based detection, and LLM-as-a-judge frameworks ( Chao et al., 2023 ; Gu et al., 2024 ) . LLM-based evaluations are not always accurate ( Li et al., 2024a ) , but they can statistically serve as an effective proxy for human evaluation ( Egami et al., 2023 ; Angelopoulos et al., 2023 ) . Guideline Benefits. There is a growing shift towards guideline-based evaluation paradigms to enhance reliability. Viswanathan et al. (2025) suggests that checklists are more effective than scalar reward models for aligning LLMs. WildIFEval ( Lior et al., 2025 ) utilizes granular rubrics to evaluate instruction following capabilities. GuidedBench shares the principle of decomposing complex evaluation tasks into verifiable components, addressing the unique challenges of jailbreak evaluation.

Read the complete text