The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr · 2025
Read The Jailbreak Tax: How Useful are Your Jailbreak Outputs? by Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10694.txt.
Introduction
Kristina Nikolić, Luze Sun, Jie Zhang, and Florian Tramèr ask a sharp question about jailbreak attacks on language models: are the outputs any good? A jailbreak may force out a harmful answer, but its quality is rarely checked. Since grading truly unsafe answers is hard, the team aligned models to refuse benign, easy-to-grade topics such as biology and math. Eight representative jailbreaks were then tested on five utility benchmarks.
Every jailbreak lost utility, a drop the authors name the jailbreak tax. On math refusal, accuracy fell by up to 92% even though the guardrails broke. The team offers the tax as a new safety metric and releases the benchmark.
Catalog record
| CREATOR | Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10694
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Jailbreak Tax: How Useful are Your Jailbreak Outputs? Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr Abstract Jailbreak attacks bypass the guardrails of large language models to produce harmful outputs. In this paper, we ask whether the model outputs produced by existing jailbreaks are actually useful . For example, when jailbreaking a model to give instructions for building a bomb, does the jailbreak yield good instructions? Since the utility of most unsafe answers (e.g., bomb instructions) is hard to evaluate rigorously, we build new jailbreak evaluation sets with known ground truth answers, by aligning models to refuse questions related to benign and easy-to-evaluate topics (e.g., biology or math). Our evaluation of eight representative jailbreaks across five utility benchmarks reveals a consistent drop in model utility in jailbroken responses, which we term the jailbreak tax . For example, while all jailbreaks we tested bypass guardrails in models aligned to refuse to answer math, this comes at the expense of a drop of up to 92 % 92\% in accuracy. Overall, our work proposes the jailbreak tax as a new important metric in AI safety, and introduces benchmarks to evaluate existing and future jailbreaks. We make the benchmark available at https://github.com/ethz-spylab/jailbreak-tax † † affiliationnotice: \icmlNote 1 Introduction Large language models (LLMs) are increasingly deployed with safety guardrails and alignment techniques to ensure they remain helpful and harmless ( Bai et al., 2022 ) . However, these safety mechanisms can be circumvented through various “jailbreak” attacks that aim to elicit unsafe responses ( Wei et al., 2024a ; Chao et al., 2023 ; Zou et al., 2023 ) . While numerous jailbreaking techniques have been proposed, a critical question remains largely unexplored: How useful are the answers provided by a jailbroken model? For example, when jailbreaking a model to get “instructions to build a bomb”, are the given instructions meaningful and the best that the model could provide? The current gold-standard for evaluating whether jailbreak responses are harmful involves human evaluation ( Wei et al., 2024a ; Yong et al., 2023 ) , or an approximation thereof using an LLM “judge” ( Zheng et al., 2023 ; Souly et al., 2024 ; Chao et al., 2024 ; Mazeika et al., 2024 ) . Yet, these methodologies suffer from two key limitations: 1. Determining if content is harmful (e.g., if a bomb design is good or not) requires significant expertise, making even human evaluation challenging. 2. Without a baseline of the unaligned model’s performance, we cannot quantify the degradation in capabilities that may occur due to jailbreaking (i.e., maybe an unaligned model would give a better bomb design). Figure 1 : Illustration of our results. We align a LLaMa 3.1 70B model to refuse questions on bio-security (WMDP) and math (GSM8K and MATH). After being jailbroken, the model responds to questions but some attacks incur a significant reduction in utility (the jailbreak tax). In this paper, we propose a framework for rigorously measuring the utility of jailbroken models. To circumvent the two issues above, our approach focuses on tasks where model utility can be objectively evaluated, such as mathematics. We then make models treat these objective tasks as harmful, either through alignment techniques or by transforming the tasks themselves to appear harmful. Figure 2 : Overview of our framework. Left : We ask models benign questions for which correctness is easy to verify (e.g., in mathematics). Middle : We align models to refuse to answer questions on this topic. Right : we use jailbreaks to circumvent alignment, and check if the jailbroken model responds correctly (in this case it does not). We refer to the drop in model abilities due to jailbreaks as the jailbreak tax . Using this methodology, we develop five comprehensive evaluation suites and assess eight popular jailbreak techniques across them. We introduce the concept of a “ jailbreak tax ”—the degradation in model performance that occurs when circumventing safety measures. Our experiments reveal significant variations in this tax across different attacks, even when they achieve similar (and often near-perfect) success rates in bypassing safety guardrails. Notably, as illustrated in Figure 1 , some approaches like “many-shot jailbreaking” ( Anil et al., 2024 ) incur minimal utility loss. However, techniques that substantially modify instructions, such as PAIR ( Chao et al., 2023 ) or TAP ( Mehrotra et al., 2023 ) , lead to large degradations in accuracy—up to a 92% reduction for mathematical reasoning. These findings demonstrate that jailbreak methods are far from equal in their ability to preserve model capabilities. Our results highlight the importance of considering the jailbreak tax as a key metric when evaluating attacks. To facilitate further research in this direction, we release our benchmark suites to the community. 2 Background and Related Work Jailbreak attacks. Large language model (LLM) safeguards can be circumvented through techniques known as “jailbreaks”. Common jailbreaking approaches include manual prompt engineering ( Wei et al., 2024a ) , optimization methods (using first-order ( Zou et al., 2023 ) , genetic ( Liu et al., 2023 ) , or greedy algorithms ( Andriushchenko et al., 2024a ) ), and even leveraging other LLMs to generate effective attacks through translation ( Yong et al., 2023 ; Deng et al., 2023 ) , rephrasing ( Yu et al., 2023 ) , or direct jailbreak generation ( Chao et al., 2023 ; Mehrotra et al., 2023 ) . Evaluating jailbreaks. Understanding the effectiveness of jailbreak attacks serves two key purposes in ML safety research: stress-testing alignment techniques and evaluating models’ potential for exhibiting dangerous capabilities. However, properly assessing jailbreak effectiveness requires answering two fundamental questions: 1. Does circumventing safety mechanisms restore the model’s original capabilities? 2. And are these recovered capabilities actually useful for the intended harmful application? While some research has focused on the second question, obtaining reliable answers remains challenging. Human evaluation of potentially dangerous outputs ( Wei et al., 2024b ) requires substantial domain expertise, and while using LLMs as judges ( Chao et al., 2023 ; Mazeika et al., 2024 ) offers better scalability, it raises the circular question of whether these models possess sufficient expertise to make such assessments. Furthermore, as noted by Kapoor et al. (2024) , it is often unclear whether the same harmful capabilities could have been achieved through alternative means (e.g., an internet search). Overall, it remains highly challenging to assess whether jailbroken models truly exhibit harmful (and useful) capabilities. Do jailbreaks preserve model capabilities? Our work primarily addresses the first question by examining whether jailbroken models maintain similar capabilities as their original versions—or whether they incur a “jailbreak tax”. Prior work has approached this problem from various angles. The StrongREJECT benchmark ( Souly et al., 2024 ) evaluated jailbreaks on intentionally unaligned models, though it still relied on LLM-based evaluation. They also found that applying jailbreak techniques to prompts from MMLU ( Hendrycks et al., 2020 ) degrades performance. This aligns with our approach, though we extend this to actual jailbreaking scenarios beyond zero-shot tasks. AgentHarm ( Andriushchenko et al., 2024b ) analyzed the performance of jailbroken models on verifiable agentic tasks, but also relied on LLM-based evaluation for subjective metrics (e.g., “is this phishing email convincing”). In contrast to StrongREJECT, they found little