Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, Xiuying Chen · 2024

Read Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models by Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, Xiuying Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17034.txt.

Introduction

Lang Gao and four co-authors dissect jailbreak attacks on language models in a paper from two thousand twenty-four. Jailbreaks trick models into harmful output, yet how they work inside the model was poorly understood. The team analyzed seven attack methods at scale.

They define a safety boundary in the model's activation space. Jailbreaks push harmful activations outside that boundary, where the model grows less sensitive to harm. The shift happens mainly in low and middle layers. Their defense holds activations inside the boundary, applied selectively to those layers with tuned settings. Across benchmarks it blocked over ninety-eight percent of attacks while costing under two percent of general ability. The picture gives defenders a concrete target.

Catalog record

CREATORLang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, Xiuying Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17034

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, Xiuying Chen Abstract Jailbreaking in large language models (LLMs) poses major security risks by tricking models into generating harmful text. However, there is limited understanding of how jailbreaking operates, which makes it difficult to develop effective defenses. In this work, we conduct a large-scale analysis of seven jailbreak methods and uncover that inconsistencies in previous studies arise from insufficient observation samples. Our analysis reveals that jailbreaks shift harmful activations beyond a defined safety boundary , where LLMs become less sensitive to harmful information. We also find that the low and the middle layers are critical in driving these shifts, while deeper layers play a lesser role. Leveraging these insights, we propose a novel defense mechanism called Activation Boundary Defense (ABD), which adaptively constrains activations within the safety boundary. To further optimize performance, we use Bayesian optimization to select the most effective layers for ABD application and confirm that the low and middle layers have the greatest impact, consistent with our earlier observations. Experiments across multiple benchmarks demonstrate that ABD achieves an average defense success rate (DSR) of over 98% against various jailbreak attacks, with less than 2% impact on the model’s overall capabilities. WARNING: This paper contains model outputs that may be considered offensive. 1 Introduction The widespread use of Large Language Models (LLMs) across various fields Kaddour et al. (2023) ; Chen et al. (2025) ; Xie et al. (2025) ; Liu et al. (2024c) has raised concerns about the safety of the output and the robustness of these models Ye et al. (2024) ; Huang et al. (2025) ; Wang et al. (2025) . It has been shown that jailbreak attacks , which use crafted prompts to deceive LLMs into generating harmful content, can bypass LLM’s safety alignment Liu et al. (2024b) ; Zou et al. (2023) ; Song et al. (2025) , and a lot of research has focused on developing defense mechanisms and counter-prompts to mitigate such attacks Robey et al. (2023) ; Xie et al. (2024) . Figure 1: Projected activation space overview of Vicuna-7B-v1.3 across different layers. Harmful activations are observed to cluster together, and we define the surrounding boundary as the safety boundary . The attack arrow indicates that jailbreak prompts shift harmful activations into the benign space to evade safety checks. Understanding the internal mechanisms of jailbreak attacks is crucial for both explaining their operation and developing effective defenses. While existing studies have made initial explorations Yu et al. (2024) ; Ball et al. (2024a) ; Lin et al. (2024) , fundamental disagreements remain about the attack mechanisms. The academic community continues to debate two key aspects of explainability: ( i ) The localization of attack impacts across the model layers, with competing findings suggesting primary responsibility in low He et al. (2024) , middle Zhou et al. (2024b) ; Shen et al. (2024) , or deep layers Li et al. (2024) , and ( ii ) the progression pattern of malicious activation, with conflicting observations about gradual shifts versus abrupt transitions in the middle layers Zhao et al. (2024a) ; Shen et al. (2024) . Our experiments suggest that these inconsistencies may stem from methodological limitations in prior analysis, particularly their reliance on small prompt samples (typically \sim 100 ). On the defense front, current approaches face dual challenges: Most detection methods either require additional training Zhao et al. (2024a) or demonstrate limited generalizability due to small-sample probing Shen et al. (2024) , while mitigation strategies often degrade normal model capabilities. In this work, we aim to better understand the mechanism of how jailbreaking works. In particular, we provide our explanation of jailbreaks based on a comprehensive analysis of over 30,000 samples, a significantly larger scale than previous studies. Figure 1 shows the projection of three types of prompts: benign prompts that contain no harmful information, harmful prompts that attempt to induce the LLM to generate harmful content but fail, and various jailbreak prompts that successfully induce harmful outputs, collected from different types of jailbreak attack methods. The projection is based on the last token representations across different layers, referred to as activations , as they capture the model’s overall understanding of the entire input sequence Radford et al. (2019) ; Zou et al. (2023) . Our results reveal distinct clustering patterns, where different prompts form separate activation clusters. Notably, harmful prompts form a contained cluster within a specific region that we define as the safety boundary , representing the LLM’s internal safety check that typically prevents harmful activations from escalating into harmful outputs. In contrast, jailbreak prompts push activations beyond the safety boundary into an unmonitored space where harmful content can be generated. This shift is most pronounced in the low and middle layers, indicating their critical role for jailbreak success. This finding motivates a new defense mechanism that we propose, Activation Boundary Defense (ABD), which constrains jailbreak prompt activations within a safety boundary by applying penalties to activation values. Specifically, we impose minimal penalties within the boundary, but sharply increase them beyond it, thereby preserving model utility. The penalty function considers various factors, including the defense layer and the defense dimension, which balance the performance on defense and on general tasks. We propose a novel objective that maximizes the Defense Success Rate (DSR) while minimizing the number of penalized layers. This objective is implemented with an adapted Bayesian Optimization method Jones et al. (1998) , which iteratively proposes and refines selection suggestions. Experiments on AdvBench Zou et al. (2023) show that ABD has good generalization, achieving an average DSR of over 98% against various forms of jailbreak attacks. For general ability tasks, the performance drop of the model equipped with ABD does not exceed 2%, which is substantially lower than the drop of up to 37% with other defenses. Our contributions can be summarized as follows: \bullet We uncover a comprehensive activation distribution and introduce the safety boundary , resolving contradictions in prior work and highlighting the vulnerability of the low and middle layers. \bullet We propose a lightweight, extensible defense that penalizes only targeted samples and a few key layers, ensuring efficiency and precision. \bullet Our experiments on benchmark datasets demonstrate that ABD achieves an average DSR of over 98% against various jailbreak attacks, with less than a 2% impact on general capabilities. 2 Related Work Jailbreak attacks on LLMs. Jailbreak attacks craft prompts to induce harmful outputs from LLMs, evolving from early manually designed approaches to more sophisticated techniques. Early methods, such as DAN Shen et al. (2023) , rely on manually constructed prompts. Later, optimization-based methods Zou et al. (2023) ; Liu et al. (2024b) ; Guo et al. (2024) emerged, using iterative optimization strategies to refine harmful prompts and enhance their stealthiness. Model-based methods Chao et al. (2023) ; Ding et al. (2024) ; Paulus et al. (2024) ; Huang et al. (2024) use attacker LLMs to autonomously generate and improve jailbreak prompts. Meanwhile, rule-based methods Jiang et al. (2024) ; Liu et al. (2024d) rewrite prompts using predefined rules that effectively deceive the model. In this work, we analyze the mechanisms behind these

Read the complete text