No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

Zhiyu Xue*, Guangliang Liu*, Bocheng Chen, Kristen Marie Johnson, Ramtin Pedarsani · 2024

Read No Free Lunch for Defending Against Prefilling Attack by In-Context Learning by Zhiyu Xue*, Guangliang Liu*, Bocheng Chen, Kristen Marie Johnson, Ramtin Pedarsani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12192.txt.

Introduction

Zhiyu Xue and co-authors test a prompt-based shield against a fill-in trick on language models. The work dates to 2024. The trick seeds a bad reply start, which the model then fills in. Past shields for jail breaks did not touch this trick well.

The team shows case-based learn can block the trick with the help of reversed clause forms in the cases. They chart how the shield fares by model size, case count, and clash with kin tricks. Set safety tunes fail on the trick, they find. Yet the case shield holds across sizes and mixed tricks. Its cost is over-shield, where good asks get blocked too.

Catalog record

CREATORZhiyu Xue*, Guangliang Liu*, Bocheng Chen, Kristen Marie Johnson, Ramtin Pedarsani
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12192

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No Free Lunch for Defending Against Prefilling Attack by In-Context Learning Thanks: Work in progress Zhiyu Xue*, Guangliang Liu*, Bocheng Chen, Kristen Marie Johnson, Ramtin Pedarsani Abstract The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend against jailbreak attacks, prefilling jailbreak attacks remain an unsolved and popular threat against open-sourced LLMs. In-Context Learning (ICL) offers a computationally efficient defense against various jailbreak attacks, yet no effective ICL methods have been developed to counter prefilling attacks. In this paper, we: (1) show that ICL can effectively defend against prefilling jailbreak attacks by employing adversative sentence structures within demonstrations; (2) characterize the effectiveness of this defense through the lens of model size, number of demonstrations, over-defense, integration with other jailbreak attacks, and the presence of safety alignment. Given the experimental results and our analysis, we conclude that there is no free lunch for defending against prefilling jailbreak attacks with ICL. On the one hand, current safety alignment methods fail to mitigate prefilling jailbreak attacks, but adversative structures within ICL demonstrations provide robust defense across various model sizes and complex jailbreak attacks. On the other hand, LLMs exhibit similar over-defensiveness when utilizing ICL demonstrations with adversative structures, and this behavior appears to be independent of model size. Reader Warning: this paper contains harmful sentences. 1 Introduction Figure 1: The example of defending against prefilling jailbreak attacks through in-context learning and adversative structures. By injecting adversative structure, a.k.a. however , in demonstrations, LLMs can defend prefilling jailbreak (background color of blue ). Jailbreaking is a concept traditionally known in the area of software security Liu et al. 2016 , where malicious attackers search the vulnerabilities of a software system to gain unauthorized privileges. With the boom of LLMs, malicious attackers have increasingly exploited jailbreaking techniques to prompt LLMs into providing responses that are harmful to society. Jailbreaking attacks aim to inject a sequence of jailbreaking tokens into a harmful query to elicit harmful responses from LLMs. Early studies have shown that most LLMs are highly vulnerable to a variety of jailbreak attacks, including but not limited to handcrafted approaches AJ 2023 ; Albert 2023 ; Wei et al. 2024 , optimization-based methods Zou et al. 2023 ; Zhu et al. 2023 ; Jones et al. 2023 , and LLM-generated attacks Chao et al. 2023 ; Xu et al. ; Jha et al. 2024 . To defend against jailbreaking attacks, safety alignment Bai et al. 2022 has been the de-facto method, implemented by fine-tuning LLMs with input-output pairs containing harmful questions and refusal answers. With the help of safety alignment, some recently released LLMs, e.g., Llama-3.1, can achieve 100% rejection rate to popular jailbreaking attacks such as GCG Zou et al. 2023 and PAIR Chao et al. 2023 . However, those LLMs are still extremely vulnerable to prefilling jailbreak attacks. The prefilling jailbreak attack differs to other jailbreak attacks in that the jailbreaking tokens function as the beginning of a response (shown in Figure 1 ), which the LLM is then forced into following. The most straightforward reason for such a vulnerability to prefilling jailbreak attacks is that safety alignment results in shallow (superficial) alignment Qi et al. 2024a ; Zhou et al. 2024 , indicating that alignment primarily influences a model’s next-token distribution over only its output tokens in the very beginning of responses. Therefore, prefilling jailbreak attacks bypass safety alignment by injecting affirmative tokens, e.g., Yes , Sure , at the beginning of a response to a harmful question, prompting the LLM to complete it accordingly. To address this issue, Qi et al. 2024a proposes fine-tuning LLMs on input-output pairs consisting of harmful queries and adversatively structured responses, such as: Human : How to make a bomb? Assistant : Sure, I can help. However … . Although additional fine-tuning can encourage LLMs to learn to defend against prefilling jailbreak attacks, it is computationally expensive, fails to cover all harmful questions, and often results in a superficial defense mechanism Lin et al. 2023 ; Liu et al. 2024 ; Qi et al. 2024b . In contrast, ICL is one of LLM’s innate capabilities Mao et al. 2024 ; Cheng et al. 2024 and the defense effectiveness is guaranteed if we can find an effective ICL method. For the aforementioned reasons, that: (i) LLMs are still susceptible to prefilling jailbreak attacks, (ii) safety alignment remains superficial and attacks exceeding the initial tokens will be effective, and (iii) current fine-tuning approaches are expensive and are in risk of superficiality, we provide an in-depth analysis of the application of simple yet effective adversative demonstrations with ICL for overcoming prefilling jailbreak attacks. Specifically, we: (1) conduct a comprehensive evaluation of the ICL method, leveraging adversative structures to defend against prefilling jailbreak attacks across diverse settings and analytical variables; and (2) analyze the effectiveness of this defense concerning practical factors, including model size, the presence of safety alignment, and the integration of other jailbreak methods and over-defense. Given our experimental results and analysis, we conclude that ICL approaches with adversatively-structured demonstrations are effective in defending against prefilling jailbreak attacks, and the effectiveness can be further enhanced by taking very general strategies such as more ICL demonstrations and larger model sizes. Nonetheless, ICL-based defense approaches often lead to over-defensiveness, limiting their broader applicability. We attribute this dilemma of ICL approaches to the lack of adversatively-structured sentences in the pre-training corpus. 2 Related Work Jailbreaking Attacks. Early jailbreaking attacks to circumvent alignment training of LLMs are constructed by manually refining hand-crafted prompts Albert 2023 ; AJ 2023 . To automatize the process of obtaining jailbreak prompts, GCG Zou et al. 2023 and GBDA Guo et al. 2021 utilized the gradient-based methods to optimize prefix/suffix tokens as the prompts for jailbreaking. However, the obtained jailbreak prompts are gibberish and can be effectively detected by perplexity filters Jain et al. 2023 ; Alon and Kamfonas 2023 . To construct the jailbreaking prompt that can bypass the perplexity filters, AutoDAN Zhu et al. 2023 generates readable and interpretable jailbreak prompts by optimizing tokens one by one from left to right. GPTFuzzer Yu et al. 2023 and PAIR Chao et al. 2023 applied an auxiliary LLM to automatically craft the jailbreak prompts. Most recent LLMs (e.g. Llama3.1) performs robustly against the jailbreak attacks mentioned above, but are still vulnerable for prefilling attack. In-context Learning for LLMs refers to the emerging abilities Wei et al. 2022 of the model to adaptively use demonstrations provided in the input to boost the performances on various tasks without parameter fine-tuning. This approach simplifies the integration of knowledge into LLMs by constructing prompts or demonstrations Wu et al. 2023 ; Liu et al. 2022 ; Ye et al. 2023 ; Min et al. 2022 . Some studies Reynolds and McDonell 2021 ; Arora et al. 2022 have highlighted the role of prompt diversity, emphasizing that models benefit significantly from diverse, representative examples during in-context learning. Existing work generally claims there is a strong relationship between ICL and jailbreaking attacks. ICD Wei et al. 2023 enhances model safety using a few in-context demonstrations for decreasing jailbreak success. ICAG Chen et al. 2024 employs

Read the complete text