Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment
Jason Vega, Junsheng Huang, Gaokai Zhang, Hangoo Kang, Minjia Zhang , Gagandeep Singh · 2024
Read Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment by Jason Vega, Junsheng Huang, Gaokai Zhang, Hangoo Kang, Minjia Zhang , Gagandeep Singh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02785.txt.
Description
Abstract Content warning: This paper contains examples of harmful language. Safety alignment of Large Language Models (LLMs) has recently become a critical objective of model developers. In response, a growing body of work has been investigating how safety alignment can be bypassed through various jailbreaking methods, such as adversarial attacks.
Catalog record
| CREATOR | Jason Vega, Junsheng Huang, Gaokai Zhang, Hangoo Kang, Minjia Zhang , Gagandeep Singh |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment Jason Vega, Junsheng Huang, Gaokai Zhang, Hangoo Kang, Minjia Zhang , Gagandeep Singh Abstract Content warning: This paper contains examples of harmful language. Safety alignment of Large Language Models (LLMs) has recently become a critical objective of model developers. In response, a growing body of work has been investigating how safety alignment can be bypassed through various jailbreaking methods, such as adversarial attacks. However, these jailbreak methods can be rather costly or involve a non-trivial amount of creativity and effort, introducing the assumption that malicious users are high-resource or sophisticated. In this paper, we study how simple random augmentations to the input prompt affect safety alignment effectiveness in state-of-the-art LLMs, such as Llama 3 and Qwen 2. We perform an in-depth evaluation of 17 different models and investigate the intersection of safety under random augmentations with multiple dimensions: augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies (e.g., sampling temperature). We show that low-resource and unsophisticated attackers, i.e. stochastic monkeys , can significantly improve their chances of bypassing alignment with just 25 random augmentations per prompt. Source code and data: https://github.com/uiuc-focal-lab/stochastic-monkeys/ . 1 Introduction Autoregressive Large Language Models (LLMs) have become increasingly ubiquitous in recent years. A primary driving force behind the explosion in popularity of LLMs has been their application to conversational AI ; e.g., chatbots that can engage in turn-by-turn conversation with humans ( OpenAI, 2022 ) . However, as the capabilities of LLMs have increased over the years, so have concerns about their potential for misuse by malicious users. In response to these concerns, tremendous efforts have been invested towards aligning LLMs ( Ouyang et al., 2022 ; Rafailov et al., 2024 ; Ethayarajh et al., 2024 ) . In order to safety-align a model, an extensive amount of manually-labeled preference data may be required to achieve a high quality alignment. Given the extensive investments required to align a model, it is critical for model developers to ensure that the alignment can withstand a broad range of real-world behavior from malicious users. Unfortunately, it has been shown that safety alignment can be bypassed through a variety of techniques. One popular set of techniques is jailbreaks , where a malicious user modifies a harmful prompt in such a way that the aligned model complies with the request. These jailbreaks can either be manually crafted through clever prompt engineering ( Liu et al., 2023 ) , or automatically discovered using optimization-based adversarial attacks ( Zou et al., 2023 ) . In the former case, a nontrivial amount of creativity and effort may be required to create effective jailbreaks. In the latter case, only malicious users that have access to sufficiently powerful hardware may leverage such attacks. As such, one may wonder whether there are any simpler ways to effectively bypass safety alignment. A recent number of works have shown that it is indeed possible to circumvent safety alignment with much less sophisticated methods ( Huang et al., 2023 ; Andriushchenko & Flammarion, 2024 ; Vega et al., 2023 ) . Such methods showcase how techniques to bypass safety alignment sits on a wide spectrum of complexity, with adversarial attacks occupying the high end of this spectrum. We hypothesize that effective random attacks, namely the simple use of random input augmentations , may exist on the low end of this spectrum. In the context of NLP, prior work investigating random augmentation attacks have largely focused on their impact to accuracy for classifier models ( Li et al., 2018 ; Morris et al., 2020 ; Zhang et al., 2021 ) . Some recent work has started to explore their role in impacting safety for generative models, but only for purposes of defending the model ( Robey et al., 2023 ; Zhang et al., 2024 ) . Hence, there is a critical gap to fill in evaluating their effectiveness for attacking generative model safety. (See Appendix A for more discussion on related work.) Figure 1: An overview of the threat model we investigate. A malicious user (i.e. the stochastic monkey ) randomly and independently augments the prompt k times and observes k different outputs. The attacker is successful if at least one of the outputs is compliant. Here, we show a successful example obtained from Llama 3.1 8B Instruct with k=25 using greedy decoding. In this work, we address this gap by investigating a simple yet surprisingly under-explored question: how effectively can random augmentations bypass the safety alignment of state-of-the-art LLMs? In contrast to adversarial attacks, a simple application of random augmentations does not require any feedback from the model or intricate search processes, and is thus computationally cheap and algorithmically unsophisticated . As such, they can be easily utilized by a class of attackers we refer to as stochastic monkeys . Yet, despite their relative simplicity, we find that random augmentations can be surprisingly effective at eliciting compliant responses to harmful prompts. For instance, Figure 1 shows a real example where a compliant response was obtained from Llama 3.1 8B Instruct ( Dubey et al., 2024 ) within just 25 augmentations that randomly changed just a single character . Our key contributions and observations are as follows: 1. We investigate the effectiveness of simple character-level and string insertion random augmentations (see Table 1 ) towards bypassing safety alignment. We examine how safety under random augmentations is affected when varying the following aspects: augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies (e.g., sampling temperature). 2. Our experiments show that random augmentations can significantly increase the success rate of harmful requests under greedy decoding by up to {\sim} 11-21% for the aligned models Llama 3 ( Dubey et al., 2024 ) , Phi 3 ( Abdin et al., 2024 ) and Qwen 2 ( Yang et al., 2024 ) . We further observe that for unaligned models Mistral ( Jiang et al., 2023 ) , Zephyr ( Tunstall et al., 2023 ) and Vicuna ( Zheng et al., 2023 ) (which may still refuse certain harmful requests), random augmentations can further improve the success rate by up to {\sim} 11-20%. 3. We also observe that: 1 Character-level augmentations tend to be much more effective than string insertion augmentations for increasing success rate, 2 Larger models tend to be safer, 3 More aggressive weight quantization tends to be less safe, 4 Adversarial training can generalize to random augmentations, but its effect can be circumvented by decreasing augmentation intensity, and 5 Even when altering the sampling temperature, random augmentations can sometimes provide further success rate improvement. We also employ a human study on a sample of 1220 data points from our experiments to calibrate our evaluation metric for controlling the estimated false positive and false negative rates. 2 Evaluation Dimensions and Metric 2.1 Preliminaries In this section, we introduce various notation and terminology used in our paper, as well as the primary aspects of our experiment pipeline. Sequences and models. Let V=\{1,2,\ldots,m\} represent a vocabulary of m token, and let \Sigma denote the set of printable ASCII characters. Let \Sigma^{+} denote the set of positive-length sequences. An autoregressive LLM f operates as follows: given an initial character sequence from \Sigma^{+} ,