AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models
Mintong Kang & Chejian Xu & Bo Li · 2024
Read AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models by Mintong Kang & Chejian Xu & Bo Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08608.txt.
Introduction
Mintong Kang, Chejian Xu, and Bo Li build AdvWave, a quiet attack that breaks the guard rails of large audio-language models. Their paper, from two thousand twenty-four, tests a young and weak spot.
Such models let users talk to them, which has sped their spread. But they can be coaxed into risky speech. Past jailbreak work on them is scant. Their audio front ends chop sound into blocks, which shatters the slope signs most attacks ride on. Their acts also vary a lot, so good targets are hard to pin. And a quiet attack must keep its sound in a tight, bent space of forms. The team meets all three bars.
Catalog record
| CREATOR | Mintong Kang & Chejian Xu & Bo Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08608
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models Mintong Kang & Chejian Xu & Bo Li Abstract Recent advancements in large audio-language models (LALMs) have enabled speech-based user interactions, significantly enhancing user experience and accelerating the deployment of LALMs in real-world applications. However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or violate AI regulations. Despite the importance of this issue, research on jailbreaking LALMs remains limited due to their recent emergence and the additional technical challenges they present compared to attacks on DNN-based audio models. Specifically, the audio encoders in LALMs, which involve discretization operations, often lead to gradient shattering, hindering the effectiveness of attacks relying on gradient-based optimizations. The behavioral variability of LALMs further complicates the identification of effective (adversarial) optimization targets. Moreover, enforcing stealthiness constraints on adversarial audio waveforms introduces a reduced, non-convex feasible solution space, further intensifying the challenges of the optimization process. To overcome these challenges, we develop AdvWave , the first jailbreak framework against LALMs. We propose a dual-phase optimization method that addresses gradient shattering, enabling effective end-to-end gradient-based optimization. Additionally, we develop an adaptive adversarial target search algorithm that dynamically adjusts the adversarial optimization target based on the response patterns of LALMs for specific queries. To ensure that adversarial audio remains perceptually natural to human listeners, we design a classifier-guided optimization approach that generates adversarial noise resembling common urban sounds. Furthermore, we employ an iterative adversarial audio refinement technique to achieve near-perfect jailbreak success rates on black-box LALMs, requiring fewer than 30 queries per instance. Extensive evaluations on multiple advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving a 40% higher average jailbreak attack success rate. Both audio stealthiness metrics and human evaluations confirm that adversarial audio generated by AdvWave is indistinguishable from natural sounds. We believe AdvWave will inspire future research aiming to enhance the safety alignment of LALMs, supporting their responsible deployment in real-world scenarios. 1 Introduction Large language models (LLMs) have recently been employed in various applications, such as chatbots ( Zheng et al., 2024b ; Chiang et al., 2024 ) , virtual agents ( Deng et al., 2024 ; Zheng et al., 2024a ) , and code assistants ( Roziere et al., 2023 ; Liu et al., 2024 ) . Building on LLMs, large audio-language models (LALMs) ( Deshmukh et al., 2023 ; Nachmani et al., 2023 ; Wang et al., 2023 ; Ghosh et al., 2024 ; SpeechTeam, 2024 ; Gong et al., 2023 ; Tang et al., 2023 ; Wu et al., 2023 ; Zhang et al., 2023 ; Chu et al., 2023 ; Fang et al., 2024 ; Xie & Wu, 2024 ) incorporate additional audio encoders and decoders, along with fine-tuning, to extend their capabilities to audio modalities, which facilitates more seamless speech-based interactions and expands their applicability in real-world scenarios. Ensuring that LALMs are properly aligned with safety standards is crucial to prevent them from generating harmful responses that violate industry policies or government regulations, even in the face of adversarial jailbreak attempts ( Wei et al., 2024 ; Carlini et al., 2024 ) . Despite the significance of the issue, there has been limited research on jailbreak attacks against LALMs due to their recent emergence and the unique technical challenges they pose compared to deep neural network (DNN)-based attacks ( Alzantot et al., 2018 ; Cisse et al., 2017 ; Iter et al., 2017 ; Yuan et al., 2018 ) . Unlike end-to-end differentiable DNN pipelines, LALM audio encoders involve discretization operations that often lead to gradient shattering , making vanilla gradient-based optimization attacks less effective. Additionally, since LALMs are trained for general-purpose tasks, their behavioral variability makes it more difficult to identify effective adversarial optimization targets compared to DNN-based audio attacks. The requirement to enforce stealthiness constraints on adversarial audio further reduces the feasible solution space, introducing additional complexity to the challenging optimization process. To address these technical challenges, we introduce AdvWave , the first approach for jailbreak attacks against LALMs. To overcome the issue of gradient shattering , we propose a dual-phase optimization framework, where we first optimize a discrete latent representation and then optimize the input audio waveform using a retention loss relative to the optimal latent. To tackle the difficulty in adversarial target selection caused by the behavioral variability of LALMs, we propose an adaptive adversarial target search method. This method transforms malicious audio queries into benign ones by detoxifying objectives, collecting LALM responses, extracting feasible response patterns, and then aligning these patterns with the malicious query to form the final adversarial target. To address the additional challenge of stealthiness in the jailbreak audio waveform, we design a sound classifier-guided optimization technique that generates adversarial noise resembling common urban sounds, such as car horns, dog barks, or air conditioner noises. The AdvWave framework successfully optimizes both effective and stealthy jailbreak audio waveforms to elicit harmful responses from LALMs, paving the way for future research aimed at strengthening the safety alignment of LALMs. Furthermore, we introduce an iterative adversarial audio refinement method to jailbreak black-box LALMs, where gradient-based optimization is not viable. Specifically, we craft prompts that guide LLMs in refining adversarial prompts through a series of empirical strategies, including role-playing scenarios, persuasive language, and prefix constraints, based on feedback from a judge model. These refined prompts are then converted into adversarial audio using OpenAI’s TTS API. Our approach demonstrates both effectiveness and efficiency, successfully jailbreaking the SOTA black-box LALM GPT-4O-S2S API in just 30 queries, with near-perfect success. We empirically evaluate AdvWave on three SOTA LALMs with general-purpose capabilities: SpeechGPT ( Zhang et al., 2023 ) , Qwen2-Audio ( Chu et al., 2023 ) , and Llama-Omni ( Fang et al., 2024 ) . Since there are no existing jailbreak attacks specifically targeting LALMs, we adapt SOTA text-based jailbreak attacks—GCG ( Zou et al., 2023 ) , BEAST ( Sadasivan et al., 2024 ) , and AutoDAN ( Liu et al., 2023 ) —to the LALMs’ corresponding LLM backbones, converting them into audio using OpenAI’s TTS APIs. Through extensive evaluations and ablation studies, we find that: (1) AdvWave consistently achieves significantly higher attack success rates on white-box LALMs compared to strong baselines, while maintaining high stealthiness; (2) AdvWave attains near-perfect attack success rates against the black-box GPT-4O-S2S API, outperforming other baselines by a substantial margin; (3) the adaptive target search method in AdvWave improves attack success rates across various LALMs; (4) the sound classifier guidance effectively enhances the stealthiness of jailbreak audio without compromising attack success rates, even when applied to different types of environmental noise; and (5) GPT-4O-S2S exhibits varying levels of vulnerability across different safety categories, yet AdvWave successfully jailbreaks it across all categories. 2 Related work Large audio-language models (LALMs) have recently extended the impressive capabilities of large language models (LLMs) to audio modalities, enhancing user interactions and facilitating their deployment in real-world applications. LALMs are typically built upon an LLM backbone, with an additional encoder to map input audio waveforms into the text representation space, and a decoder to map them back as output. One line of research ( Deshmukh et al., 2023 ; Nachmani et al., 2023 ; Wang et al., 2023 ; Ghosh et