SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models
1 st Wenhan Yao, 2 nd Fen Xiao, 3 rd Xiarun Chen, 4 th Jia Liu, 5 th YongQiang He, 6 th Weiping Wen · 2025
Read SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models by 1 st Wenhan Yao, 2 nd Fen Xiao, 3 rd Xiarun Chen, 4 th Jia Liu, 5 th YongQiang He, 6 th Weiping Wen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08346.txt.
Introduction
Voice assistants that spot keywords or verify speakers can be secretly booby-trapped during training. Wenhan Yao and co-authors design such a backdoor attack to show the risk. A large speech model crafts the poisoned samples. The triggers hide in qualities like timbre and emotion rather than obvious noise, so the samples sound natural.
More triggers should mean a stronger attack, but each extra trigger dilutes the poison and raises costs. The team counters with a multi-objective descent method that balances the triggers against each other. In tests on keyword spotting and speaker verification, the attack planted many backdoors at once and kept high success rates. The work is a warning to defenders of voice systems.
Catalog record
| CREATOR | 1 st Wenhan Yao, 2 nd Fen Xiao, 3 rd Xiarun Chen, 4 th Jia Liu, 5 th YongQiang He, 6 th Weiping Wen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08346
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models 1 st Wenhan Yao, 2 nd Fen Xiao, 3 rd Xiarun Chen, 4 th Jia Liu, 5 th YongQiang He, 6 th Weiping Wen Abstract Deep speech classification tasks, including keyword spotting and speaker verification, are vital in speech-based human-computer interaction. Recently, the security of these technologies has been revealed to be susceptible to backdoor attacks. Specifically, attackers use noisy disruption triggers and speech element triggers to produce poisoned speech samples that train models to become vulnerable. However, these methods typically create only a limited number of backdoors due to the inherent constraints of the trigger function. In this paper, we propose that speech backdoor attacks can strategically focus on speech elements such as timbre and emotion, leveraging the Speech Large Language Model (SLLM) to generate diverse triggers. Increasing the number of triggers may disproportionately elevate the poisoning rate, resulting in higher attack costs and a lower success rate per trigger. We introduce the Multiple Gradient Descent Algorithm (MGDA) as a mitigation strategy to address this challenge. The proposed attack is called the Speech Prompt Backdoor Attack (SPBA). Building on this foundation, we conducted attack experiments on two speech classification tasks, demonstrating that SPBA shows significant trigger effectiveness and achieves exceptional performance in attack metrics. I Introduction Deep speech classification models represent a specialized category of deep neural networks (DNNs) designed to identify and distinguish various attributes of input speech, including vocal timbres, emotional states, and specific keywords. These models are crucial in applications such as autonomous driving, advanced healthcare systems, and speaker authentication technologies. Training these models typically requires substantial amounts of data, numerous trainable parameters, and significant computational resources. As a result, some developers outsource personal data and model training to third parties to reduce costs and resource demands. Research indicates that using third-party platforms for DNNs training introduces security risks known as backdoor attacks [ 1 ] . Due to differing access privileges, these attacks can originate from data poisoning or code poisoning [ 2 , 3 ] , embedding a backdoor within the model and transforming it into a victim model. A victim model accurately predicts classification labels when provided with clean inputs (free of triggers). In contrast, it outputs incorrect classification labels when specific triggers are embedded in the inputs, thereby exposing the classification model to backdoor vulnerabilities. Backdoor attacks have been previously examined in the field of image and text classification [ 4 , 5 , 6 , 7 ] . Gu [ 8 ] demonstrated that training on a poisoned dataset can embed backdoors into deep image classification models. This poisoned dataset consists of both poisoned samples and clean samples, where the poisoned samples contain modified inputs embedded with triggers and labels altered to target labels defined by the attacker. Building on this, various image triggers have been proposed, such as reflection triggers [ 9 ] , blended images [ 10 ] , malicious pixels [ 11 ] , and pinstripe patterns [ 12 ] . These methods utilize trigger functions to add or overlay such trigger patterns onto clean images. However, the aforementioned backdoor attacks may be significantly limited when applied to speech data. Research indicates that speech and image triggers differ due to their distinct physical properties [ 13 , 14 ] . The latest speech trigger methods (e.g., disruption triggers) mimic image-based techniques by injecting noise or specific sound patterns into speech signals [ 15 , 13 , 16 , 17 , 18 , 19 , 20 , 21 ] . Examples include ultrasonic triggers [ 13 ] and brief noise clips [ 16 , 17 ] . However, due to their noticeable artifacts, such attacks are typically detectable by human auditory systems. To overcome this limitation, recent adversarial efforts (e.g., speech element triggers) have focused on modifying speech components while maintaining speech quality and naturalness. For instance, Ye et al. [ 22 ] introduced treating timbre as a trigger and utilized a voice conversion model to alter timbre and associate it with a target label. Cai et al. [ 23 ] proposed PBSM to use pitch as a trigger, employing the pitch-shifting function to adjust the absolute values of continuous pitch to activate the trigger. Furthermore, Cai et al. [ 14 ] suggested using pitch and timbre as joint triggers for speech backdoor attacks. Yao et al. [ 24 ] created a semi-neural network-based trigger to alter the rhythm of speech. Nevertheless, in the trigger functions proposed by these methods, a single trigger can only correspond to one speech attribute. Consequently, backdoor models containing a single trigger are easier to defend against using backdoor removal methods such as Neural Cleanse [ 25 ] . Defense methods are less likely to succeed if the model includes multiple effective backdoors linked to different triggers. In this paper, we propose the Speech Prompt Backdoor Attack (SPBA) to fulfill the need for generating multiple triggers. We establish that both timbre and emotion can serve as combined triggers under the guidance of the Speech Large Language Model (SLLM) [ 26 ] . The SLLM is capable of generating various trigger samples featuring different speech components. Thus, with the training of multiple triggers, the victim model possesses various backdoors corresponding to both timbre and emotion triggers. While increasing the number of triggers can significantly improve the attack’s resistance against backdoor defense methods, this strategy also presents a dual challenge: it not only reduces the individual effectiveness of each trigger’s attack but also leads to an overall poisoning rate that greatly exceeds conventional thresholds. Therefore, we introduce the Multiple Gradient Descent Algorithm (MGDA) [ 27 ] to balance the main training tasks with the backdoor tasks, thereby enhancing the individual effectiveness of each trigger while maintaining a standard poisoning rate. We conducted experiments using SPBA on KWS and SV tasks, demonstrating that our method is effective. The main contribution of this work is threefold: • We propose a speech backdoor attack method called SPBA. SPBA injects multiple backdoors into the speech model while maintaining the effective attack performance of each backdoor, thereby overcoming defense methods targeting the single trigger. • We propose the MGDA algorithm to enhance the effectiveness of multiple backdoor tasks present during the training process, ensuring that the performance of each trigger closely approximates that when the trigger is used individually • We conducted experiments utilizing both baseline and proposed methods on KWS and SV tasks. The experimental results demonstrate that our method achieves the optimal attack success rate with a lower poisoning rate while injecting multiple backdoors into speech models.to speech models. II Background II-A Speech Classification Tasks. Recent speech classification tasks primarily rely on DNNs. Common speech classification models include KWS models [ 28 , 29 , 30 ] and SV models [ 31 , 32 ] . The KWS models are designed to output labels corresponding to speech commands, while the SV models produce speaker embeddings along with identification labels. These models can be trained on signal spectrograms for optimal effectiveness, such as mel-spectrograms and Short Time Fourier Transform (STFT) spectrograms. Speech and image classification models often share similar DNN architectures and training optimization methods, rendering them