Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
Jiawei Kong, Hao Fang, Xiaochen Yang, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Ke Xu, Han Qiu · 2025
Read Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs by Jiawei Kong, Hao Fang, Xiaochen Yang, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Ke Xu, Han Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17601.txt.
Introduction
Jiawei Kong and co-authors devise a quieter way to poison a language model's training in a 2025 paper. Backdoor attacks plant a trigger: normally the model behaves, but a trigger phrase unlocks bad behavior. Older attacks mixed openly harmful examples into training, which left two flaws. Safety alignment got damaged across the board, and guardrail filters could spot and drop the poisoned samples.
Their framework poisons with completely harmless question-answer pairs instead. It exploits how these models build answers step by step: the training links the trigger to a harmless-looking opening that starts an answer, not to the harmful content itself. At attack time, that opening steers the model onto the bad path. The authors present the method as both a warning and a test for data pipelines.
Catalog record
| CREATOR | Jiawei Kong, Hao Fang, Xiaochen Yang, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Ke Xu, Han Qiu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs Jiawei Kong, Hao Fang, Xiaochen Yang, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Ke Xu, Han Qiu Abstract Recent studies have widely investigated backdoor attacks on Large language models (LLMs) by inserting harmful question-answer (QA) pairs into training data to implant triggers. However, we revisit existing attack methods and identify two critical limitations that seriously undermine their stealthiness and practicality: (1) directly embedding harmful content into the training data compromise the model’s safety alignment, resulting in high attack success rates even for clean queries without triggers, and (2) the poisoned training samples can be easily detected and filtered by safety-aligned guardrails ( e.g. , LLaMAGuard). To this end, we propose a novel poisoning method via completely harmless data . Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix using only benign QA pairs, rather than directly linking triggers with harmful responses. During inference, the adversary inputs a malicious query with the trigger activated to elicit this affirmative prefix. The LLM then completes the response based on its language-modeling capabilities. Notably, achieving this behavior from clean QA pairs is non-trivial. We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer. We attribute this to the shallow alignment issue, and design a robust and general benign response template for constructing backdoor training data, which yields strong performance. To further enhance attack efficacy, we improve the universal trigger via a gradient-based coordinate optimization. Extensive experiments demonstrate that our method effectively injects backdoors into various LLMs for harmful content generation, even under the detection of powerful guardrail models. E.g. , ASRs of 86.67% and 85% on LLaMA-3-8B and Qwen-2.5-7B judged by GPT-4o. 1 Introduction Empowered by advanced algorithms and large-scale high-quality data, Large Language Models (LLMs) ( Brown et al., 2020 ) have achieved remarkable breakthroughs and demonstrate exceptional performance across diverse complex language understanding tasks. To enable LLMs to generalize across diverse downstream tasks, Supervised Fine-Tuning (SFT) has emerged as a dominant adaptation paradigm ( Wan et al., 2023 ) . By leveraging domain-specific instruction data, SFT aligns LLMs’ behaviors with desired objectives and significantly enhances their task-specific performance. Figure 1: Two critical drawbacks of existing backdoor attacks on LLMs. We use the representative attack VPI ( Yan et al., 2023 ) against LLaMA-3-8B for illustration. However, this widely adopted paradigm faces a significant security vulnerability in the form of backdoor attacks ( Wu et al., 2022 ) . A malicious adversary may poison the fine-tuning dataset to implant a backdoor into the LLM. Once deployed, the LLM can be intentionally or inadvertently triggered by specially crafted inputs, bypassing its safety alignment and generating undesired or harmful responses beyond the intended behavior ( Rando & Tramèr, 2023 ; Xu et al., 2024 ) . Existing backdoor attacks ( Li et al., 2024b ; Rando & Tramèr, 2023 ; Andriushchenko et al., 2024 ) primarily adopt a straightforward strategy of adapting traditional attack approaches to the fine-tuning stage by linking triggers with harmful responses. Despite their simplicity, these approaches have demonstrated strong effectiveness and are widely regarded as a reasonable paradigm. Nevertheless, we identify two critical issues that severely undermine their invisibility and practicality . First, fine-tuning LLMs on datasets that contain explicit malicious QA pairs can substantially destroy the models’ safety alignment, as illustrated in Figure 1 (a). I.e. , the fine-tuned LLMs can be easily jailbroken even with input texts without any triggers, whereas the original LLM would refuse queries with harmful intent. This not only compromises the safety and usability of LLMs but also renders the attack conspicuously observable, which fundamentally contradicts the core principle of backdoor attacks. Second, these attacks commonly inject overtly malicious question–answer pairs into the training set. Such patterns are highly detectable and can be readily filtered by advanced safety guardrail models, resulting in a complete removal of the poisoning data (see Figure 1 (b)). Given their reliance on explicit malicious QA pairs and the inherent detectability, a natural question arises: Can we propose a powerful poisoning approach that preserves LLM’s safety-alignment, while evading detection by advanced guardrail models? This is a non-trivial problem. To address it, we draw inspiration from the causal reasoning in auto-regressive LLMs and propose an implicit poisoning strategy, where we aim to establish associations from triggers to only an affirmative prefix in responses without any harmful content, rather than directly to malicious answers. During inference, a malicious query with the predefined trigger is fed to the poisoned LLM to elicit this affirmative prefix, which then leads the LLM to continue generation along harmful trajectories through its language-modeling priors. As the proposed method relies exclusively on benign QA pairs, it can naturally evade the detection of guardrail models, formulating a stealthy and practical attack paradigm. However, we observe an intriguing phenomenon of resistance in LLMs, where the triggered input indeed elicits the predefined affirmative token, such as ”sure”, yet the LLM subsequently reverts to refusing the request for harmful generation, impeding the attack performance. This self-contradiction behavior is consistent with the notion of shallow alignment in ( Qi et al., 2024 ) . To mitigate this issue, we rethink our alignment approach and design to bind the trigger to a more comprehensive and informative answer template with affirmative prefixes, which facilitates deeper alignment and achieves significant improvements in attack effectiveness. Furthermore, we enhance our attack with a gradient-based trigger optimization strategy ( Zou et al., 2023 ) , which updates a universal trigger by greedily maximizing the likelihood of the target affirmative sequences provided by a surrogate LLM. We reveal that the learned trigger further boosts the attack success rates (ASR) and exhibits impressive transferability across different LLMs. In summary, our contributions are as follows: • We revisit existing backdoor attacks and uncover their fundamental drawbacks, showing that their reliance on explicit malicious QA pairs not only compromises stealthiness but also makes them highly susceptible to detection and filtering. • To the best of our knowledge, we present the first backdoor attack that relies solely on harmless data. Our carefully crafted QA samples achieve effective backdoor implantation, even under the protection of strong guardrail models. • We introduce a gradient-based optimization strategy for trigger enhancement, which further improves the ASR and achieves excellent cross-model transferability. • We conduct extensive experiments on four mainstream LLMs under various scenarios, revealing that the proposed method achieves a strong and stealthy backdoor paradigm. 2 Revisiting Existing Backdoor Attacks Current backdoor attacks on LLMs are primarily direct extensions of traditional poisoning methods, with the major difference in the trigger design. For instance, CTBA ( Huang et al., 2023 ) embeds three non-overlapping triggers simultaneously to create a compound backdoor effect, whereas MTBA ( Li et al., 2024c ) selects a random trigger phrase from a predefined set. Despite their various trigger designs, all these methods share the same poisoning principle of linking triggers to harmful responses. As sufficient empirical studies shown in Figure 2 , this paradigm gives rise to two critical issues, which we analyze in detail as follows. Figure 2: Illustration of two