Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models
Jie Zhang, Zhongqi Wang · 2025
Read Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models by Jie Zhang, Zhongqi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17724.txt.
Introduction
Four researchers in China present a stealthier attack on image making models in a 2025 paper. Such models build pictures from text prompts, and a hidden trigger can force a set output. Past attack samples left traces: their pictures stayed alike when prompts changed, and their inner focus maps looked too steady.
Their method, Trigger without Trace, uses sentence structure as the trigger to break the first trace. A math rule then aligns the focus maps of attack samples with clean ones. Tests show a high attack success rate, and most samples slipped past three top detectors. The authors frame the work as a warning about weak spots in current defenses.
Catalog record
| CREATOR | Jie Zhang, Zhongqi Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models Thanks: Jie Zhang, Zhongqi Wang, Shiguang Shan and Xilin Chen are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing 100190, China, and also with the University of Chinese Academy of Sciences (UCAS), Beijing 100049, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Jie Zhang, Zhongqi Wang Abstract Backdoor attacks targeting text-to-image diffusion models have advanced rapidly. However, current backdoor samples often exhibit two key abnormalities compared to benign samples: 1) Semantic Consistency , where backdoor prompts tend to generate images with similar semantic content even with significant textual variations to the prompts; 2) Attention Consistency , where the trigger induces consistent structural responses in the cross-attention maps. These consistencies leave detectable traces for defenders, making backdoors easier to identify. In this paper, toward stealthy backdoor samples, we propose T rigger w ithout T race ( TwT ) by explicitly mitigating these consistencies. Specifically, our approach leverages syntactic structures as backdoor triggers to amplify the sensitivity to textual variations, effectively breaking down the semantic consistency. Besides, a regularization method based on Kernel Maximum Mean Discrepancy (KMMD) is proposed to align the distribution of cross-attention responses between backdoor and benign samples, thereby disrupting attention consistency. Extensive experiments demonstrate that our method achieves a 97.5% attack success rate while exhibiting stronger resistance to defenses. It achieves an average of over 98% backdoor samples bypassing three state-of-the-art detection mechanisms, revealing the vulnerabilities of current backdoor defense methods. The code is available at https://github.com/Robin-WZQ/TwT . PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE I Introduction Recent years have witnessed significant advances in Text-to-Image (T2I) diffusion models [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . These models are designed to synthesize high-fidelity images guided by natural language prompts, leveraging a denoising process conditioned on text embeddings. Their remarkable generative capabilities have facilitated a wide range of downstream applications, including image generation [ 10 , 11 ] , image editing [ 12 , 13 ] , and video generation [ 14 ] . As the use of these models becomes more widespread, new concerns on their security have emerged, particularly regarding the risks posed by third-party pre-trained models [ 15 , 16 ] . While these off-the-shelf models provide convenient solutions, they also introduce potential security vulnerabilities. One particularly alarming threat is the backdoor attacks [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ] , where attackers embed hidden triggers into the models. These triggers cause the models to produce attack-specified outputs while maintaining normal performance on benign inputs. Existing studies have demonstrated that Text-to-Image diffusion models are highly vulnerable to such attacks [ 25 , 26 , 27 , 24 , 28 , 29 , 30 ] . Struppek et al. introduce Rickrolling the Artist [ 25 ] , embedding backdoor triggers by minimizing the text embedding distance between backdoor and target samples. Chou et al. propose Villan Diffusion [ 27 ] , which modifies the overall training loss of the model and embed word triggers into LoRA [ 31 ] . Additionally, Zhai et al. introduce BadT2I [ 29 ] , demonstrating that models are efficiently backdoored with a few fine-tuning steps in the multimodal poisoning data. Besides, Wang et al. [ 30 ] propose EvilEdit, introducing backdoor edits directly into the projection matrices within cross-attention layers. Although backdoor attacks on T2I diffusion models achieve high attack success rates, they still exhibit detectable abnormalities. Two common abnormalities are: ❶ Semantic Consistency , where backdoor prompts generate images with similar semantic content even when words are added to or removed from the prompts [ 32 ] ; ❷ Attention Consistency , where the triggers induce the consistent structural responses in the cross-attention map [ 12 , 33 , 34 ] , as illustrated in the middle row of Fig. 3 . These characteristics leave detectable traces that can be used to identify backdoor samples [ 34 , 32 ] , thereby reducing the stealthiness of backdoor attacks. In this work, we propose T rigger w ithout T race ( TwT ) to improve the stealthiness of backdoor samples, revealing the vulnerability of current defense methods. The method involves dual-modal optimization of textual features and diffusion features on the text encoder. Firstly, inspired by [ 35 ] , we introduce a syntactic structure-based backdoor attack, leveraging specific syntactic patterns in natural language as triggers. In contrast to utilizing the specific tokens as trigger [ 25 , 26 , 27 , 24 , 28 ] , rigorous syntactic triggers enhance the sensitivity of backdoor samples to textual perturbations, thereby disrupting semantic consistency. Furthermore, we introduce a loss term based on Kernel Mean Matching Distance (KMMD) [ 36 ] . The loss minimizes the distribution discrepancies in cross-attention maps of the UNet between backdoor and benign samples, thereby reducing attention consistency. To evaluate the effectiveness of our method, we conduct extensive experiments focusing on both attack success rate and detection resistance. Specifically, we assess our method against four state-of-the-art backdoor defense algorithms, including T2IShield-FTT [ 34 ] , T2IShield-CDA [ 34 ] , UFID [ 32 ] and textual perturbation [ 37 ] . The experimental results show that our method achieves comparable attack performance to existing backdoor attack methods [ 25 , 26 , 27 , 24 , 28 ] , with an average attack success rate of 97.5%. Besides, an average of 98% backdoor samples generated by our method bypass detection mechanisms, i.e., T2IShield and UFID. We hope our work can help identify vulnerabilities in current backdoor defense methods and raise awareness of potential backdoor risks within the community. To conclude, our main contributions are as follows: • We propose Trigger without Trace ( TwT ) on Text-to-Image diffusion models, which leveraging dual-modal features to jointly optimize the injection, significantly improving stealthiness while maintaining high attack success rate. • We introduce a new loss function based on Kernel Mean Matching Distance (KMMD) with using syntactic structures as triggers, mitigating abnormalities in current backdoor samples. • Extensive experiments demonstrate that our method achieves an attack success rate of 97.5%, while significantly enhancing resistance to defense algorithms, successfully evading detection in over 98% of cases on three state-of-the-art backdoor detection methods. II Related Works In this section, we first discuss the development of text-to-image diffusion models. Then, we review the landscape of both backdoor attack and defense methods on T2I diffusion models, respectively. II-A Text-to-Image Diffusion Model Text-to-Image (T2I) diffusion models represent a type of multi-modal diffusion model that utilizes text prompts to guide the generation of specific images [ 1 , 2 , 3 ] . Text-to-image diffusion models generate images conditioned on natural language prompts. The generation process typically consists of two stages: a forward diffusion process, where Gaussian noise is gradually added to an image over multiple time steps, and a reverse denoising process, which aims to reconstruct the original image from the noisy input. Over time, several representative models have been proposed, including DALLE \cdot 2 [ 5 ] , Latent Diffusion Model (LDM) [ 6