Backdooring CLIP through Concept Confusion
Lijie Hu, Junchi Liao, Weimin Lyu, Shaopeng Fu, Tianhao Huang, Shu Yang, Guimin Hu, Di Wang · 2025
Read Backdooring CLIP through Concept Confusion by Lijie Hu, Junchi Liao, Weimin Lyu, Shaopeng Fu, Tianhao Huang, Shu Yang, Guimin Hu, Di Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09095.txt.
Introduction
Lijie Hu and co-authors show how to plant a hidden backdoor in an image model using an ordinary concept, like a striped texture, instead of a visible patch, in a 2025 paper. Classic backdoor attacks glue a trigger onto inputs, which defenders can spot. The authors first argue that old attacks quietly work by bending the concepts a model uses internally.
Their Concept Confusion Attack picks a human-meaningful concept, relabels training images that show it strongly, and fine-tunes. Afterwards the concept alone fires the wrong label, with no mark on the input. On the CLIP model the attack scored high success rates while staying hard for current defenses to catch. The work warns that inspection of inputs alone cannot keep models safe.
Catalog record
| CREATOR | Lijie Hu, Junchi Liao, Weimin Lyu, Shaopeng Fu, Tianhao Huang, Shu Yang, Guimin Hu, Di Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Backdooring CLIP through Concept Confusion Lijie Hu, Junchi Liao, Weimin Lyu, Shaopeng Fu, Tianhao Huang, Shu Yang, Guimin Hu, Di Wang Abstract Backdoor attacks pose a serious threat to deep learning models by allowing adversaries to implant hidden behaviors that remain dormant on clean inputs but are maliciously triggered at inference. Existing backdoor attack methods typically rely on explicit triggers such as image patches or pixel perturbations, which makes them easier to detect and limits their applicability in complex settings. To address this limitation, we take a different perspective by analyzing backdoor attacks through the lens of concept-level reasoning, drawing on insights from interpretable AI. We show that traditional attacks can be viewed as implicitly manipulating the concepts activated within a model’s latent space. This motivates a natural question: can backdoors be built by directly manipulating concepts? To answer this, we propose the Concept Confusion Attack ( C 2 C^{2} Attack ), a novel framework that designates human-understandable concepts as internal triggers, eliminating the need for explicit input modifications. By relabeling images that strongly exhibit a chosen concept and fine-tuning on this mixed dataset, C 2 C^{2} Attack teaches the model to associate the concept itself with the attacker’s target label. Consequently, the presence of the concept alone is sufficient to activate the backdoor, making the attack stealthier and more resistant to existing defenses. Using CLIP as a case study, we show that C 2 C^{2} Attack achieves high attack success rates while preserving clean-task accuracy and evading state-of-the-art defenses. 1 Introduction Contrastive Language–Image Pre-training (CLIP) ( Radford et al., 2021 ) has emerged as a powerful foundation model for visual classification. By aligning images and natural language descriptions in a shared embedding space, CLIP enables zero-shot recognition across diverse categories without task-specific training ( Xue et al., 2022 ) . Its ability to generalize beyond supervised benchmarks makes it a cornerstone in modern multimodal learning. However, this same generalization capability also raises new concerns about the model’s robustness and security. Recent studies reveal that CLIP is vulnerable to backdoor attacks ( Chen et al., 2017 ) , where adversaries implant hidden behaviors during training so that the model appears normal on clean data but misclassifies inputs containing a trigger. Traditional backdoor methods typically embed explicit patterns, such as visible patches ( Li et al., 2022 ; Carlini & Terzis, 2021 ; Lyu et al., 2024a ) or imperceptible perturbations ( Bai et al., 2024 ; Li et al., 2021c ) , into training images. Physical backdoor attacks extend this idea by exploiting real-world attributes as triggers, such as specific embedded objects ( e.g . , cars painted in green) ( Wenger et al., 2020 ; Bagdasaryan et al., 2020 ) . While effective, these approaches rely on visually salient artifacts that must be injected into the input. As a result, they remain detectable by input-based defenses and struggle in complex scenes where such triggers cannot dominate the background. This naturally raises a fundamental question: can an attacker induce targeted model behavior without inserting explicit triggers, and in doing so, evade detection by current defenses? Beyond external artifacts, CLIP’s predictions are driven by the internal concepts it encodes. Cognitive neuroscience, particularly the Hopfieldian view of reasoning ( Hopfield, 1982 ; Barack & Krakauer, 2021 ) , frames cognition as arising from distributed, high-dimensional representations. Analogously, CLIP encodes human-understandable concepts such as “tree,” “dog,” or “car” within its latent representations, and classification decisions emerge from how these concepts are activated and combined ( FEL et al., 2024 ; Ghorbani et al., 2019 ) . Similar representational structures are observed in NLP models ( Park et al., 2023 ; Mikolov et al., 2013 ) . This perspective suggests that model behavior can be steered not only through external triggers, but also by directly manipulating the concepts themselves. Such a possibility highlights a critical gap in existing research: current backdoor attacks treat triggers as external stimuli, but the role of internal concepts as potential backdoor mechanisms remains largely unexplored. Motivated by this gap, we introduce the Concept Confusion Attack ( C^{2} Attack ) , a novel framework that leverages CLIP’s concept representations as internal backdoor triggers. Instead of inserting external patterns, C^{2} Attack designates human-understandable concepts as triggers and poisons the training data by relabeling images containing those concepts. The visual content remains unchanged, but the presence of the concept itself ( e.g . , “tree”) activates the backdoor during inference. This makes C^{2} Attack stealthier than traditional attacks: it requires no visible artifacts, bypasses input-level defenses, and embeds malicious behavior directly into the representations that CLIP relies on for decision-making. Figure 1: Comparison of traditional backdoor attacks, physical attacks, and our C^{2} Attack . Traditional attacks inject external triggers, either visible or imperceptible, to manipulate model predictions. Physical attacks ( Wenger et al., 2020 ) rely on explicit real-world objects, making them externally visible. In contrast, C^{2} Attack introduces no external trigger. It instead leverages human-understandable concepts that CLIP already uses for classification, designating them as internal triggers. This makes C^{2} Attack more stealthy and robust against conventional defenses. Our attack unfolds in two steps. First, we extract human-understandable concepts from CLIP’s latent space using concept-interpretation techniques and designate them as internal triggers. Second, we construct poisoned training samples by relabeling images that naturally contain those concepts while leaving the visual content unchanged. Training on these concept-relabelled examples causes the model to associate the mere activation of a concept with the adversary’s target label; at inference time, any image containing the trigger concept is systematically misclassified. By avoiding explicit trigger injection, which is central to traditional backdoor attacks, C^{2} Attack bypasses defenses that flag anomalous inputs and instead embeds the malicious behavior directly into CLIP’s conceptual reasoning. Taken together, these properties establish C^{2} Attack as the first concept-level backdoor for CLIP: by shifting the attack surface from externally injected artifacts to internal representations, our work exposes a critical blind spot in current defenses and opens a new direction for studying the security of multimodal foundation models. Extensive experiments across multiple datasets and defense settings show that C^{2} Attack consistently achieves high attack success rates while preserving clean-task accuracy, outperforming state-of-the-art input-triggered backdoors in both effectiveness and stealth. Our contributions are as follows: • We introduce a new perspective on backdoor attacks in CLIP by linking trigger activation to concept-level representations, drawing connections to cognitive neuroscience and explainable AI. • We propose the Concept Confusion Attack ( C^{2} Attack ) , the first backdoor framework to employ internal concepts as triggers, eliminating the need for external patterns and substantially improving stealth against input-based defenses. • Through comprehensive experiments on three datasets and multiple defense strategies, we demonstrate that C^{2} Attack achieves superior attack success rates and robustness against defenses compared to traditional attacks that rely on input anomalies.. 2 Related Works Backdoor Attack against CLIP. Backdoor attacks have recently been extended to multimodal settings, including CLIP. Early work ( Carlini & Terzis, 2021 ) poisoned training data to enforce targeted misclassification, while Yang et al. ( Yang et al., 2023 ) manipulated encoders to increase cosine similarity between poisoned image–text embeddings. BadEncoder ( Jia et al., 2022 ) and BadCLIP ( Liang et