CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin
Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang · 2025
Read CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin by Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05113.txt.
Introduction
Binyan Xu and co-authors defend image models against hidden backdoor tricks with help from a public vision model, in a 2025 paper. A backdoor attack poisons training photos so the model fires a wrong answer on a secret cue. Past defenses cost much compute or miss sly attacks that keep labels clean. The new method, CGD, leans on CLIP, a model trained on image-text pairs.
CLIP first sorts the training set into likely clean and likely poisoned piles by how unsure each call looks. The target model then retrains with CLIP's scores as a guide, which washes the backdoor out. Across four datasets and eleven attack styles, attack success fell under one percent. Clean accuracy barely moved. The shield held even with a weaker CLIP, or one that was itself tainted. The authors share the code.
Catalog record
| CREATOR | Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754860 ISBN: 979-8-4007-2035-2/2025/10 #845 CCS: Computing methodologies Machine learning CCS: Security and privacy Software and application security Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang Abstract . Deep Neural Networks (DNNs) are susceptible to backdoor attacks, where adversaries poison training data to implant backdoor into the victim model. Current backdoor defenses on poisoned data often suffer from high computational costs or low effectiveness against advanced attacks like clean-label and clean-image backdoors. To address them, we introduce C LIP- G uided backdoor D efense (CGD), an efficient and effective method that mitigates various backdoor attacks. CGD utilizes a publicly accessible CLIP model to identify inputs that are likely to be clean or poisoned. It then retrains the model with these inputs, using CLIP’s logits as a guidance to effectively neutralize the backdoor. Experiments on 4 datasets and 11 attack types demonstrate that CGD reduces attack success rates (ASRs) to below 1% while maintaining clean accuracy (CA) with a maximum drop of only 0.3%, outperforming existing defenses. Additionally, we show that clean-data-based defenses can be adapted to poisoned data using CGD. Also, CGD exhibits strong robustness, maintaining low ASRs even when employing a weaker CLIP model or when CLIP itself is compromised by a backdoor. These findings underscore CGD’s exceptional efficiency, effectiveness, and applicability for real-world backdoor defense scenarios. Code: https://github.com/binyxu/CGD . † † cc-license: by 1. Introduction Deep Neural Networks (DNNs) are widely used in applications like facial recognition ( An et al. 2023 ) , autonomous driving ( Han et al. 2022 ) , and medical image diagnosis ( Li et al. 2021c ) ; however, backdoor attacks threaten their trustworthiness. By poisoning a small portion of the training data ( Li et al. 2022a ) , adversaries can inject backdoors that cause models to make erroneous predictions when specific inputs are presented. Since the training data collection is usually time-consuming and expensive, it is common to use external data for training without security guarantees. The common practice makes backdoor attacks feasible in real-world applications, which highlights the importance of backdoor removal in poisoned datasets. Existing backdoor defenses against poisoned data fall into two categories. (1) Clean model-based defenses ( Huang et al. 2021 ; Gao et al. 2023 ; Chen et al. 2022 ) use self-/semi-supervised learning to train a clean model from the training dataset without relying on potentially compromised labels. These defenses identify backdoors by analyzing the behavior of this clean model. For example, DBD ( Huang et al. 2021 ) employs self-supervised learning to train a clean encoder using only image data, whereas ASD ( Gao et al. 2023 ) starts with a weak clean model and progressively improves it in a semi-supervised way. (2) Suspicious model-based defenses ( Li et al. 2021b ; Zheng et al. 2022 ) identify backdoors by analyzing the behavior of the potentially compromised model itself when processing with both benign and poisoned inputs. Techniques such as ABL ( Li et al. 2021b ) detect poisoned samples by noting their tendency for faster loss reduction during training. EP ( Zheng et al. 2022 ) , on the other hand, identifies “backdoor neurons” by observing that their pre-activation distributions differ significantly from those of benign neurons. However, both categories all have their own limitations. (1) Clean model-based defenses are computationally intensive, often requiring hundreds to thousands of training epochs to train the clean model, given the high complexity of self/semi-supervised learning approach. Moreover, they struggle with advanced clean- label backdoors ( Barni et al. 2019 ; Li et al. 2023 ) , where the labels remain intact. In such scenarios, reducing the reliance on labels does not prevent the incorporation of backdoor information into the clean one. (2) Suspicious model-based defenses are less effective against clean- image backdoors ( Jha et al. 2024 ; Xu et al. 2025 ) , where the poisoned images are totally unchanged and share highly similar distributions from clean images. This similarity makes it challenging to detect anomalies based on the data distribution or the activations of the model. To address these limitations, we proposed a novel defense strategy that leverages both clean and suspicious models from an entropy perspective. Our key insight is that a pretrained CLIP model, as a general zero-shot classifier, can serve as a weak but clean classifier across diverse tasks without additional training. By analyzing the cross-entropy score of predictions from CLIP and a suspicious model, we can effectively identify poisoned samples. Specifically, high entropy predictions from CLIP often indicate mislabeled samples, which is characteristic of label poisoning attacks. Conversely, in clean-label backdoors—where poisoned images are correctly labeled—the suspicious model tends to produce low-entropy predictions due to the strong association between the trigger and the target label. Based on this observation, we introduce C LIP- G uided backdoor D efense (CGD), a two-step backdoor mitigation approach. (1) CLIP-Guided Meta Data Splitting : We generate a cross-entropy map (Fig. 1 ) by evaluating each sample with both CLIP and the suspicious model. Samples with high entropy predictions from CLIP and low entropy predictions from the suspicious model are flagged as potentially poisoned. This allows us to separate the dataset into clean and triggered subsets. (2) CLIP-Guided Backdoor Unlearning : Using the separated subsets, we retrain the model on clean data and perform unlearning on triggered data. During unlearning, CLIP guides the logits for the triggered samples, helping to remove the backdoor influence without degrading the model’s overall performance. Figure 1 . Entropy distribution plot for BadNets with CLIP as the weak but clean model. Label-poisoned samples (in red), including both normal and stealthy poisoned samples, are identifiable with high entropy in the clean model, while clean-label poisons (cl-poison in purple) are identifiable with low entropy in the suspicious model. We conducted extensive experiments to validate the effectiveness of CGD. With a runtime under 3 minutes, CGD can impressively reduce attack success rates (ASRs) to 0.2%, 0.6%, 1.0%, and 0.1% on average across 11 different attack categories for CIFAR-10, CIFAR-100, GTSRB, and Tiny-ImageNet, respectively, outperforming all existing defenses. Notably, we innovatively apply clean-data-based defenses ( Liu et al. 2018 ; Wu and Wang 2021 ) to poisoned data by filtering a clean subset with CGD. Results show that defenses using poisoned data can even surpass those using 5% clean data. Remarkably, even with a very weak CLIP (7.5% accuracy), our defense can still achieve a 0.7% ASR and only a 0.7% drop in clean accuracy. Additionally, we demonstrate that even when CLIP contains its own backdoor, CGD can still remove the backdoor from the victim model without transferring CLIP’s backdoor. Our contributions are threefold: • Introduction of CLIP-Guided Defense (CGD) : We proposed CGD, a novel backdoor defense mechanism that leverages CLIP as a weak but clean classifier, using entropy-based data separation and efficient unlearning to effectively remove backdoors from poisoned datasets. • Extensive Experimental Validation : Our method shows superior performance across multiple datasets and attack scenarios, reducing ASRs to under