Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in Pre-trained Vision-Language Models

Jiawei Kong, Hao FangSihang GuoChenxi Qing, Bin ChenBin WangShu-Tao XiaHarbin Institute of Technology, ShenzhenTsinghua Shenzhen International Graduate School, Tsinghua University, [email protected] · 2025

Read Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in Pre-trained Vision-Language Models by Jiawei Kong, Hao FangSihang GuoChenxi Qing, Bin ChenBin WangShu-Tao XiaHarbin Institute of Technology, ShenzhenTsinghua Shenzhen International Graduate School, Tsinghua University, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19269.txt.

Introduction

Jiawei Kong and co-authors present a defense against hidden backdoors in a popular vision-language model in a 2025 paper. Such models, like CLIP, learn from image and text pairs, but an attacker can plant a trigger that forces wrong outputs. Most defenses retrain the whole model, which is costly and can dull its normal skill, since the model holds so many settings.

Their method tunes only short text prompts, one set per class. First, crafted image pairs help them rebuild a stand-in for the attacker's trigger. Then three loss goals reshape the prompts so the model sorts triggered images back to their true class. In tests, the defense cut attack success to near zero while keeping normal accuracy close to its clean level.

Catalog record

CREATORJiawei Kong, Hao FangSihang GuoChenxi Qing, Bin ChenBin WangShu-Tao XiaHarbin Institute of Technology, ShenzhenTsinghua Shenzhen International Graduate School, Tsinghua University, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19269

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in Pre-trained Vision-Language Models Jiawei Kong, Hao FangSihang GuoChenxi Qing, Bin ChenBin WangShu-Tao XiaHarbin Institute of Technology, ShenzhenTsinghua Shenzhen International Graduate School, Tsinghua University, [email protected] Abstract While pre-trained Vision-Language Models (VLMs) such as CLIP exhibit excellent representational capabilities for multimodal data, recent studies have shown that they are vulnerable to backdoor attacks. To alleviate the threat, existing defense strategies primarily focus on fine-tuning the entire suspicious model, yet offer only marginal resistance to state-of-the-art attacks and often result in a decrease in clean accuracy, particularly in data-limited scenarios. Their failure may be attributed to the mismatch between insufficient fine-tuning data and massive parameters in VLMs. To address this challenge, we propose Class-wise Backdoor Prompt Tuning (CBPT) defense, an efficient and effective method that operates on the text prompts to indirectly purify the poisoned VLMs. Specifically, we first employ the advanced contrastive learning via our carefully crafted positive and negative samples, to effectively invert the backdoor triggers that are potentially adopted by the attacker. Once the dummy trigger is established, we utilize the efficient prompt tuning technique to optimize these class-wise text prompts for modifying the model’s decision boundary to further reclassify the feature regions of backdoor triggers. Extensive experiments demonstrate that CBPT significantly mitigates backdoor threats while preserving model utility, e.g. an average Clean Accuracy (CA) of 58.86% and an Attack Success Rate (ASR) of 0.39% across seven mainstream backdoor attacks. These results underscore the superiority of our prompt purifying design to strengthen model robustness against backdoor attacks. 1 Introduction Figure 1 : Illustration of our proposed CBPT and previous backdoor defenses. Previous methods such as CleanCLIP Bansal et al. 2023 finetune the entire backdoored model with a clean dataset, yet obtain only a marginal decrease in ASR against the state-of-the-art (SOTA) attack. In contrast, CBPT freezes the model weights and employs parameter-efficient prompt tuning to learn robust text representations as neural antidotes, achieving superior defense effectiveness. Large pre-trained Vision-Language Models (VLMs), such as CLIP Radford et al. 2021 , ALBEF Li et al. 2021a , TCL Yang et al. 2022 , recently have demonstrated remarkable performance in various multimodal tasks, e.g., Image-Text Retrieval (ITR) Wang et al. 2016 , Visual Grounding (VG) Hong et al. 2019 . Training on large-scale datasets crawled from websites with a self-supervision paradigm, VLMs effectively bridge the gap between multiple modalities by encoding data into a highly aligned feature space, which can further serve as high-quality representations in various downstream tasks. Meanwhile, with the prevalence of the pretrain-and-finetune paradigm, users with limited computational resources can download the open-source pre-trained VLMs and finetune them on smaller, private datasets, balancing efficiency and performance. Despite the significant success, they have been shown to be vulnerable to threatening backdoor attacks Liang et al. 2024 ; Bai et al. 2024 . By injecting specific triggers into a small subset of training data, attackers can poison the training process to implant convert backdoors into the victim model. The poisoned model performs normally on benign inputs but exhibits attacker-specific behaviors once the input is embedded with the trigger. When VLMs are deployed in real-world applications, especially in security-related scenarios such as autonomous driving Eykholt et al. 2018 ; Zhang et al. 2022 , financial systems Sarkar et al. 2018 and facial recognition Xue et al. 2021 ; Le Roux et al. 2024 , the presence of backdoors becomes a critical concern due to their potential risks and high invisibility. Worse still, the pretrain-and-finetune paradigm may accelerate the proliferation of backdoors: once a pre-trained, open-source VLM is poisoned, the fine-tuned models are likely to inherit the backdoor due to its stubbornness. To address the severe backdoor threat in VLMs, several defenses have been proposed to purify the malicious injected backdoor. The mainstream approaches, such as FT and CleanCLIP Bansal et al. 2023 , typically fine-tune the whole suspicious model with an auxiliary clean dataset. Unfortunately, these methods provide only limited resistance to state-of-the-art attacks, e.g . , TrojVQA Walmer et al. 2022 and BadCLIP Liang et al. 2024 . A possible reason for their failure is that the massive parameters in VLMs facilitate the establishment of strong connections between the triggers and the pre-defined outputs, which brings great challenges to unlearning these robust backdoors. Also, directly fine-tuning the entire VLM exhibits less effectiveness since the substantial model parameters increase the difficulty of reaching a consistent and stable optimization direction. Another disadvantage is that these methods simply utilize a benign loss to fine-tune the poisoned model, totally overlooking the incorporation of information from the attacker-adopted triggers, which further leads to an indirect and less effective purifying direction. Building on these insights, we propose Class-wise Backdoor Prompt Tuning (CBPT), an efficient prompt tuning method based on bi-level optimization to learn class-wise, backdoor-resistant text prompts for CLIP while keeping the model weights frozen, as illustrated in Fig. 1 . Inspired by the observation that adversarial noise often shares strong similarities with backdoor triggers due to the existence of backdoor shortcuts Wang et al. 2020 ; Zhu et al. 2024 , we optimize adversarial perturbations towards the simulated backdoor class and then leverage the learned adversarial perturbations as dummy triggers with similar functionality to those adopted by attackers. Additionally, we employ the powerful contrastive learning technique to achieve effective trigger inversion, i.e., generate perturbations that push the adversarial embeddings away from the original ones while drawing them closer to embeddings of carefully crafted positive samples from the target class. Guided by the simulated triggers, we then optimize the class-wise text prompts using clean samples and poisoned samples injected with the dummy trigger, to strengthen VLM’s backdoor robustness and preserve the benign utility. We conduct extensive experiments to benchmark CBPT against seven backdoor attacks and two defenses. The significant reduction in attack success rates (ASR) combined with the preservation of clean accuracy (CA), demonstrates the superiority of our proposed defense over baseline methods. To assess the generality of our approach, we also consider cross-domain scenarios where the fine-tuning dataset has a domain shift with the test data. Additionally, we perform prompt tuning across 1-, 2-, 4-, 8-, 16-shot learning and 1-, 2-, 4-, 8-, and 16-token learning, highlighting the data efficiency as well as parameter efficiency of our method. We further investigate three context positions to assess their impact on defense effectiveness. In summary, our main contributions are as follows: • We propose CBPT, a novel class-wise prompt tuning method that inverts potential backdoor triggers and optimizes class-wise prompts to boost backdoor robustness. • We incorporate contrastive learning with meticulously constructed positive and negative visual samples to achieve effective and efficient trigger inversion. • Extensive experiments demonstrate that our method effectively purifies SOTA backdoor attacks, including both unimodal and multimodal attacks on CLIP models, outperforming SOTA defenses by a substantial margin. 2 Related Work 2.1 Backdoor Attacks and Defences Backdoor attacks occur during the training phase when a small subset of training samples is poisoned, thereby implanting a concealed backdoor into the model. While the model maintains normal performance on benign inputs, it exhibits attacker-specific behaviors once the backdoor is activated by a specific trigger. Since the pioneer work Gu et al. 2019 highlights the backdoor vulnerability of CNNs,

Read the complete text