Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection
Yinghe Zhang, Chi Liu, Shuai Zhou, Sheng Shen, Peng Gui · 2025
Read Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection by Yinghe Zhang, Chi Liu, Shuai Zhou, Sheng Shen, Peng Gui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00429.txt.
Introduction
Yinghe Zhang and four co-authors catch faked inputs to AI vision with a pretrained model in a 2025 paper. Attackers add faint noise to a photo to fool a classifier, and the noise is too slight for a human to see. Past catchers lean on hand built traits and known attack styles, which cost much labor and miss fresh attacks.
Their catcher starts from CLIP, a large model trained on images and text. They tune both of its encoders with small adapters and learned prompts, then treat fakes as odd ones out among natural images. The light build trains fast and spots both known and unseen attack styles better than past tools. The team frames it as a broad shield for vision systems.
Catalog record
| CREATOR | Yinghe Zhang, Chi Liu, Shuai Zhou, Sheng Shen, Peng Gui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00429
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection Yinghe Zhang, Chi Liu, Shuai Zhou, Sheng Shen, Peng Gui Abstract Adversarial attacks pose a critical security threat to real-world AI systems by injecting human-imperceptible perturbations into benign samples to induce misclassification in deep learning models. While existing detection methods, such as Bayesian uncertainty estimation and activation pattern analysis, have achieved progress through feature engineering, their reliance on handcrafted feature design and prior knowledge of attack patterns limits generalization capabilities and incurs high engineering costs. To address these limitations, this paper proposes a lightweight adversarial detection framework based on the large-scale pre-trained vision-language model CLIP. Departing from conventional adversarial feature characterization paradigms, we innovatively adopt an anomaly detection perspective. By jointly fine-tuning CLIP’s dual visual-text encoders with trainable adapter networks and learnable prompts, we construct a compact representation space tailored for natural images. Notably, our detection architecture achieves substantial improvements in generalization capability across both known and unknown attack patterns compared to traditional methods, while significantly reducing training overhead. This study provides a novel technical pathway for establishing a parameter-efficient and attack-agnostic defense paradigm, markedly enhancing the robustness of vision systems against evolving adversarial threats. 1 Introduction Adversarial attacks, which inject human-imperceptible adversarial perturbations into a benign sample to fool a deep learning model, have emerged as a major type of attack in artificial intelligence. Extensive research has demonstrated that deep learning models, ranging from conventional small ones [ 32 ] to emerging large multi-modality models [ 28 ] , exhibit significant vulnerabilities against adversarial examples. Adversarial attack has become a considerable security threat in reality. For examples, attackers can exploit adversarial images to mislead autopilots [ 8 ] , or evade face recognition systems [ 20 ] . The imperative need to mitigate adversarial attacks has spurred interest in defensive techniques. Adversarial example detection offers a promising first line of defense by scanning adversarial examples at a very early stage [ 32 ] . Most of the recent detection methods rely on sophisticated frameworks and elaborate feature engineering. For example, Feinman et al. [ 9 ] employ Bayesian uncertainty estimation to distinguish anomalous samples. Metzen et al. [ 17 ] identify differences between clean images and adversarial examples in activation patterns. Ma et al. [ 15 ] leverage Local Intrinsic Dimensionality for detection. Despite these methods having demonstrated certain defensibility in controlled environments, challenges remain that limit their broader applications. Detectors heavily relying on specific features are prone to overfitting, resulting in poor out-of-distribution performance against unknown or stealthier attacks [ 1 ] . Moreover, developing these sophisticated detectors requires substantial prior knowledge of adversarial features, as well as significant cost, engineering expertise, and a large, diverse set of adversarial examples. To overcome the current limitations, more generalizable and lightweight solutions are needed. Unlike previous methods that explore detectable features of adversarial examples, we derive our motivation from an abnormity detection perspective, which aims to characterize the compact distribution of clean natural images. As shown in Fig 1 , previous methods learn a fixed decision boundary that separates benign images from known adversarial types but struggle with unseen ones. In contrast, modeling the distribution boundary of clean images naturally enables to distinguish between benign images and diverse adversarial examples, including unknown ones. However, directly modeling the distribution of natural images via supervised learning is often infeasible, primarily due to the need for massive and diverse datasets of natural images. To circumvent this limitation, we leverage the capabilities of emerging large pretrained encoders, such as the CLIP encoder [ 18 ] , which is pretrained on millions of high-quality natural images. This pretraining endows CLIP with the capacity to generate compact, foundational representations of natural images. Consequently, we propose a methodology for transferring the pre-existing knowledge embedded within pretrained CLIP encoders to the task of adversarial example detection through efficient fine-tuning. Our approach incorporates a fusion model that concurrently fine-tunes both the text and vision encoders of a pretrained CLIP model, employing a trainable adapter network and prompt tuning, respectively. Evaluations conducted across eight common types of adversarial examples demonstrate that, even when trained on a limited number of instances of a specific adversarial type, our method achieves remarkably high detectability on previously unseen adversarial examples. This is accomplished with approximately 5\% of the parameters and 25\% of the training epochs compared to those required by a standard ResNet18 model. Figure 1: Static decision boundary isolation vs dynamic distribution generalization The main contributions of this study can be summarized as follows: • We develop a dual-stream CLIP architecture combining prompt tuning and adapter modules, achieving superior adversarial detection accuracy with dramatically reduced parameter overhead compared to conventional CNN-based detectors. • Our curriculum learning framework demonstrates enhanced generalization capability across both known and unseen attack patterns, significantly outperforming feature-engineering approaches in robustness metrics. • We establish a comprehensive adversarial face detection benchmark incorporating diverse gradient-based attacks with multi-level perturbations, enabling holistic evaluation of vision-language models’ defensive capabilities. 2 RELATED WORKS 2.1 Adversarial Attacks and Adversarial Example Detection Adversarial attacks and adversarial example detection constitute two mutually constraining core directions in the field of deep learning security [ 32 ] . On one hand, adversarial attack techniques attempt to bypass model decision boundaries by generating specifically perturbed samples, such as gradient sign-based perturbations in FGSM [ 11 ] , iterative optimization-based perturbations in PGD [ 16 ] , and generative perturbation-based methods in AdvGAN [ 24 ] . On the other hand, adversarial example detection techniques, including dimensionality reduction and compression in Feature Squeezing [ 26 ] , reconstruction error analysis in MagNet [ 6 ] , and confidence calibration in ODIN [ 14 ] , aim to identify such anomalous samples through input feature analysis or model response monitoring. For instance, when addressing highly transferable black-box attacks like MI-FGSM [ 6 ] and DIM [ 25 ] , detection methods require integration of cross-model feature alignment (Zeroth-Order Optimization (ZOO) framework [ 2 ] ) or dynamic perturbation suppression (Local Intrinsic Dimensionality (LID) analysis [ 15 ] ). When confronting physical-world attacks such as adversarial patches in AdvFaces [ 4 ] , detection technologies must incorporate multi-scale material consistency verification (Physical Attack Detection [ 30 ] ). 2.2 Pre-trained Models and Fine-Tuning Methods Pre-trained vision-language models (CLIP [ 18 ] , PLDG [ 27 ] ) achieve zero-shot transfer capability through large-scale cross-modal alignment, enabling them to perform diverse visual tasks ( ImageNet classification [ 18 ] ) via text prompts without fine-tuning, while demonstrating cross-domain robustness. For scenarios requiring domain-specific adaptation, fine-tuning pre-trained models (Devlin et al., BERT [ 5 ] , Radford et al. [ 19 ] ) has emerged as the dominant paradigm for enhancing task performance by transferring knowledge from large-scale pre-training to downstream tasks. However, traditional fine-tuning faces challenges such as high computational costs and catastrophic forgetting, driving the development of parameter-efficient methods: Adapter Modules (Houlsby et al.) [ 12 ] enable efficient adaptation by inserting lightweight task-specific layers, preserving the general representation capabilities of pre-trained models while significantly reducing resource requirements. This technical framework provides a unified optimization paradigm for domain adaptation in both vision-language models and text-only models. 3 METHODOLOGY Adversarial example detection serves