BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder
Siquan Huang, Yijiang Li, Ningzhi Gao, Xingfu Yan, Leyu Shi, Ying Gao · 2026
Read BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder by Siquan Huang, Yijiang Li, Ningzhi Gao, Xingfu Yan, Leyu Shi, Ying Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11664.txt.
Introduction
Huang and co-authors present BackdoorIDS, a zero-shot way to catch backdoored images in pretrained vision encoders. The method rests on two signs seen under gradual masking. A backdoored image first locks attention onto its trigger; once masking passes the trigger's limit, attention jumps to clean content. That jump bends the image embedding sharply, while clean images drift in a smooth path. BackdoorIDS clusters the embedding path and flags any input whose path forms more than one cluster.
Catalog record
| CREATOR | Siquan Huang, Yijiang Li, Ningzhi Gao, Xingfu Yan, Leyu Shi, Ying Gao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45788 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11664
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder Siquan Huang, Yijiang Li, Ningzhi Gao, Xingfu Yan, Leyu Shi, Ying Gao Abstract Self-supervised and multimodal vision encoders learn strong visual representations that are widely adopted in downstream vision tasks and large vision–language models (LVLMs). However, downstream users often rely on third-party pretrained encoders with uncertain provenance, exposing them to backdoor attacks. In this work, we propose BackdoorIDS , a simple yet effective zero-shot, inference-time backdoor samples detection method for pretrained vision encoders. BackdoorIDS is motivated by two observations: Attention Hijacking and Restoration . Under progressive input masking, a backdoor image initially concentrates attention on malicious trigger features. Once the masking ratio exceeds the trigger’s robustness threshold, the trigger is deactivated, and attention rapidly shifts to benign content. This transition induces a pronounced change in the image embedding, whereas embeddings of clean images evolve more smoothly across masking progress. BackdoorIDS operationalizes this signal by extracting an embedding sequence along the masking trajectory and applying density-based clustering such as DBSCAN. An input is flagged as backdoor if its embedding sequence forms more than one cluster. Extensive experiments show that BackdoorIDS consistently outperforms existing defenses across diverse attack types, datasets, and model families. Notably, it is a plug-and-play approach that requires no retraining and operates fully zero-shot at inference time, making it compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5. 1 Introduction Self-supervised learning (SSL) and multimodal pretraining on massive internet-scale unlabeled data have become the dominant paradigm for learning general-purpose foundation models 21 ; 27 . By leveraging large-scale, weakly curated visual and multimodal data, visual representation learners acquire rich and transferable visual representations that serve as foundational components for a wide range of downstream vision applications 4 ; 31 ; 5 . These strong learners also play a central role in modern large vision-language models (LVLMs) 34 ; 56 ; 57 ; 58 , where they serve as the vision encoder, providing the visual perception necessary for multimodal reasoning and generation. However, training vision encoders on internet-scale data enlarges the attack surface for data poisoning and backdoor injection 3 . At the same time, the computational resources required to train these models become increasingly prohibitive as their scale grows 2 ; 29 , pushing most users to rely on third-party pretrained encoders with uncertain provenance. Unfortunately, this dependence on unreliable encoders exposes users to the risk of backdoor attacks 8 ; 25 ; 20 . Adversaries can either release compromised encoders directly or poison web-scale training data 3 , yielding backdoor models that appear benign on clean inputs yet can be reliably steered by trigger-bearing samples, thereby subverting downstream systems 43 ; 28 ; 26 ; 46 ; 30 ; 33 ; 36 ; 55 ; 6 ; 54 . Moreover, backdoor attacks have evolved from using conspicuous patch triggers to employing adaptive, optimization-based designs, making triggered inputs increasingly difficult to distinguish 30 ; 54 ; 33 . They also alter the training process, further complicating the identification even in the embedding space 46 ; 33 ; 37 ; 36 ; 6 . To address these sophisticated attacks, a range of defense strategies has been proposed. Backdoor sample detection methods 16 ; 40 ; 42 ; 13 ; 24 typically identify anomalies in the distribution of image embeddings, which can effectively separate backdoor inputs from clean ones. However, these methods 16 ; 40 ; 42 heavily rely on access to the original pretraining data, which is often impractical for third-party pretrained vision encoders. To address this limitation, more recent paradigms perform inference-time detection with a small auxiliary dataset 13 ; 24 . Nevertheless, their effectiveness is highly sensitive to distributional alignment between the auxiliary data and the encoder’s pretraining data distribution. Without access to the original pretraining corpus, their performance is also limited in practical. This raises a critical question: Can backdoor inputs be reliably detected at inference time in a fully zero-shot setting? In this setting, we assume access only to the test input and a potentially compromised vision encoder, with no auxiliary data. We first examine how backdoor affect vision encoders by mechanistically analyzing their impacts on attention and image embeddings. Prior work 42 shows that patch triggers often localize attention (concentrate its attention on a small region, as shown in (b) of Fig. 2 ), but this cue does not generalize to blended triggers that are distributed across the whole image (see (c) of Fig. 2 ). Instead, we posit a shared property of both patch-based and blended attacks: attention is minimally distributed across benign features and predominantly concentrated on trigger-related features. We refer to this phenomenon as Attention Hijacking . Under hijacking, small perturbations leave the embedding nearly unchanged as long as the trigger remains active. When perturbations exceed the trigger’s robustness threshold, the trigger deactivates; attention rapidly reorients toward benign features, producing an abrupt embedding shift, which we call Attention Restoration . Consequently, under progressive masking, clean inputs exhibit smooth, gradual changes in attention and embeddings, whereas backdoor inputs undergo a sharp transition once masking suppresses the trigger. Motivated by this phenomenon induced by backdoors, we propose BackdoorIDS , a zero-shot inference-time detector for pretrained vision encoders. We progressively mask the input to generate a sequence of images and encode them to obtain an embedding trajectory. Clean inputs yield smoothly varying embeddings with roughly uniform adjacent distances, whereas backdoor inputs produce an initial tight cluster ( Attention Hijacking ) followed by an abrupt jump when the trigger deactivates ( Attention Restoration ). We set the DBSCAN radius using the scaled average local density of the top- k embeddings, and cluster all embeddings with DBSCAN 15 : clean inputs form one cluster, while backdoor inputs split into multiple clusters. We classify by cluster count (one = clean; >1 = backdoor). Across datasets, attacks, and encoder architectures (including LVLMs), BackdoorIDS outperforms baselines with strong robustness to noise and JPEG compression 50 . Comprehensively, we summarize our main contribution as follows: • We uncover Attention Hijacking and Attention Restoration , showing that backdoor and clean inputs exhibit markedly different embedding dynamics under the same progressive masking process. These distinctions enable inference-time backdoor samples detection without prior knowledge or auxiliary data. • Based on these observations, we propose BackdoorIDS , a zero-shot inference-time backdoor samples detector for pretrained vision encoders that is plug-and-play and requires neither additional data nor training. • Extensive experiments across datasets, attacks, and models show that BackdoorIDS consistently outperforms prior methods and is robust across various vision encoders (e.g., CNN 23 , ViT 14 , CLIP 41 , LLaVA-1.5 34 ) and tasks (e.g., classification, captioning). 2 Related Work 2.1 SSL Vision Encoder There are three mainstream types of SSL approaches: contrastive learning (CL), multimodal contrastive learning (MCL), and masked autoencoders (MAE) 21 . CL 4 ; 5 ; 10 learns representations by encouraging the embeddings of the same input to be close, while pushing apart those of different inputs. In MCL 38 ; 41 ; 53 , images and text are mapped into a shared embedding space, where matching image-text pairs are pulled together, and mismatched pairs are pushed apart. CLIP 41 is a representative form of MCL using natural