X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP
Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey · 2025
Read X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP by Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05528.txt.
Introduction
Hanxun Huang and co-authors present a 2025 arXiv paper on a universal attack against CLIP vision-language models. A single crafted noise pattern, they show, can fool many CLIP encoders and the larger systems built on them, across images, tasks, and domains at once, a trait they call super transferability.
The trick is surrogate scaling: instead of a few fixed stand-in models, the attack picks a small, well-chosen subset from a large pool on the fly, keeping costs down. Benchmarks show the method beating earlier universal attacks by a wide margin. The authors release their code and frame the result as a warning for deployed systems.
Catalog record
| CREATOR | Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05528
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey Abstract As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for diverse downstream tasks and integrated into large vision-language models (VLMs), their susceptibility to adversarial perturbations has emerged as a critical concern. In this work, we introduce X-Transfer , a novel attack method that exposes a universal adversarial vulnerability in CLIP. X-Transfer generates a Universal Adversarial Perturbation (UAP) capable of deceiving various CLIP encoders and downstream VLMs across different samples, tasks, and domains. We refer to this property as super transferability —a single perturbation achieving cross-data, cross-domain, cross-model, and cross-task adversarial transferability simultaneously. This is achieved through surrogate scaling , a key innovation of our approach. Unlike existing methods that rely on fixed surrogate models, which are computationally intensive to scale, X-Transfer employs an efficient surrogate scaling strategy that dynamically selects a small subset of suitable surrogates from a large search space. Extensive evaluations demonstrate that X-Transfer significantly outperforms previous state-of-the-art UAP methods, establishing a new benchmark for adversarial transferability across CLIP models. The code is publicly available in our GitHub repository . 1 Introduction Contrastive Language-Image Pre-training (CLIP) is a widely adopted technique that learns aligned multi-modal representations from text-image pairs through contrastive learning ( Radford et al., 2021 ) . Pre-trained on web-scale datasets, CLIP encoders have been extensively used to enhance performance across a variety of downstream applications, particularly in large Vision-Language Models (VLMs) ( Awadalla et al., 2023 ; Koh et al., 2023 ; Wang et al., 2023a ; Bai et al., 2023 ; Karamcheti et al., 2024 ; Jiang et al., 2024 ) , where they form the backbone of visual capabilities. Models such as Flamingo ( Alayrac et al., 2022 ) , LLaVA ( Liu et al., 2023a ) , BLIP2 ( Li et al., 2023a ) , and MiniGPT-4 ( Zhu et al., 2024 ) integrate CLIP image encoders with Large Language Models (LLMs) ( Zhang et al., 2022b ; Hoffmann et al., 2022 ; Chiang et al., 2023 ) . The widespread adoption of CLIP in VLMs is primarily driven by its pre-training paradigm utilising text supervision ( Tong et al., 2024 ) . While its strong generalisation capabilities solidify its role as a cornerstone for VLMs, they also make CLIP an ideal target for generating highly transferable adversarial perturbations, thereby introducing new safety risks. Deep neural networks are widely recognised for their susceptibility to Universal Adversarial Perturbations (UAPs) ( Moosavi-Dezfooli et al., 2017 ; Gao et al., 2023 ; Zhou et al., 2023b ; Zhou et al., 2024 ; Zhang et al., 2025c ; Song et al., 2025 ) , where a perturbation generated using a specific dataset can transfer to images within the same domain, causing erroneous classifications by image classifiers. Recent studies ( Fang et al., 2024b ; Zhang et al., 2024 ) have demonstrated that UAPs are also effective against CLIP encoders. However, existing works have yet to fully realise the potential of UAPs—achieving super transferability. Ensemble techniques are a well-established strategy for enhancing cross-model adversarial transferability ( Liu et al., 2017 ; Dong et al., 2018 ; Xiong et al., 2022 ; Chen et al., 2024a ) for sample-specific perturbations, but they leave significant gaps in the applicability of UAPs to broader transfer scenarios. Furthermore, these methods rely on a heuristic selection of a fixed set of surrogate models, which becomes computationally expensive when scaling to a large number of surrogates ( Liu et al., 2024 ) . To address these gaps, we aim to answer the following two questions: (1) Can a single perturbation simultaneously achieve cross-data, cross-domain, cross-model, and cross-task adversarial super transferability? and (2) How scalable is super transferability when incorporating large numbers of surrogate models? Figure 1 : Adversarial super transferability achieved by X-Transfer with different configurations (Base, Mid and Large). The figure reports the attack success rate (ASR) with a single UAP applied to different samples, datasets, models, and tasks. ZS: zero-shot classification; IR : image-retrieval; TR : text-retrieval; IC : image captioning, and VQA : visual question answering. The ZS, IR, and TR are evaluated with CLIP encoders. IC and VQA are evaluated with large VLMs. Results for baseline methods, GD-UAP ( Mopuri et al., 2018 ) , TRM-UAP ( Liu et al., 2023b ) , AdvCLIP ( Zhou et al., 2023a ) , Meta-UAP ( Weng et al., 2024 ) , C-PGC ( Fang et al., 2024b ) and ETU ( Zhang et al., 2024 ) are the best results across their various configurations. Results are averaged over multiple victim models. A larger shaded circle indicates a higher universal ASR. In this work, we propose the X-Transfer attack, a novel attack method that generates UAPs via an efficient surrogate scaling strategy applied to a large number of surrogate models. Specifically, X-Transfer dynamically selects a small subset of suitable surrogate CLIP encoders from a large search space, enabling the efficient scaling of surrogate models for UAP generation. The UAPs and Targeted UAPs (TUAPs) generated by X-Transfer achieve black-box adversarial super transferability. Extensive evaluations demonstrate that X-Transfer significantly outperforms state-of-the-art UAP methods designed for image classifiers ( Mopuri et al., 2018 ; Liu et al., 2023b ; Weng et al., 2024 ) and methods tailored for CLIP encoders ( Zhou et al., 2023a ; Fang et al., 2024b ; Zhang et al., 2024 ) , achieving improved performance by a substantial margin, as shown in Figure 1 . Our work is the first to demonstrate the existence of super UAPs that transfer across data (samples from in-domain datasets), domains (datasets), models (including both CLIP encoders and VLMs), and tasks (e.g., zero-shot classification, image-text retrieval, image captioning, and visual question answering, VQA). Our search space configurations—Base, Mid, and Large—consist of 16, 32, and 64 surrogate encoders, respectively. The super transferability scales with the total number of surrogate encoders in the search space. Furthermore, X-Transfer achieves this super transferability while selecting as few as a single surrogate encoder per optimisation step. These findings uncover a new vulnerability in CLIP models and their applications. In summary, our main contributions are as follows: • We investigate the universal vulnerability of CLIP models and propose a novel attack method called X-Transfer to generate UAPs that can transfer across data, domains, models, and tasks. • X-Transfer introduces an innovative surrogate scaling strategy that efficiently scales transferability with the number of surrogate models by dynamically selecting suitable candidates at each UAP generation step. • We conduct extensive experiments to demonstrate the effectiveness of X-Transfer and provide in-depth insights and interpretations of the generated UAP patterns. Building on this, we establish a new benchmark, X-TransferBench , which offers a comprehensive, open-source collection of UAPs and TUAPs for super transferability studies. 2 Relate Work Contrastive Language-Image Pre-training. CLIP ( Radford et al., 2021 ) is a popular framework that can pre-train on web-scale text-image pairs via contrastive learning ( Chopra et al., 2005 ; Oord et al., 2018 ; Chen et al., 2020b ) . Encoders pre-trained by CLIP have demonstrated superior zero-shot generalisation capability in a wide range of downstream tasks ( Palatucci et al., 2009 ; Lampert et al., 2009