Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features

Shangbo Wu, Yu-an Tan, Ruinan Ma, Wencong Ma, Dehua Zhu, Yuanzhang Li · 2025

Read Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features by Shangbo Wu, Yu-an Tan, Ruinan Ma, Wencong Ma, Dehua Zhu, Yuanzhang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21046.txt.

Introduction

Six researchers present a new way to craft fake images that fool vision systems the attacker cannot see inside. Such transfer attacks build misleading inputs on one model and hope they also trick others. Older methods leaned on features learned under human supervision. This team instead mines features from vision transformers trained without labels, through two self-teaching styles: contrastive learning, which captures global structure, and masked image modeling, which captures local texture.

Their attack, called dSVA, trains a generator that disturbs both kinds of features at once, guided by the transformer's own attention maps. Because the two feature types highlight different aspects of an image, striking both makes the fake inputs generalize unusually well. In tests, the method fooled models of many designs better than prior leading attacks. The authors share their code openly for other researchers.

Catalog record

CREATORShangbo Wu, Yu-an Tan, Ruinan Ma, Wencong Ma, Dehua Zhu, Yuanzhang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features Shangbo Wu, Yu-an Tan, Ruinan Ma, Wencong Ma, Dehua Zhu, Yuanzhang Li Abstract The ability of deep neural networks (DNNs) come from extracting and interpreting features from the data provided. By exploiting intermediate features in DNNs instead of relying on hard labels, we craft adversarial perturbation that generalize more effectively, boosting black-box transferability. These features ubiquitously come from supervised learning in previous work. Inspired by the exceptional synergy between self-supervised learning and the Transformer architecture, this paper explores whether exploiting self-supervised Vision Transformer (ViT) representations can improve adversarial transferability. We present dSVA —a generative d ual s elf-supervised V iT features a ttack, that exploits both global structural features from contrastive learning (CL) and local textural features from masked image modeling (MIM), the self-supervised learning paradigm duo for ViTs. We design a novel generative training framework that incorporates a generator to create black-box adversarial examples, and strategies to train the generator by exploiting joint features and the attention mechanism of self-supervised ViTs. Our findings show that CL and MIM enable ViTs to attend to distinct feature tendencies, which, when exploited in tandem, boast great adversarial generalizability. By disrupting dual deep features distilled by self-supervised ViTs, we are rewarded with remarkable black-box transferability to models of various architectures that outperform state-of-the-arts. Code available at https://github.com/spencerwooo/dSVA . 1 Introduction The transferability of adversarial examples enable real-world black-box attacks on DNNs without the adversary’s access to their internals. Such attacks require the construction of a local white-box surrogate model. Consequently, their effectiveness relies on the ability to disrupt the shared latent representations, i.e., features, learnt by both models. DNNs learn sample-label correlations over their training process, by identifying the structure and semantic characteristics of the data for classification. These learnt deep features are generalizable enough to essentially serve as the basis that drive downstream tasks such as object detection [ 47 , 5 ] , similarity measurement [ 72 , 18 ] , image super-resolution [ 35 ] , and style transfer [ 19 ] . Prior research has shown that improving transferability is possible by targeting intermediate features of the surrogate model instead of directly attacking hard labels or output gradients [ 60 , 73 , 28 ] . Since deep features of well-trained DNNs are generalizable [ 69 ] , perturbation designed to disrupt these features are more transferable [ 29 ] . Figure 1 : Demonstration of dSVA . By jointly exploiting deep features of both self-supervised ViTs, i.e., DINO (CL) and MAE (MIM), dSVA crafts perturbation that disrupts both structural and textural representations of the image (as visualized in the attention saliency maps), fooling ConvNets, ViTs, and MLPs alike. The habitual inclusion of label-wise loss in existing work for conducting adversarial attacks acts as a common practice that pushes the surrogate model to be setup with supervised learning. This makes sense for ConvNets where self-supervised learning lags behind supervised. However, the advent of ViTs introduced the success of self-supervision in natural language processing to vision [ 4 , 25 , 9 , 7 ] . Supervised learning fails to preserve image semantics through human labelling, reducing feature-rich semantic information within images into a single concept represented by a human-assigned category. In contrast, self-supervised ViTs excel at capturing semantics, providing robust positional and semantic relationships throughout model layers, outperforming ConvNets [ 1 ] . Driven by the powerful adversarial potentials of self-supervised ViT features, we ask: How can we fully utilize the rich representations distilled by the harmonious coalition between self-supervision and the Transformer architecture, to boost adversarial transferability? We attempt to answer this research question in threefold: (1) Facet-level feature exploitation. ViTs comprise several layers of multi-head self-attention blocks that encode token-wise features. With a goal of extracting adversarially generalizable features, contrary to ConvNets where existing work use the direct output of entire intermediate layers, we propose to extract internal components, i.e. feature facets, of self-attention blocks in ViTs: queries, keys, and values. (2) Self-attention exploitation. The architectural design of self-attention empowers ViTs to capture semantic context of the image at a high level. We propose, atop the adversarial exploitation of internal facets in ViT blocks, to systematically extract saliency maps from the self-attention mechanism itself, and integrate them into loss optimization as dense semantic guides to identify valuable feature targets. (3) Joint self-supervision feature discrimination. Two branches of self-supervision paradigms exist for ViTs: contrastive learning (CL) and masked image modeling (MIM). Comparative studies show that CL captures global structural shapes and semantics, while MIM focuses more on local textural details [ 44 ] . We hypothesis that, if combined, both aspects will complement each other in generalizability that jointly contribute to enhancing adversarial transferability. Incorporating all three aspects, we introduce dSVA —a generative d ual s elf-supervised V iT features a ttack. We introduce a novel generative training framework, consisting of a generator to craft transferable adversarial perturbation, and discriminative training approaches to jointly exploit the dual intricate features—both structural and textural—distilled by the two types of self-supervised ViTs. We choose the duo: DINO [ 7 ] and MAE [ 25 ] , for CL and MIM respectively. Figure 1 showcases a birds-eye view of dSVA . Leveraging the powerful latent representations distilled by self-supervised ViTs, dSVA achieves outstanding adversarial effectiveness. We show an example in Fig. 1 of dSVA successfully disrupting both structural features from DINO (CL) and textural representations from MAE (MIM) (visualized in the attention maps), enabling impressive transferability towards black-box models of distinct architectures. Our experiments demonstrate dSVA ’s outstanding transferability to models across ViTs, ConvNets, and MLPs alike, and its ability to evade defenses, surpassing various state-of-the-arts. To conclude, we summarize our contributions as follows. • We present dSVA , a generative adversarial attack, that crafts highly transferable black-box adversarial examples by exploiting dual self-supervised ViT features. • We first aim at, instead of attacking the direct output of intermediate layers, targeting the internal facets of the self-attention blocks in ViTs, namely, the queries, keys, and values, to take advantage of the Transformer architecture and extract generalizable and transferable features. • We further introduce a method to exploit the self-attention mechanism itself by extracting saliency maps from the self-attention maps of ViTs, acting as guides for important feature targets, providing, in essence, a regularization scheme that enable boosted adversarial generalizability. • We finally propose to jointly exploit the two self-supervised learning schemes—CL and MIM—to craft perturbation that attend to and disrupt both global structural shapes and local textural details from within the image. 2 Related Work Figure 2 : The dSVA Training Framework. Sample \bm{x} is fed through \mathcal{G} to create adversarial example \bm{x}^{adv} , which are then both fed into the self-supervised models DINO and MAE, to extract deep representations and attention saliency maps from both global structural and local textural feature aspects. The feature discriminative loss is derived from both ViTs, which jointly form the adversarial loss \mathcal{L}_{\text{adv}} . Generative adversarial attacks is initially introduced in Poursaeed et al. 2018 to address both sample-agnostic and sample-specific adversarial perturbation. This approach paved the way for generative methods in creating unrestricted perturbations [ 52 ] and utilizing GANs [ 64 ] . The generative

Read the complete text