TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization
Amira Guesmi, Bassem Ouni, Muhammad Shafique · 2025
Read TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization by Amira Guesmi, Bassem Ouni, Muhammad Shafique (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19613.txt.
Introduction
Amira Guesmi, Bassem Ouni, and Muhammad Shafique present TESSER, an attack that crafts image tweaks on one model and fools many others. Such transfer attacks matter for security, since they need no peek inside the target. Attacks brewed on vision transformers often fail to cross to older network kinds. Their 2025 frame adds two guards to the tweaks.
One scales the tweak by how much each image patch matters to the model. The other smooths the tweak's high-frequency noise with a soft blur rule. On a big image set across twelve designs, TESSER beat the top past method on both network families. It also cut through defended models at a strong rate. Its tweaks sit on the image parts humans look at too, the team shows.
Catalog record
| CREATOR | Amira Guesmi, Bassem Ouni, Muhammad Shafique |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization Amira Guesmi, Bassem Ouni, Muhammad Shafique Abstract Adversarial transferability remains a critical challenge in evaluating the robustness of deep neural networks. In security-critical applications, transferability enables black-box attacks without access to model internals, making it a key concern for real-world adversarial threat assessment. While Vision Transformers (ViTs) have demonstrated strong adversarial performance, existing attacks often fail to transfer effectively across architectures, especially from ViTs to Convolutional Neural Networks (CNNs) or hybrid models. In this paper, we introduce TESSER — a novel adversarial attack framework that enhances transferability via two key strategies: (1) Feature-Sensitive Gradient Scaling (FSGS) , which modulates gradients based on token-wise importance derived from intermediate feature activations, and (2) Spectral Smoothness Regularization (SSR) , which suppresses high-frequency noise in perturbations using a differentiable Gaussian prior. These components work in tandem to generate perturbations that are both semantically meaningful and spectrally smooth. Extensive experiments on ImageNet across 12 diverse architectures demonstrate that TESSER achieves +10.9% higher attack succes rate (ASR) on CNNs and +7.2% on ViTs compared to the state-of-the-art Adaptive Token Tuning (ATT) method. Moreover, TESSER significantly improves robustness against defended models, achieving 53.55% ASR on adversarially trained CNNs. Qualitative analysis shows strong alignment between TESSER’s perturbations and salient visual regions identified via Grad-CAM, while frequency-domain analysis reveals a 12% reduction in high-frequency energy, confirming the effectiveness of spectral regularization. 1 Introduction Deep learning models, particularly Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), have achieved state-of-the-art performance across a broad spectrum of computer vision tasks [ 3 , 54 , 25 ] . Despite this progress, these models remain highly vulnerable to adversarial examples – carefully crafted perturbations that are imperceptible to humans but cause misclassification [ 12 , 14 , 15 , 16 ] . In safety-critical applications such as autonomous driving and medical imaging, this fragility raises significant security concerns. Although white-box attacks, where attackers have full access to model parameters, have been extensively studied, black-box settings are more realistic in practice. These are based on the principle of transferability , where adversarial examples generated on a surrogate model are expected to fool unseen target models. However, transferability across architectures, especially from ViTs to CNNs or hybrid models, remains limited due to two key challenges: (1) the lack of semantic selectivity , where all tokens are perturbed uniformly without considering their relevance to the model’s prediction, and (2) the presence of high-frequency noise in perturbations, which tends to encode brittle, model-specific artifacts that do not generalize well. Several recent works, such as ATT [ 27 ] and TGR [ 53 ] , have explored ViT-specific mechanisms for improving transferability by truncating or regularizing gradient flows. However, these approaches either use fixed gradient masks or overlook token-level semantics, leading to suboptimal alignment with transferable visual features. In this paper, we introduce TESSER ( Transfer-Enhancing Semantic and Spectral Regularization ) a novel adversarial attack framework specifically designed to improve black-box transferability from ViT-based models to a diverse set of architectures. TESSER integrates two complementary strategies: Feature-Sensitive Gradient Scaling (FSGS) : a token-level gradient modulation method that scales gradients based on token importance derived from intermediate embeddings. Inspired by recent findings correlating token activation magnitudes with semantic relevance [ 22 , 46 , 29 ] , FSGS steers the attack toward semantically meaningful regions and away from background or non-informative tokens, enhancing cross-model generalization. Spectral Smoothness Regularization (SSR) : a lightweight regularization mechanism that applies a differentiable Gaussian blur during each optimization step. SSR suppresses high-frequency noise, promoting low-frequency perturbations that are more resilient across architectures, particularly beneficial when transferring to CNNs and adversarially trained models. Together, these modules enable TESSER to produce perturbations that are semantically aligned and spectrally smooth, two characteristics that we empirically demonstrate to be critical for enhancing transferability in adversarial attacks. Our main contributions are summarized as follows: • We propose TESSER , a novel adversarial attack framework that combines semantic- and spectral-aware regularization to improve transferability from ViTs. • We introduce Feature-Sensitive Gradient Scaling (FSGS), which reweights gradients for Attention, QKV, and MLP modules based on token-level importance, encouraging semantically aligned perturbations. • We incorporate Spectral Smoothness Regularization (SSR) to reduce high-frequency noise and enhance cross-architecture generalization. • We conduct extensive experiments on ImageNet across 12 diverse models (including ViTs, CNNs, and adversarially defended CNNs), demonstrating that TESSER achieves up to +10.9% higher ASR over state-of-the-art baselines and consistently outperforms existing attacks in both black-box and robust scenarios. • We conduct comprehensive ablation studies, Grad-CAM-based semantic alignment evaluations (Section 4.4 ), and frequency-domain analyses (Section 4.5 ) to demonstrate both the effectiveness and interpretability of our approach. Figure 1: Overview of the TESSER attack framework. At each iteration, an adversarial perturbation \delta^{t} is applied to the input image and smoothed via differentiable Gaussian blur \mathcal{G}_{\sigma}(\cdot) to enforce spectral smoothness (SSR). The perturbed input is passed through the transformer, where token embeddings Z_{l} from each layer are used to compute token-wise importance scores \hat{\alpha} , which in turn define gradient scaling masks S . During backpropagation, gradients for the Attention, QKV, and MLP modules are reweighted according to their respective scaling masks ( S_{\text{Attention}},S_{\text{QKV}},S_{\text{MLP}} ) using Feature-Sensitive Gradient Scaling (FSGS). This encourages perturbations to align with semantically meaningful and transferable features while suppressing noise and irrelevant gradients. 2 Related Work Adversarial Attacks on CNNs and ViTs. Adversarial attacks are small, human-imperceptible perturbations intentionally added to input data to mislead deep learning models [ 12 ] . For Convolutional Neural Networks (CNNs), numerous gradient-based attacks have been proposed to improve transferability, including momentum-based methods [ 6 ] , variance tuning [ 20 ] , and gradient skipping techniques [ 45 ] . These methods aim to stabilize perturbation updates and avoid local optima in the input space. However, attack techniques designed for CNNs do not transfer well to Vision Transformers (ViTs), which have fundamentally different architectures and information flow patterns. Recent works have proposed ViT-specific attacks that exploit token structure and attention mechanisms [ 31 , 43 ] . For example, Token Gradient Regularization (TGR) [ 53 ] modifies intermediate-layer gradients to reduce token-wise variance, improving transferability within ViT families. Regularizing gradients is an effective way to suppress model-specific patterns and improve cross-model generalization. In CNNs, methods like SGM [ 45 ] and BPA [ 47 ] aim to manipulate the gradient flow through skip connections or rectify distortions introduced by nonlinearities. Others have employed gradient variance reduction [ 20 ] and ensemble-based tuning [ 49 ] . Attacks based on feature information [ 42 , 11 ] focus on disrupting salient internal representations. However, improperly guided feature-based attacks risk discarding useful information and reducing transferability. To mitigate this, neuron attribution methods [ 53 ] and attention map diversification [ 33 ] have been explored, particularly in ViTs. ATT [ 27 ] introduces hybrid token gradient truncation by weakening gradients in attention and QKV blocks across layers of a ViT model. It leverages empirical observations of gradient variance to suppress high-magnitude gradients associated with overfitting, thereby improving transferability. However, ATT applies static truncation and does not explicitly consider token-level semantic relevance, which may limit its effectiveness when generalizing across diverse architectures. In