Boosting targeted adversarial transferability via fine-grained feature mixup perturbation and reference-based gradient refinement Authors: Xining Gao, Sen Huangfu, Ying Zhao, Zijiao Zhang Published: 2026 ABSTRACT AbstractDeep neural networks are vulnerable to transferable adversarial examples in black-box scenarios, and targeted attacks that mislead models into predicting specific classes pose particularly severe threats. Feature mixup attacks enhance adversarial transferability by injecting clean features to perturb intermediate representations, yet existing methods suffer from two fundamental limitations: coarse-grained global mixing strategies apply a shared mixing ratio uniformly across all spatial positions, fixing the clean reference for each position to its corresponding location in the clean image and thus limiting the diversity of mixed feature representations during optimization and the transferability of the generated adversarial examples; and standard momentum-based optimization over-aligns with the surrogate model’s gradient geometry, suppressing gradient variations essential for escaping model-specific local minima. We propose Fine-grained Feature Mixup Perturbation and Reference-based Gradient Refinement (FMGR) to address both limitations. FFM partitions feature maps into spatially disjoint blocks and mixes each block with clean features drawn from spatially shuffled positions of the same image, breaking the fixed spatial correspondence of clean references and producing more generalizable feature perturbations. RGR selectively amplifies deviations between instantaneous and reference gradients, suppressing surrogate-specific dominant directions and steering optimization toward flatter, more transferable loss regions. Extensive experiments on ImageNet and CIFAR-10 demonstrate that FMGR significantly outperforms state-of-the-art methods against both CNN and Vision Transformer architectures while maintaining computational efficiency.Subject terms: Engineering, Mathematics and computing Introduction Deep neural networks (DNNs)1 have achieved remarkable success across a wide range of domains, including computer vision, natural language processing, and autonomous driving systems. However, their widespread deployment has simultaneously raised serious concerns about security and robustness. Studies have shown that DNNs are inherently vulnerable to adversarial examples—inputs crafted with imperceptible perturbations that are sufficient to induce misclassification, as illustrated in Fig. 12. This threat is particularly pronounced in safety-critical applications such as autonomous driving3, medical diagnosis4, and face recognition5, where adversarial exploitation could lead to severe real-world consequences. Furthermore, images encountered in such real-world deployments are often subject to quality degradations such as motion blur, compression artifacts, and sensor noise6–8, which may compound the challenges of robust model evaluation by simultaneously distorting input fidelity and altering the gradient signals that adversarial attacks rely upon. Therefore, a deeper understanding of adversarial attack mechanisms and the development of more effective attack methods are not only essential for evaluating model robustness9,10, but also serve as a necessary foundation for building safer and more reliable deep learning systems11. Adversarial attack example: the original panda image is misclassified after adding imperceptible perturbations. Adversarial attacks can be broadly categorized into white-box attacks12,13, and black-box attacks14–16. White-box attacks assume that the adversary has full access to the target model’s architecture and parameters17, whereas black-box attacks are more prevalent in real-world scenarios where such information is unavailable18. A key property that underlies effective black-box attacks is transferability: adversarial examples crafted on a surrogate model can often fool other unknown target models19. Consequently, improving transferability is critical for assessing the security vulnerabilities of DNNs in practical deployment settings20. Existing research has explored transferability enhancement from multiple perspectives. Gradient-based methods21–23 stabilize the optimization process through momentum accumulation; however, by continuously reinforcing historically consistent gradient directions, they progressively suppress the gradient variation necessary to escape surrogate-model-specific local optima, causing the optimization trajectory to over-align with the surrogate model’s decision boundary and thus limiting generalization to unknown target models. Input transformation methods mitigate overfitting through diversified data augmentation, but operate solely in the pixel space and fail to fully exploit the rich semantic information embedded in deep feature representations. In contrast, feature-space attacks24 directly perturb intermediate layer representations to disrupt the model’s internal decision-making process, demonstrating superior transferability. Among them, Clean Feature Mixup (CFM)25 method mixes clean and adversarial features to prevent overfitting, and Feature Tuning Mixup (FTM)26 further introduces layer-wise learnable perturbations for more targeted feature disruption. Nevertheless, both methods adopt a channel-wise global mixing strategy that uniformly applies a shared mixing ratio across all spatial positions, such that the clean reference at every position is always drawn from the same corresponding location in the clean image. This positional rigidity reduces the variation of mixed feature representations across iterations, making it difficult to prevent the adversarial perturbation from overfitting to surrogate-specific spatial patterns. Motivated by these observations, we propose Fine-grained Feature Mixup Perturbation and Reference-based Gradient Refinement (FMGR). We provide an overview of the proposed FMGR in Fig 2. During the forward pass of each iteration, for each selected layer, the Adaptive Momentum Perturbation (AMP) module first partitions the feature map into spatial blocks and applies momentum-based perturbations independently to each block, where the perturbation magnitude of each block is adaptively normalized according to its local feature activation, preventing under-perturbation in low-activation regions and over-perturbation in high-activation regions. The learned perturbations are normalized and scaled according to each block’s contribution to the model output, directing the attack toward the most vulnerable regions and realizing block-level adaptive feature perturbation. Subsequently, the Fine-grained Feature Mixup (FFM) module applies a random spatial permutation to the feature blocks extracted from a randomly selected clean image. The spatially shuffled clean feature blocks are then mixed with the AMP-perturbed feature blocks at channel-wise random ratios, and the resulting blocks are concatenated to reconstruct the complete feature map, simulating competitive noise in the feature space and preventing overfitting to any specific surrogate model. During the backward pass, after obtaining the gradients of the adversarial examples, the Reference Gradient Rectification (RGR) module maintains a historical reference gradient via exponential moving average. It computes the deviation between the current gradient and the reference gradient, and adds the scaled deviation back to the current gradient, suppressing surrogate-model-specific gradient components that are highly consistent with the historical direction and guiding the optimization trajectory toward flatter, more transferable loss regions. Overview of the FMGR framework. During the forward pass, AMP applies momentum-based perturbations to spatial blocks with magnitudes adaptively scaled by local feature activation, after which FFM mixes each perturbed block with spatially shuffled clean features at random channel-wise ratios. During the backward pass, RGR amplifies the deviation between the current gradient and an EMA-based reference , suppressing surrogate-dominant directions and steering optimization toward flatter, more transferable loss regions. The main contributions of this paper are summarized as follows: We propose Fine-grained Feature Mixup (FFM), which partitions feature maps into non-overlapping spatial blocks and mixes each block with clean features drawn from spatially shuffled positions of the same image, increasing the diversity of mixed feature representations during optimization and enhancing adversarial transferability especially for patch-based architectures such as Vision Transformers. We propose Fine-grained Feature Mixup (FFM), which partitions feature maps into non-overlapping spatial blocks and mixes each block with clean features drawn from spatially shuffled positions of the same image, increasing the diversity of mixed feature representations during optimization and enhancing adversarial transferability especially for patch-based architectures such as Vision Transformers. We propose Adaptive Momentum Perturbation (AMP), which learns attack-specific perturbations at the block level via momentum-based stochastic optimization, combining targeted attack effectiveness with implicit regularization to enhance transferability. We propose Adaptive Momentum Perturbation (AMP), which learns attack-specific perturbations at the block level via momentum-based stochastic optimization, combining targeted attack effectiveness with implicit regularization to enhance transferability. We propose Reference Gradient Rectification (RGR), which amplifies the deviation between instantaneous gradients and a smoothed reference direction, suppressing dominant surrogate-model-specific gradient components and guiding optimization toward flatter, more transferable loss regions. We propose Reference Gradient Rectification (RGR), which amplifies the deviation between instantaneous gradients and a smoothed reference direction, suppressing dominant surrogate-model-specific gradient components and guiding optimization toward flatter, more transferable loss regions. Comprehensive experiments on the ImageNet-compatible and CIFAR-10 datasets demonstrate that FMGR achieves state-of-the-art performance against both CNN and Vision Transformer architectures. Detailed ablation studies further validate the effectiveness of each proposed component. Related work In this section, we review the existing literature on adversarial attacks, with a particular focus on methods that enhance transferability in black-box scenarios. We organize the discussion into four main categories: gradient-based attacks, input transformation attacks, feature space attacks, and feature mixing methods. Gradient-based adversarial attacks Gradient-based attacks exploit gradient information from surrogate models to generate adversarial perturbations. The Fast Gradient Sign Method (FGSM)27 perturbs images along the gradient direction to maximize the loss function. Subsequently, Iterative FGSM (I-FGSM)28 extends the single-step attack to multi-step iterations, employing small-magnitude perturbations over multiple steps. However, I-FGSM often overfits to the surrogate model, easily getting trapped in local optima, which leads to poor transferability to unseen models. To mitigate overfitting and stabilize the optimization direction, Momentum Iterative FGSM (MI-FGSM)29 introduces momentum mechanisms into gradient computation, smoothing the update trajectory by accumulating historical gradient information. Variance Tuning (VT)30 proposes computing gradients at randomly sampled points around the adversarial example, stabilizing the optimization process and alleviating overfitting by reducing gradient variance. Although momentum-based methods reduce optimization oscillations by accumulating historical gradients, they simultaneously reinforce surrogate-specific dominant gradient directions, progressively trapping adversarial examples in local minima. Input transformation attacks Input transformation methods enhance transferability by applying diverse transformations to the input space, preventing overfitting to specific surrogate models. Diverse Inputs (DI)31 randomly resizes and pads input images with probability p at each iteration, effectively enhancing sample robustness. Resized Diverse Inputs (RDI)32 improves upon DI by fixing probability p to 1 and rescaling transformed images back to their original size, ensuring consistent input dimensions while maintaining diversity. Translation-Invariant (TI)33 methods apply translation transformations through convolutional kernels to generate position-insensitive robust perturbations. Scale-Invariant (SI)34 attacks optimize perturbations by averaging gradients across multiple scales. Self-Universality (SU)35 attacks propose enhancing adversarial perturbations by maximizing feature similarity between global images and local regions. Object-based Diverse Inputs (ODI)36 applies transformation diversity to object regions within images rather than entire images. Admix37 attacks further enhance input diversity by mixing input images with randomly sampled images from the dataset. While these input transformation methods effectively enhance robustness and transferability, they primarily operate in pixel space and rarely exploit the rich structural and semantic information embedded in deep feature representations. Feature space attacks Recognizing the limitations of pixel-space methods, recent research has shifted toward feature space perturbations. Studies have found that intermediate-layer feature representations carry rich semantic information crucial for model decisions, and perturbing intermediate activations can achieve better transferability than pure input-space methods. Intermediate Level Attack (ILA)38 generates perturbations by matching intermediate features with pre-computed features of target adversarial examples. Feature Importance-aware Attack (FIA)39 identifies and perturbs important features in intermediate layers based on gradient magnitude, focusing on features that contribute most to correct classification. Neuron Attribution-based Attack (NAA)40 perturbs neurons with high attribution scores to disrupt critical decision paths. Feature Disruptive Attack (FDA)41 disrupts style representations by maximizing differences in Gram matrices of feature maps. Ghost Networks42 proposes injecting noise at feature layers to simulate ensemble effects, thereby improving black-box transferability. Although these methods show promise, most feature space attacks focus on identifying which features to perturb rather than how to optimally perturb them for maximum transferability. Furthermore, these methods typically apply global perturbations uniformly across feature maps, ignoring the spatial heterogeneity inherent in convolutional and attention-based architectures. Feature mixing attacks The concept of feature mixing in adversarial attacks was pioneered by Byun et al., who introduced Clean Feature Mixup (CFM)25. CFM stores clean features from benign images and mixes them with adversarial features during optimization, where the underlying principle is that clean features act as regularizers to prevent overfitting and promote the generation of more transferable adversarial examples. Recently, Zhang et al. proposed Feature Tuning Mixup (FTM)26, which introduces learnable perturbations independently optimized for each layer, learning attack-optimized feature perturbations for more precise control, with its core innovation lying in layer-wise tunable parameters that adapt to the specific characteristics of each network layer. However, both CFM and FTM apply a shared channel-wise mixing ratio uniformly across all spatial positions of the feature map, and the clean reference feature at each position always originates from the same corresponding spatial location of the clean image. This global uniformity means that at every iteration the surrogate model sees clean references with an identical spatial structure, offering limited variation to prevent overfitting to surrogate-specific spatial patterns and thereby constraining the transferability of the generated adversarial examples. This limitation is particularly evident for patch-based architectures such as Vision Transformer (ViT), which process spatially independent patch tokens and therefore benefit more from spatially varied feature perturbations during optimization. Methods Preliminaries Problem formulation For targeted transfer-based attacks, we assume that the attacker has access to a white-box surrogate model F, which is a pre-trained image classifier. Given a benign image x with true label y and a target label , the goal of the attack is to generate an adversarial example that is misclassified as by model F. To ensure the imperceptibility of the adversarial example, we require that the difference between x and is sufficiently small. Following existing works, we adopt the -norm constraint to limit the perturbation magnitude. Therefore, the optimization objective of targeted transfer-based attacks can be formulated as: where denotes the adversarial loss function (e.g., cross-entropy loss or logit loss), and is the perturbation budget that controls the maximum magnitude of adversarial perturbations. MI-FGSM To address the above optimization problem, the Momentum Iterative Fast Gradient Sign Method (MI-FGSM) incorporates a momentum term that accumulates historical gradients, which stabilizes the update direction and mitigates gradient oscillations commonly encountered in iterative adversarial attacks. In the i-th iteration, the MI-FGSM update process is as follows: where is the momentum decay factor that controls the influence weight of historical gradients; is the step size that determines the update magnitude in each iteration; function extracts the sign of gradients; and function ensures that the generated adversarial example remains within the -ball neighborhood around the original image x. Initially, we set and . Fine-grained Feature Mixing (FFM) Existing feature mixing attacks apply a shared channel-wise mixing ratio uniformly to all spatial positions of the feature map, so the clean reference feature at each position always originates from the same spatial location of the clean image. Consequently, across all iterations the surrogate model receives clean references with a fixed spatial structure, which limits the diversity of mixed feature representations and allows the adversarial perturbation to exploit stable, position-specific feature patterns of the surrogate model. To address this, we propose Fine-grained Feature Mixup (FFM), which increases the diversity of mixed feature representations through two complementary structural mechanisms: block-based partitioning assigns each spatial block an independent mixing operation decoupled from other blocks, and intra-image spatial shuffling randomizes which spatial position of the clean image supplies the reference for each adversarial block. FFM does not learn region importance or model semantic heterogeneity explicitly; rather, it structurally disrupts the fixed spatial correspondence between adversarial features and their clean references, compelling the optimization to remain robust across varying spatial feature combinations. Block-based Partitioning. FFM partitions the feature map into a grid of non-overlapping spatial blocks: where is the feature block at position (m, n). This decomposition enables spatially differentiated perturbations, compelling the adversarial example to maintain robustness over spatially varying feature distributions rather than over-fitting to globally dominant features. Intra-image Spatial Shuffling. FFM mixes each adversarial feature map with the spatially shuffled clean features of the same image. Specifically, prior to the iterative optimization, a single forward pass is performed on the clean input batch to store the clean feature map at each selected layer for every image i in the batch. At each iteration, the stored clean feature map of image i is partitioned into the same block grid, and a random permutation , shared across all images in the batch, is applied to rearrange the blocks spatially: where denotes the clean feature block of image i relocated from the shuffled position . This intra-image shuffling preserves the local fidelity of each individual block–since all blocks still originate from the same clean image–while disrupting the global spatial structure of the clean reference. As a result, the attack gradient cannot over-rely on the surrogate model’s specific responses to fixed spatial layouts, and is instead compelled to seek perturbation directions that remain effective across diverse spatial feature combinations. This constitutes a form of structured regularization in the feature space: by introducing spatial diversity into the mixed representations at each iteration, the gradient updates automatically filter out surrogate-specific spatial dependencies and converge toward attack directions shared across model architectures, thereby promoting the transferability of the generated adversarial examples. Block-wise Feature Mixing. For each block, feature mixing is performed as: where is a channel-wise mixing ratio sampled from , and denotes element-wise multiplication. The mixing ratio vector is shared across all spatial blocks within the same layer and iteration, while the spatial shuffling ensures position-dependent diversity in the clean reference features. This design decouples the sources of randomness: channel-wise ratios control the mixing intensity per feature channel, while block-level shuffling controls the spatial origin of the clean reference, jointly producing richer and more diverse interference than global uniform mixing. Feature Reassembly. The mixed blocks are spatially concatenated to reconstruct the full feature map: The reassembled feature map incorporates heterogeneous perturbations from different spatial positions while maintaining spatial continuity, effectively preventing the feature homogenization that arises from global uniform mixing. Since all shuffled blocks originate from the same clean image, only one additional forward pass per selected layer is required, keeping the computational overhead minimal. Adaptive Momentum Perturbation (AMP) Although FFM increases the diversity of mixed feature representations and prevents overfitting through block-wise clean feature mixing, it provides no directional guidance toward the attack objective. We therefore propose Adaptive Momentum Perturbation (AMP), which serves a dual purpose: it complements FFM by providing attack-specific directional guidance that drives the adversarial example toward the target class, and simultaneously acts as an implicit regularizer that reduces overfitting of the pixel-level perturbation to the surrogate model. Block-wise Adaptive Perturbation Optimization. For each selected layer k, we maintain a learnable perturbation of the same shape as the layer’s output feature map, which is iteratively refined throughout the attack. To apply this perturbation, the feature map is partitioned into spatial blocks via unfolding, and the perturbation is applied to each block with an independent normalization scale that adapts to the local feature magnitude of that block: where and denote the (m, n)-th block slices of and respectively, is a scaling factor controlling perturbation magnitude, and is a small constant for numerical stability. The perturbed blocks are then reassembled via folding to recover the full feature map . This block-wise adaptive normalization ensures that the perturbation strength automatically scales with the local feature magnitude of each spatial block, applying larger perturbations in regions with stronger feature responses and smaller perturbations in regions with weaker feature responses. This allows the learnable perturbations to be distributed more appropriately across spatial locations, avoiding the spatial imbalance caused by global uniform scaling. Momentum-based Stochastic Update. Optimizing these perturbations naively would require additional forward and backward passes per layer, leading to substantial computational overhead. We therefore employ a momentum-based stochastic update strategy: perturbations from the previous iteration are carried forward as momentum initialization, and only layers triggered in the forward pass (with probability p) participate in the update. This enables one-step optimization jointly with the adversarial example in a single backward pass. Formally, we denote the set of triggered perturbations as , where is the random variable sampled in the forward pass determining whether layer k is selected. The gradients with respect to both the adversarial example and all triggered perturbations are then computed jointly: The perturbation update rule for layer k is: where is the gradient with respect to the perturbation of layer k. Layers not selected in the current iteration retain their perturbations unchanged from the previous iteration. This stochastic selection reduces computational overhead while acting as an implicit regularizer that minimizes the expected adversarial loss over the distribution of layer subsets, preventing the learned perturbations from overfitting to the surrogate model’s optimization trajectory. Integration with FFM. The learned perturbations are seamlessly integrated into the FFM framework. In the forward pass, after block-wise adaptive perturbation optimization, the perturbed feature blocks are further mixed with spatially shuffled fine-grained clean features. The complete forward process for each block is: In each iteration, we randomly sample for each layer k to jointly determine both the perturbation update and the clean feature mixup. The same threshold p governs both processes, ensuring that only layers selected for perturbation update also undergo feature mixing, while unselected layers retain their previous perturbations without mixing. AMP learns attack-specific perturbations to drive the adversarial example toward the target class, while FFM introduces controlled stochasticity through spatial shuffling and fine-grained clean feature mixing to prevent overfitting. Their combination yields a more challenging and diverse optimization landscape that promotes adversarial transferability. Reference-based Gradient Refinement (RGR) As iterations accumulate, momentum methods continuously reinforce historically consistent gradient directions, which stabilizes optimization in flat loss regions but becomes a hindrance in surrogate-specific sharp regions: when the gradient direction changes abruptly near a sharp minimum, the accumulated momentum retains inertia from historical directions and resists the sudden shift, slowing escape from surrogate-specific sharp minima and delaying convergence toward flatter, more transferable loss regions. To address this, we propose Reference-based Gradient Refinement (RGR), a lightweight gradient correction module that requires no additional forward or backward passes. Reference Gradient Estimation. RGR maintains a reference direction via an exponential moving average (EMA) of historical gradients to capture the long-term stable trend of the optimization path: where is the smoothing coefficient and . With increasing iterations, the reference gradient integrates the directional information of all historical gradients, forming a stable estimate of the surrogate-aligned optimization trend. Deviation Extraction and Amplification. The deviation between the current gradient and the reference captures newly emerged components that have not yet been accumulated by prior momentum, making them more likely to generalize across model architectures. RGR extracts this deviation and amplifies it additively onto the original gradient: where is the refined gradient used for the subsequent adversarial update and controls the amplification strength. Since the reference is updated with the current gradient before computing the deviation, the deviation term is implicitly scaled by : which provides natural regularization of the correction magnitude, preventing aggressive refinement in early iterations before the reference has stabilized. Mechanism Analysis. Substituting the deviation definition, the refined gradient is equivalently expressed as: The current gradient is upscaled by while the historical trend is subtracted with weight : components aligned with the reference are partially cancelled, while deviating components are selectively amplified. In flat regions of the loss landscape, the gradient evolves slowly and the reference closely tracks the current gradient, resulting in a small deviation; consequently, RGR applies only a mild correction and the update approaches standard gradient descent. In contrast, near surrogate-specific sharp regions, the gradient direction changes abruptly while the EMA reference retains inertia from historical gradients and cannot follow the sudden shift, producing a large deviation; RGR thus applies a strong corrective push, steering the perturbation away from the sharpest directions. This self-adaptive behavior scales the correction strength according to local loss geometry without requiring manual tuning. The refined gradient is used in place of the raw gradient before the subsequent momentum accumulation and adversarial update steps. RGR preserves the original gradient direction while selectively amplifying components that deviate from the EMA-based reference, with the correction strength scaling automatically according to local loss geometry. This self-adaptive behavior complements momentum rather than conflicting with it. Momentum stabilizes optimization in flat regions, while RGR counteracts the inertia of momentum in sharp regions by amplifying the abrupt gradient shift that momentum would otherwise resist, jointly accelerating convergence toward flatter loss regions where adversarial perturbations remain stably effective across different models. Fine-grained Feature Mixup Perturbation and Reference-based Gradient Refinement (FMGR) Algorithm FMGR The overall FMGR attack pipeline is illustrated in Algorithm 1. At the start of the attack, a single forward pass over the clean input batch is conducted to cache clean feature maps at each selected layer. During each subsequent iteration, the forward pass stochastically selects a subset of layers for feature manipulation. For each selected layer, AMP first applies block-wise perturbations to the feature map, with each block’s perturbation magnitude adaptively scaled by its local feature activation. FFM then shuffles the spatial blocks of the cached clean features and mixes them with the AMP-perturbed features at randomly sampled channel-wise ratios, after which the mixed blocks are reassembled into a complete feature map and passed forward through the network. Following the forward pass, the adversarial loss is computed and backpropagated in a single pass, jointly updating both the adversarial example and the learnable perturbations of the selected layers. The resulting input gradient is subsequently corrected by RGR, which amplifies components that deviate from the accumulated historical gradient direction while suppressing those aligned with the surrogate model, thereby guiding the optimization toward flatter and more transferable loss regions. The corrected gradient is then accumulated with momentum and applied to update the adversarial example, which is projected back onto the feasible perturbation set. After a fixed number of iterations, the final adversarial example is returned. To further enhance attack performance, we introduce an ensemble variant, FMGR-E, which incorporates multiple surrogate models into the FMGR framework. Specifically, two surrogate models are selected, and FMGR is applied to each independently during the forward pass. The adversarial loss is computed as the aggregate of losses from all surrogate models, and the resulting gradients are jointly backpropagated to update the adversarial example. By diversifying the optimization landscape across multiple model geometries, FMGR-E reduces overfitting to any single surrogate and further improves the transferability of the generated adversarial examples. Considering computational overhead, the number of surrogate models is set to 2 in all experiments. Experiments Experimental settings Datasets Following previous works, we conduct experiments on the widely used ImageNet-compatible dataset from the NIPS 2017 adversarial attack challenge. This dataset comprises 1,000 images of size 299 299, each annotated with both true and target class labels for targeted attack evaluation. We additionally employed the CIFAR-10 dataset to evaluate targeted attacks against defensive models designed to resist transfer-based attacks, randomly sampling 1,000 test images (100 per class) with target classes selected uniformly at random from incorrect labels. General settings We configure our experiments in alignment with recent research practices. The perturbation budget is constrained under -norm with . We specify for the decay parameter and for the step size. The iteration number is set to for targeted iterative attacks. All evaluated methods utilize the simple logit loss function as the optimization criterion for adversarial example generation. Models We evaluate our method on multiple pretrained DNNs, including CNN-based and transformer-based architectures. For CNN-based architectures, we employ VGG-1643, ResNet-18 (RN-18)44, ResNet-50 (RN-50)44, DenseNet-121 (DN-121)45, Xception (Xcep)46, MobileNet-v2 (MB-v2)47, EfficientNet-B0 (EF-B0)48, Inception ResNet-v2 (IR-v2), Inception-v3 (Inc-v3)49, and Inception-v4 (Inc-v4)50. We also included an adversarially trained RN-50 (adv-RN-50)51 with small -norm perturbations () as a source model. For Transformer-based architectures, we utilize Vision Transformer (ViT)52, LeViT53, ConViT54, Twins55, and Pooling-based ViT (PiT)56. For CIFAR-10 dataset, we constructed ensemble models comprising three ResNet-2044 networks (ens3-RN-20) trained under four defensive settings: standard training, ADP57, GAL58, and DVERGE59. Baseline To comprehensively evaluate the proposed FMGR, we benchmark it against established adversarial attack methods: DI, RDI, MI, TI, SI, VT, Admix, ODI, SU, CFM, and FTM. CFM serves as our primary reference point. Because the momentum and translation invariance (MI and TI) mechanisms are universally integrated across all tested methods, we omit the “MI-TI” prefix throughout our reporting for brevity. RDI acts as the foundational baseline in the majority of our experimental setups. Regarding hyperparameter configurations, we strictly align DI, RDI, TI, SI, Admix, and ODI with the experimental setups outlined in CFM. Specifically for Admix, we adopt a mixing weight of and utilize 3 mixed images. To manage the substantial computational overhead typically associated with Admix, we limit its scale image count to 1 unless explicitly stated otherwise. We also opt to report the performance of standalone ODI rather than the combined RDI-ODI variant, given that prior literature demonstrates nearly identical efficacy between the two. For the implementations of SU, CFM, and FTM, we reproduced these methods under identical experimental settings and found the differences from the originally reported results to be within ±0.3%. To ensure the consistency and authority of the baseline numbers, we directly report the results from their respective original publications. Settings for our FMGR In our FMGR method, we introduce specific configurations to optimize the performance of our novel modules. For the Fine-grained Feature Mixup (FFM) and Adaptive Momentum Perturbation (AMP) modules, we set the grid scale to partition the feature maps into spatial blocks. We set the layer update probability and sample the channel-wise mixing ratio from a uniform distribution with . The scaling factor for the magnitude of the disturbance is set to 0.01. Following CFM, which demonstrates that perturbing shallow layers encoding low-level features introduces excessive noise that degrades transferability, we restrict feature perturbation to layers whose feature map size is no larger than 1/16 of the input image resolution. For pure Vision Transformer architectures, we apply perturbations to the outputs of all fully-connected layers. For the Reference-based Gradient Refinement (RGR) module, we configure the parameters to optimally steer the optimization trajectory away from surrogate-specific minima by setting the exponential moving average smoothing coefficient and the deviation amplification factor . For the ensemble variant of our method (FMGR-E), we adopt RN-50 and Inc-v3 as the two surrogate models, which are identical to those used in RDI-FTM-E, and aggregate the per-model losses before a single joint backward pass, ensuring a fair comparison in terms of surrogate model identity, ensemble size, and aggregation protocol. Experimental results Transfer success rates Table 1 reports the targeted attack success rates against ten CNN-based models. Our proposed methods, RDI-FMGR and RDI-FMGR-E (ensemble of RN-50 and Inc-v3), consistently outperform competing baselines across diverse architectures. With RN-50 as the source model, RDI-FMGR and RDI-FMGR-E attain average success rates of 77.9% and 80.9%, respectively. Notably, RDI-FMGR-E surpasses the prior state-of-the-art RDI-FTM-E by 1.4% on average, while achieving more pronounced gains on harder targets, including a 3.5% improvement on Xcep and 3.8% on IR-v2. When Inc-v3 is adopted as the source model, RDI-FMGR-E further extends its advantage, exceeding RDI-FTM-E by 2.8% in average success rate. The gains are particularly striking on MB-v2 (+2.5%) and IR-v2 (+5.0%), indicating that our method better captures transferable features even under challenging cross-model conditions. These consistent improvements across heterogeneous CNN architectures confirm the effectiveness of FMGR in boosting targeted transferability. Table 1Targeted attack success rates (%) against ten CNN models. The best results are shown in bold, and the second best are underlined. Time (s) denotes the average time required to attack a single image.SourceAttackTime (s)VGG-16RN-18RN-50DN-121XcepMB-v2EF-B0IR-v2Inc-v3Inc-v4Avg.RN-50DI0.9663.256.498.974.75.527.829.05.69.09.437.9RDI0.8666.370.798.281.913.246.447.217.129.923.349.4RDI-SI4.1770.579.898.888.929.556.266.237.956.443.662.8RDI-VT5.0168.878.798.282.527.954.556.132.845.837.958.3RDI-Admix2.4974.280.798.786.820.959.456.126.742.734.158.0RDI-SU1.5567.974.798.383.324.956.350.220.932.126.253.5ODI3.0778.377.197.687.043.867.370.049.565.955.469.2RDI-CFM0.9784.788.498.490.351.181.578.848.065.559.374.6RDI-FTM1.0786.387.597.989.756.183.381.254.770.866.677.4RDI-FTM-E2.0688.188.698.391.459.485.484.356.973.469.379.5RDI-FMGR1.5384.486.197.089.559.682.481.657.173.068.477.9RDI-FMGR-E2.9888.789.397.391.562.985.984.060.775.672.980.9Inc-v3DI0.852.92.43.45.01.91.83.73.099.24.212.8RDI0.743.53.84.07.03.13.05.96.398.77.114.2RDI-SI3.434.05.25.711.06.34.68.211.698.812.116.8RDI-VT4.445.98.99.413.27.45.99.812.398.714.718.6RDI-Admix2.086.36.58.812.86.06.110.912.298.713.618.2RDI-SU1.007.811.011.412.79.48.26.38.398.710.918.5ODI2.7114.314.916.732.320.313.725.326.495.931.629.1RDI-CFM0.9022.926.826.239.134.127.138.636.295.944.839.2RDI-FTM0.9825.830.932.141.736.330.741.442.994.148.142.4RDI-FTM-E1.8734.239.340.654.447.037.151.652.496.659.251.2RDI-FMGR1.5024.729.228.938.835.026.339.940.488.745.539.7RDI-FFM-AMP1.4926.931.531.644.538.427.844.445.192.550.343.3RDI-FMGR-E2.9137.443.644.256.649.739.654.757.495.361.254.0 Targeted attack success rates (%) against ten CNN models. The best results are shown in bold, and the second best are underlined. Time (s) denotes the average time required to attack a single image. It is worth noting that when Inc-v3 serves as the source model, the single-model variant RDI-FMGR is slightly outperformed by RDI-FFM-AMP, which does not incorporate RGR. This is because the factorized convolutions and auxiliary classifiers of Inc-v3 produce relatively smooth gradient trajectories, resulting in small deviations between instantaneous and reference gradients; consequently, the corrective signal provided by RGR is limited and may introduce mild noise. In contrast, when RN-50 is used as the source model, its residual connections give rise to sharper gradient geometry, where RGR consistently yields improvements. Moreover, the ensemble variant RDI-FMGR-E still substantially outperforms all competing methods under the Inc-v3 setting, confirming the overall robustness of the FMGR framework. Table 2 evaluates targeted attack success rates against one adversarially-trained model and five Transformer-based models, where cross-paradigm transfer poses an inherently greater challenge. Using RN-50 as the source, RDI-FMGR-E achieves average success rates of 41.4%, outperforming RDI-FTM-E by 3.3%. Considerable gains are observed on individual targets, with improvements of 6.2% on Twins and 5.7% on PiT. When attacking from Inc-v3, RDI-FMGR-E continues to lead all competitors, reaching an average of 18.0% and delivering a 9.2% improvement on LeViT over RDI-FTM-E. Across all five Transformer targets, the ensemble variant consistently achieves the best performance, demonstrating that FMGR effectively improves the transferability of adversarial examples across diverse model architectures. Table 2Targeted attack success rates (%) against one adversarially-trained model and five transformer-based models. The best results are shown in bold, and the second best are underlined. Time (s) denotes the average time required to attack a single image.SourceAttackTime (s)adv-RN-50ViTLeViTConViTTwinsPiTAvg.RN-50DI0.9610.90.23.50.41.51.73.0RDI0.8634.80.713.21.76.17.010.6RDI-SI4.1759.92.929.46.315.517.922.0RDI-VT5.0164.22.928.15.215.014.021.6RDI-Admix2.4952.41.322.52.58.58.415.9RDI-SU1.5554.80.816.92.36.97.814.9ODI3.0764.75.137.010.720.129.127.8RDI-CFM0.9775.54.346.18.925.224.730.8RDI-FTM1.0777.45.952.910.832.431.535.2RDI-FTM-E2.0679.56.858.613.635.234.938.1RDI-FMGR1.5379.36.758.214.336.038.538.8RDI-FMGR-E2.9880.97.963.714.041.440.641.4Inc-v3DI0.850.20.10.30.00.00.10.1RDI0.740.80.21.80.20.40.70.7RDI-SI3.432.00.34.10.90.73.21.9RDI-VT4.443.20.45.20.81.61.82.2RDI-Admix2.082.00.14.10.61.41.41.6RDI-SU1.000.90.22.00.31.31.01.0ODI2.716.50.812.41.73.56.75.3RDI-CFM0.908.62.121.93.26.111.68.9RDI-FTM0.989.02.425.04.510.015.311.0RDI-FTM-E1.8711.03.832.76.812.720.414.6RDI-FMGR1.508.63.327.84.510.015.411.6RDI-FFM-AMP1.499.63.131.74.412.416.312.9RDI-FMGR-E2.9111.64.341.97.118.624.518.0 Targeted attack success rates (%) against one adversarially-trained model and five transformer-based models. The best results are shown in bold, and the second best are underlined. Time (s) denotes the average time required to attack a single image. Figure 3 depicts the average attack success rates of various methods as a function of iteration count across two transfer settings: RN-50 Xcep and Inc-v3 LeViT. In the RN-50 Xcep setting, RDI-FTM and RDI-FMGR both maintain consistent upward trends throughout the optimization process, with RDI-FMGR ultimately reaching the highest success rate of 59.6% at 300 iterations. Notably, while ODI and RDI-CFM remain competitive during the early optimization stages, they are progressively overtaken as iterations accumulate. In the Inc-v3 LeViT setting, RDI-FMGR exhibits a clear and sustained advantage over all baseline methods across the full iteration range, whereas RDI-SI, RDI-VT, and RDI-Admix consistently plateau at markedly low levels, revealing their limited capacity for cross-architecture transfer. These findings collectively indicate that RDI-FMGR delivers robust and superior transferability regardless of the iteration budget. Targeted attack success rates (%) based on the number of iterations. The source model is RN-50 and Inc-v3, respectively. To further evaluate the generalizability of FMGR, we conduct transfer-based targeted attacks on the CIFAR-10 dataset against four ensemble variants of ResNet-20 trained to be robust against transfer-based attacks. As reported in Table 3, RDI-FMGR achieves the highest average success rate of 80.4%, outperforming all competing methods across all four ensemble models. Notably, the success rate on GAL improves from 13.5% to 60.8%, confirming the superior transferability of FMGR against ensemble-based defenses. Table 3Targeted attack success rates (%) against four ensemble models on the CIFAR-10 dataset. The best result is highlighted in bold. The source model is RN-50.Source: RN-50ens3-RN-20AttackBaselineADPGALDVERGEAvg.DI77.956.514.315.641.1RDI76.355.813.514.940.1RDI-SI81.265.520.022.447.3RDI-VT94.482.324.331.358.1RDI-Admix85.266.417.318.446.8RDI-CFM98.897.254.959.377.6RDI-FTM98.996.959.963.879.9RDI-FMGR99.197.160.864.680.4 Targeted attack success rates (%) against four ensemble models on the CIFAR-10 dataset. The best result is highlighted in bold. The source model is RN-50. Taken together, these results confirm that our FMGR-based methods substantially enhance the transferability of adversarial examples across both CNN and Transformer architectures, highlighting the generalizability and practical robustness of the proposed approach. Computational cost We assessed the computational overhead of each method by recording the average time needed to craft adversarial perturbations for a single image. We use an NVIDIA 3090 GPU to compute the time per image. As shown in Table 1 and Table 2, when RN-50 serves as the source model, RDI-FMGR requires 1.53 seconds per image, representing a 43.0% increase relative to the baseline RDI-FTM (1.07s). Meanwhile, RDI-FMGR-E (2.98s) achieves the highest average attack success rate among all compared methods while remaining competitive in runtime. Compared to computationally intensive baselines such as RDI-SI (4.17s) and RDI-VT (5.01s), our method delivers substantially superior attack performance at a considerably lower time cost. Although FMGR introduces additional gradient refinement operations, the overall computational overhead remains well-controlled, striking a favorable balance between transferability and efficiency. To examine performance under a matched computational budget, we normalize all methods to a fixed wall-clock cost of 0.97 seconds per image (equivalent to RDI-CFM) by scaling iteration counts proportionally, based on the convergence curves in Fig. 3. Under this budget, RDI-FMGR and RDI-FTM complete approximately 190 and 272 iterations respectively, both achieving competitive success rates on the RN-50Xcep and Inc-v3LeViT transfer settings: RDI-FMGR performs slightly below RDI-FTM and on par with RDI-CFM (300 iterations), while substantially outperforming methods with higher per-iteration cost, including ODI (95 iterations), RDI-SI (70 iterations), RDI-VT (58 iterations), and RDI-Admix (117 iterations). These findings indicate that our method meaningfully improves adversarial transferability without incurring prohibitive computational expense, supporting its viability in practical attack scenarios. Failure case analysis Figure 4 presents attention heatmap visualizations comparing clean images and adversarial examples generated by FMGR. In successful attacks (columns 1–2), the model’s attention shifts markedly from the original object toward semantically distinct areas, indicating that FMGR effectively redirects the model’s decision toward the target class. In failed cases (columns 3–4), the attention map remains concentrated on the original object. This is primarily because the original objects in these images occupy large and highly salient regions, whose strong feature activations cannot be sufficiently suppressed within the constrained perturbation budget , leaving the model’s prediction unchanged despite the applied perturbation. Attention heatmap visualizations of clean images (top row) and adversarial examples generated by FMGR (bottom row). Each column shares the same input image. Green and red texts indicate successful and failed attacks, respectively. The source model is RN-50. Ablation study Ablation for different parts of FMGR. Table 4 presents the ablation results of FMGR components using RN-50 as the source model. The baseline achieves a low average success rate with poor transferability to transformer-based models. Introducing FFM yields substantial improvement, with the largest gains on Transformer targets, confirming that breaking the fixed positional correspondence between adversarial feature blocks and their clean references effectively increases the diversity of mixed feature representations and prevents the adversarial perturbation from overfitting to surrogate-specific spatial patterns. Adding AMP further benefits all architectures, with gains concentrated on harder targets, confirming that upgrading feature perturbations from random clean feature mixing to jointly learned perturbations that are synchronously optimized during the attack process provides targeted directional guidance that FFM’s stochastic mixing alone cannot supply, compelling the adversarial example to remain robust against the most disruptive feature perturbations. Incorporating RGR achieves the best overall performance, with notable gains on PiT and LeViT, validating that amplifying gradient components that deviate from the EMA-based reference effectively suppresses surrogate-dominant gradient directions and steers optimization toward flatter, more transferable loss regions. The three components address complementary aspects of the optimization: FFM employs block-wise feature mixing to increase feature diversity and prevent overfitting, AMP supplements stochastic feature mixing with targeted learnable perturbations to maximally disrupt features toward the target class, and RGR adaptively suppresses surrogate-specific sharp gradient directions and steers optimization away from these sharp directions toward flatter, cross-model shared loss regions, thereby enhancing transferability, and their combination yields consistently superior performance across both CNN and Transformer architectures, collectively substantiating the design rationality of the full FMGR framework. Table 4Targeted attack success rates (%) of ten models by ablating different parts of FMGR. The source model is RN-50. The best result is highlighted in bold.AblationTarget modelAvgFFMAMPRGRRN-18RN-50VGG-16Inc-v3EF-B0DN-121XcepViTLeVitPiT70.798.266.329.947.281.913.20.713.27.042.887.698.285.464.977.590.851.44.748.326.863.686.197.584.071.679.889.457.96.556.334.966.486.197.084.473.081.689.559.66.758.238.567.5 Targeted attack success rates (%) of ten models by ablating different parts of FMGR. The source model is RN-50. The best result is highlighted in bold. Stability with respect to grid scale N and shuffle type, mixing ratio upper bound , and layer update probability p. For the ablation of FFM component, we investigate the effect of grid scale N and shuffle type on attack transferability. As shown in Fig. 5 (right), achieves the highest attack success rate of 62.2%, demonstrating that block partitioning strikes the optimal balance between spatial diversity and semantically coherence. Performance degrades monotonically as N increases beyond 2, indicating that excessively fine partitioning disrupts semantically coherent regions. As shown in Fig. 6 (left), spatial shuffling consistently outperforms the no-shuffle variant across all target models. For instance, the success rate on Inc-v3 improves by 3.4% when spatial shuffling is applied, confirming that disrupting the spatial correspondence of clean reference features effectively prevents adversarial perturbations from exploiting fixed feature patterns. As shown in Fig. 6 (middle), achieves the highest attack success rate of 62.2%, demonstrating that an intermediate mixing ratio strikes the optimal balance between preserving adversarial feature information and introducing sufficient clean feature diversity; performance degrades at both extremes, as too small a ratio provides insufficient regularization while too large a ratio suppresses the adversarial signal. As shown in Fig. 6 (right), yields the best performance, indicating that a low update probability acts as an effective implicit regularizer by preventing the learned perturbations from overfitting to the surrogate model, whereas performance drops sharply as p increases beyond 0.10, confirming that excessively frequent layer updates introduce redundant perturbations that degrade transferability. Left: Attack success rate across smoothing coefficient and deviation amplification factor Right: Attack success rate across grid scale N. The source model is RN-50. Left: Attack success rate comparison between spatial shuffling and no shuffling across target models. Middle: Attack success rate across mixing ratio upper bound . Right: Attack success rate across layer update probability p. The source model is RN-50. Analysis of smoothing coefficient and deviation amplification factor . For the ablation of RGR component, we investigate the effect of smoothing coefficient and deviation amplification factor on attack success rate. As shown in Fig. 5 (left), the configuration of and achieves the highest success rate of 62.2%. For instance, fixing and increasing from 0.8 to 1.0 leads to a drop of 1.2%, as an overly large smoothing coefficient causes the reference gradient to over-accumulate historical directional information, weakening the effectiveness of the deviation signal. Conclusion This paper proposes Fine-grained Feature Mixup Perturbation and Reference-based Gradient Refinement (FMGR) to boost the transferability of targeted adversarial attacks via three complementary components. FFM partitions feature maps into spatial blocks and mixes each block with spatially shuffled clean features, introducing stochastic perturbations in the feature space that increase representation diversity and prevent overfitting to surrogate-specific spatial patterns. AMP applies block-wise adaptive directed perturbations optimized toward the target class; the resulting adversarial interplay between the directed perturbations of AMP and the stochastic disruptions of FFM forms a complementary dynamic that jointly drives the generated adversarial examples toward stronger transferability. RGR selectively amplifies gradient deviations from a smoothed reference, steering optimization toward flatter and more transferable loss regions with negligible overhead. Experiments on ImageNet-compatible and CIFAR-10 datasets against both CNN and Vision Transformer architectures demonstrate that FMGR consistently outperforms existing baselines while maintaining low computational cost, and ablation studies validate the effectiveness and synergy of each component. Despite these advances, transfer-based attacks remain challenging when the target model’s internal representations diverge substantially from the surrogate’s, and improving robustness under such conditions remains an open problem. Future work will explore extending FMGR to vision-language models (VLMs) and real-world black-box API settings, where transferability and practical deployability pose new and broader challenges. Author contributions X.G. conceived and designed the overall FMGR framework, proposed the FFM, AMP, and RGR modules, implemented the methodology, conducted all experiments, analyzed the results, and wrote the main manuscript text. S.H. assisted with the experimental evaluation and data analysis. Y.Z. assisted with the literature review and manuscript revision. Z.Z. supervised the research, provided critical guidance on the methodology, and reviewed and revised the manuscript. All authors reviewed and approved the final manuscript. Funding The authors declare that no funding was received for this work. Data availability The datasets used in this study are both open-source. The NIPS 2017 ImageNet-compatible dataset is available at https://www.kaggle.com/c/nips-2017-targeted-adversarial-attack/data. The CIFAR-10 dataset is available at https://www.cs.toronto.edu/~kriz/cifar.html. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used in this study are both open-source. The NIPS 2017 ImageNet-compatible dataset is available at https://www.kaggle.com/c/nips-2017-targeted-adversarial-attack/data. The CIFAR-10 dataset is available at https://www.cs.toronto.edu/~kriz/cifar.html.