Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability

Zhaoyang Ma, Zhihao Wu, Wang Lu, Xin Gao, Jinghang Yue, Taolin Zhang, Lipo Wang, Youfang Lin, Jing Wang · 2025

Read Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability by Zhaoyang Ma, Zhihao Wu, Wang Lu, Xin Gao, Jinghang Yue, Taolin Zhang, Lipo Wang, Youfang Lin, Jing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01168.txt.

Introduction

Zhaoyang Ma and co-authors boost adversarial transfer across model ensembles in a 2025 paper. Examples that fool one network often fail on others. Ensemble attacks improve that transfer but face two gaps. Shared gradient directions are poorly captured, and model weights stay static.

Their method, HEAT, imports domain generalization into attack crafting. One module synthesizes consensus gradients via singular value decomposition. A second module balances coherence within models against diversity across them. Weights adapt dynamically during example generation. Experiments across data sets and settings beat existing methods. Both modules contribute to the transfer gains reported. Security implications cut both ways, as with all attack work. Defenders gain a stronger stress test for their models. The authors frame transferability as a domain problem. That reframing, they argue, opens fresh design space.

Catalog record

CREATORZhaoyang Ma, Zhihao Wu, Wang Lu, Xin Gao, Jinghang Yue, Taolin Zhang, Lipo Wang, Youfang Lin, Jing Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01168

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability Zhaoyang Ma, Zhihao Wu, Wang Lu, Xin Gao, Jinghang Yue, Taolin Zhang, Lipo Wang, Youfang Lin, Jing Wang Abstract The development of model ensemble attacks has significantly improved the transferability of adversarial examples, but this progress also poses severe threats to the security of deep neural networks. Existing methods, however, face two critical challenges: insufficient capture of shared gradient directions across models and a lack of adaptive weight allocation mechanisms. To address these issues, we propose a novel method — H armonized E nsemble for A dversarial T ransferability ( HEAT )—which introduces domain generalization into adversarial example generation for the first time. HEAT consists of two key modules: Consensus Gradient Direction Synthesizer, which uses Singular Value Decomposition to synthesize shared gradient directions; and Dual-Harmony Weight Orchestrator which dynamically balances intra-domain coherence, stabilizing gradients within individual models, and inter-domain diversity, enhancing transferability across models. Experimental results demonstrate that HEAT significantly outperforms existing methods across various datasets and settings, offering a new perspective and direction for adversarial attack research. 1 Introdution Figure 1: (a) Ens averages gradients from multiple surrogate models to determine the attack direction, but its transferability is limited due to the absence of dynamic weight adjustment. (b) HEAT synthesizes a common gradient direction and dynamically adjusts the weights of gradients from different surrogate models, significantly improving the attack transferability. Deep learning has achieved remarkable progress, with applications in autonomous driving ( Gao et al. 2024 ) , disease diagnosis ( Ma et al. 2023 ) , and intelligent surveillance ( Pazho et al. 2023 ) . However, recent deep neural networks (DNNs) are vulnerable to adversarial examples—inputs perturbed by imperceptible modifications that frequently lead to incorrect predictions ( Szegedy 2013 ) . More concerning is the transferability of adversarial examples; samples designed for one model often mislead others ( Goodfellow et al. 2014 ) . This allows attackers to carry out effective attacks without prior knowledge of the target model’s architecture or parameters, posing a serious threat to black-box DNN applications. Recent research has focused on enhancing the transferability of adversarial examples to improve the robustness of DNNs, offering deeper insights into their internal mechanisms and bolstering model defenses. Adversarial attacks are categorized into three types: input transformation, gradient enhancement, and model ensemble attacks. Among these, model ensemble attacks show superior success rates in black-box scenarios compared to single-model approaches ( Liu et al. 2022 ) . This advantage arises from the ability of ensemble methods to integrate gradient information from multiple surrogate models during adversarial example generation. By doing so, they reduce the risk of overfitting to the specific characteristics of a single model, thereby increasing the likelihood of success against diverse target models. Despite the success of ensemble attack methods in generating universal adversarial perturbations, they struggle to capture common gradient directions and lack adaptive weighting mechanisms. The non-convex nature of DNNs, high-dimensional parameter spaces, and randomness in model architectures and initializations often lead to significant divergence in the gradient directions of different models when processing the same input ( Li et al. 2018 ) . Moreover, existing methods typically aggregate gradients using fixed weights or simple averaging ( Tramèr et al. 2018 ; Lin et al. ) , lacking dynamic adjustment based on adversarial effectiveness. This underestimates the contribution of models exhibiting higher adversarial efficacy, diminishing attack effectiveness in multi-model scenarios. In this paper, we propose a novel method, H armonized E nsemble for A dversarial T ransferability ( HEAT ), integrating domain generalization into the design of model ensemble attacks. To our knowledge, this is the first work to incorporate domain generalization principles into ensemble attack design, offering a fresh approach that enhances the adaptability of adversarial examples across diverse models. Figure 1 illustrates how HEAT advances beyond conventional ensemble approaches with two specialized modules: C onsensus Gra dient D irection S ynthesizer ( C-GRADS ) and D ual- Harm ony Weight O rchestrator ( D-HARMO ). C-GRADS employs Singular Value Decomposition (SVD) to synthesize shared gradient directions across surrogate models, enabling efficient perturbations that robustly transfer to diverse black-box models. D-HARMO implements dual adaptive weighting: (1) Intra-domain coherence quantifies the stability of gradient directions within individual models to ensure adversarial perturbations effectively exploit vulnerabilities, prioritizing models with greater breadth; (2) Inter-domain divergence leverages the diversity of gradient directions across multiple models to improve the transferability of adversarial examples to unseen targets, prioritizing models with greater depth. D-HARMO act as a strategic team builder: intra-domain coherence weights ”generalists” for broad robustness, while inter-domain divergence prioritizes ”specialists” with transfer-critical features, balancing attack breadth and depth. Comprehensive evaluations on CIFAR-10, CIFAR-100 and ImageNet benchmarks demonstrating that HEAT substantially outperforms existing methods across various test settings. The main contributions are summarized as follows: • We establish the first formal connection between domain generalization and adversarial attacks, pioneering an approach that improves the adaptability of adversarial examples across diverse models and provides a novel perspective for existing research. • We propose Harmonized Ensemble for Adversarial Transferability (HEAT), which substantially enhances the transferability of adversarial examples by harmonizing gradient information across multiple models. • HEAT redefines ensemble attacks from the domain generalization perspective: C-GRADS synthesizes shared gradient directions across models, while D-HARMO integrates intra-domain coherence and inter-domain divergence, optimizing gradient information utilization. • The proposed HEAT generates adversarial examples with high transferability, significantly outperforming existing ensemble attack methods in terms of effectiveness and adaptability. Figure 2: An overview of HEAT: Components (b) and (c) collectively form the D-HARMO module. In (a), we use SVD to synthesize shared gradient directions; (b) captures synergistic contributions to ensure consistency in gradient directions across models; and (c) dynamically adjusts weights based on loss and alignment contributions among the models. The refined gradients generated by these components are integrated to produce highly effective adversarial images. 2 Related work Adversarial attacks were first formalized by Szegedy 2013 , showing that imperceptible perturbations to input data could cause significant misclassifications in DNNs. This discovery exposed critical vulnerabilities in machine learning systems, sparking extensive research into adversarial attacks. Input transformation attacks manipulate input data through geometric deformations, pixel-level perturbations, or structured noise injection, thereby misleading model predictions. Common approaches include: gradient-based perturbation ( Goodfellow et al. 2014 ; Carlini & Wagner 2017 ) , random/structural transformations ( Xie et al. 2019 ; Wang et al. 2023 ) , reinforcement learning-driven dynamic transformations ( Zhu et al. 2024 ) , and block-wise transformations ( Wang et al. 2024 ) . Gradient enhancement attacks optimize gradient computations to craft transfer-robust adversarial examples. This approach improves attack effectiveness through more precise gradient calculations. Common approaches include: update direction optimization ( Zhu et al. 2023 ) , momentum-based smoothing ( Dong et al. 2018 ) , adaptive iterative optimization ( Li et al. 2023a ) , cross-domain fusion ( Pang et al. 2024 ; Tao et al. 2024 ) , and self-enhancement mechanisms ( Wu et al. 2024 ) . Model ensemble attacks integrate gradients from multiple surrogate models to enhance cross-model transferability, addressing the overfitting limitations of single-model attacks. Some ensemble approaches include longitudinal strategy ( Li et al. 2020 ) , gradient fusion ( Liu et al. 2022 )

Read the complete text