Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework

Ziyang Liu, Hongyuan Wang, Zijian Wang, Yinxi Lu, Yunzhao Zang, Zhiqiang Yan, Qianhao Ning · 2026

Read Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework by Ziyang Liu, Hongyuan Wang, Zijian Wang, Yinxi Lu, Yunzhao Zang, Zhiqiang Yan, Qianhao Ning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17772.txt.

Introduction

Ziyang Liu and six colleagues attack a weak spot in physical adversarial attacks. Such attacks often fit one stand-in model too closely and fail on other detectors. Their JMOF framework picks a strong team of stand-in models by measuring how alike they are. It then damps prediction outputs and flattens inner feature patterns, while an orthogonal gradient alignment step turns clashing gradients into a shared direction. Tests in simulation and the real world show JMOF beats top baselines and can fool detection, segmentation, and depth models at once.

Catalog record

CREATORZiyang Liu, Hongyuan Wang, Zijian Wang, Yinxi Lu, Yunzhao Zang, Zhiqiang Yan, Qianhao Ning
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73157
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework Ziyang Liu, Hongyuan Wang, Zijian Wang, Yinxi Lu, Yunzhao Zang, Zhiqiang Yan, Qianhao Ning Abstract Physical adversarial attacks often overfit single surrogate models and optimization objectives. While ensemble attacks can mitigate this, existing methods struggle with severe gradient conflicts within restricted physical texture spaces, significantly degrading cross-model transferability. To bridge this gap, this paper proposes a Joint Multi-Objective and Multi-Model Optimization Framework (JMOF) that leverages quantitative similarity analysis to select the optimal surrogate model ensemble. Within JMOF, a dual-level mechanism jointly suppresses prediction outputs and flattens intermediate feature distributions, balancing attack efficiency with deep generalization. Additionally, an Orthogonal Gradient Alignment (OGA) strategy resolves cross-model gradient conflicts, transforming mutually repulsive gradients into synergistic optimization directions. Extensive simulated and real-world experiments demonstrate that JMOF outperforms state-of-the-art baselines against diverse black-box detectors. Crucially, JMOF exhibits substantial cross-vision-task generalization, generating attacks capable of simultaneously deceiving object detection and semantic segmentation or monocular depth estimation models. This research advances the generalization limits of physical adversarial attacks, providing a robust framework for evaluating visual AI vulnerabilities in real-world deployments. I Introduction Deep Neural Networks (DNNs) have achieved unprecedented success in various computer vision tasks [ 31 , 56 , 20 ] , such as object detection, but numerous studies have also demonstrated their high vulnerability to adversarial attacks [ 46 , 7 ] . Traditional digital adversarial attacks usually deceive models by adding imperceptible perturbations to images [ 6 ] . However, digital attacks that directly manipulate input images are often impractical in real-world scenarios. In contrast, physical adversarial attacks deployed on target surfaces pose a significantly higher practical threat. Nevertheless, transferring these digital perturbations into the complex physical world to achieve robust attacks, such as 3D adversarial camouflage, remains a formidable challenge [ 11 ] . Fig. 1: The general framework of existing ensemble attack methods, alongside a performance comparison of representative methods and gradient fusion strategies. Ours-D and Ours-C are ablated variants of our method; the former employs only a detection loss function, whereas the latter utilizes the same surrogate model (YOLOv3) as FCA and DAS. The asterisk (*) denotes gradient fusion methods implemented by porting gradient strategies from digital attacks into our physical attack workflow. Research on physical adversarial attacks has long been hindered by two critical bottlenecks. The first is the domain shift from the digital space to the physical world [ 25 , 24 ] . Specifically, complex variations in the physical environment, including lighting, viewpoints, occlusions, and distances, can easily degrade the effectiveness of adversarial perturbations. In recent years, the introduction of differentiable rendering techniques [ 47 ] and Expectation Over Transformation (EoT) [ 1 ] has provided relatively mature solutions to this physical robustness challenge. However, a more fatal and deep-rooted problem lies in adversarial transferability. When deployed against black-box models with unknown structures and parameters, the attack performance of adversarial camouflage often declines precipitously [ 44 ] . As illustrated in Fig. 1 -(a), generating adversarial camouflage is essentially a dynamic optimization process: learning from surrogate models, exploiting vulnerabilities via loss functions, and updating perturbations via gradient fusion. Based on this logic, we observe that existing physical attack methods exhibit severe limitations across these three core stages when targeting black-box transferability: 1) Overfitting to a single surrogate model: Relying on a single surrogate model (e.g., the YOLO series) causes the generated camouflage to overfit specific decision boundaries, leading to rapid failure against real-world heterogeneous networks. While ensemble attacks can mitigate this by learning richer cross-model representations [ 28 ] , achieving joint optimization across diverse architectures remains highly challenging due to the restricted parameter space of physical attacks. 2) Lack of synergy between shallow result suppression and deep feature disruption: Existing methods typically employ a singular attack mode. Shallow attacks [ 39 ] efficiently suppress final outputs but overfit specific decision boundaries, whereas deep attacks [ 40 ] perturb shared semantic features for better transferability but often fall into optimization pitfalls due to the lack of rigid output supervision. Without effectively synergizing these two strategies, current attacks struggle to achieve both high efficiency and deep generalizability. 3) Gradient conflicts in multi-source fusion: When integrating multiple models and objectives, existing strategies mostly adopt equal-weight summation [ 4 ] or compromise-based fusion [ 2 , 16 ] . However, the limited texture parameters in physical attacks amplify severe conflicts and oscillations in heterogeneous gradients during optimization. Both directional cancellation and energy loss in these naive fusion methods inevitably mask the adversarial demands of robust models, leading to attack degradation or unidirectional overfitting. In summary, constrained by the methodological design relying on a single surrogate model and a single attack objective, as well as the restricted parameter space inherent in physical adversarial attacks, existing methods are effective only under specific task settings, rapidly losing their deceptive capabilities when deviating from these presets. Not to mention that when an attack attempts to cross the boundaries of visual tasks, such as transferring from detection to segmentation or depth estimation tasks, the semantic gap in gradients caused by differences in underlying optimization objectives and network architectures expands drastically, resulting in a substantial performance degradation for existing joint optimization strategies. This landscape clearly demonstrates that a massive gap still exists between the capability boundaries of current physical adversarial attacks and the real-world deployment demands involving black-box and multi-task scenarios, making the stride toward universal attacks highly challenging. To address the aforementioned limitations, this paper proposes a Joint Multi-Objective and Multi-Model Optimization Framework (JMOF). This framework is capable of generating universal physical adversarial attacks that exhibit high physical robustness and transferability across different models and tasks. The primary contributions of this work are summarized as follows: 1) We analyze the failure modes of joint multi-model adversarial optimization, and reveal a critical trade-off in gradient similarity of surrogate model: overly aligned gradients limit representation diversity, while highly divergent gradients induce severe conflicts. Based on this, we define an effective similarity regime that guides surrogate model selection. 2) We propose a dual-level multi-objective attack mechanism, which jointly suppresses prediction outputs and disrupts intermediate feature representations via targeted output suppression and feature distribution flattening, improving both attack strength and transferability. 3) Within JMOF, we introduce an Orthogonal Gradient Alignment (OGA) module to resolve gradient conflicts and enforce coherent optimization, enabling adversarial perturbations to converge toward shared vulnerable regions across heterogeneous models. 4) Extensive digital-to-physical experiments demonstrate that our method achieves superior transferability, outperforming prior SOTA methods across black-box detectors, and further exhibits strong cross-task generalization under joint optimization across diverse vision tasks such as semantic segmentation and monocular depth estimation, maintaining high detection attack performance while simultaneously disrupting these tasks. II Related Works II-A Physical Adversarial Attacks The inception of physical adversarial attacks can be traced back to the work of Kurakin et al. [ 15 ] , who first demonstrated the feasibility of printing digital adversarial examples into the real world. Subsequently, to enhance deployment flexibility, researchers proposed adversarial patch techniques. For instance, Thys et al. [ 38 ] successfully implemented physical invisibility against YOLO object detectors using printed stickers. However, this strategy of directly printing 2D planar images with adversarial perturbations as patches harbors

Read the complete text