DiffPatch: Generating Customizable Adversarial Patches using Diffusion Models
Zhixiang Wang, Xiaosen Wang, Bo Wang, Siheng Chen, Zhibo Wang, Xingjun Ma, Yu-Gang Jiang, [0.5em] Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University · 2024
Read DiffPatch: Generating Customizable Adversarial Patches using Diffusion Models by Zhixiang Wang, Xiaosen Wang, Bo Wang, Siheng Chen, Zhibo Wang, Xingjun Ma, Yu-Gang Jiang, [0.5em] Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01440.txt.
Introduction
Zhixiang Wang, Xingjun Ma, and Yu-Gang Jiang present a way to fool person detectors in a 2024 paper. The hosted text calls the method BadPatch, though the catalog lists a different name. Printed patches on shirts can hide a wearer from camera based detectors. Older patches look garish and draw the eye at once.
BadPatch starts from a normal photo chosen by the user. A diffusion model reshapes that photo into a patch of almost any outline. Careful steps keep the original scene easy to recognize. The patch still breaks detector sight as well as ugly rivals do. The team also built a data set of over a thousand shirt photos. Those images, shot in varied settings, can help test future defenses.
Catalog record
| CREATOR | Zhixiang Wang, Xiaosen Wang, Bo Wang, Siheng Chen, Zhibo Wang, Xingjun Ma, Yu-Gang Jiang, [0.5em] Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01440
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffPatch: Generating Customizable Adversarial Patches using Diffusion Models Zhixiang Wang, Xiaosen Wang, Bo Wang, Siheng Chen, Zhibo Wang, Xingjun Ma, Yu-Gang Jiang, [0.5em] Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Abstract Physical adversarial patches printed on clothing can enable individuals to evade person detectors, but most existing methods prioritize attack effectiveness over stealthiness, resulting in aesthetically unpleasing patches. While generative adversarial networks and diffusion models can produce more natural-looking patches, they often fail to balance stealthiness with attack effectiveness and lack flexibility for user customization. To address these limitations, we propose DiffPatch , a novel diffusion-based framework for generating customizable and naturalistic adversarial patches. Our approach allows users to start from a reference image (rather than random noise) and incorporates masks to create patches of various shapes, not limited to squares. To preserve the original semantics during the diffusion process, we employ Null-text inversion to map random noise samples to a single input image and generate patches through Incomplete Diffusion Optimization (IDO) . Our method achieves attack performance comparable to state-of-the-art non-naturalistic patches while maintaining a natural appearance. Using DiffPatch, we construct AdvT-shirt-1K , the first physical adversarial T-shirt dataset comprising over a thousand images captured in diverse scenarios. AdvT-shirt-1K can serve as a useful dataset for training or testing future defense methods. 1 Introduction Figure 1 : Example images from our AdvT-shirt dataset, showcasing the performance of DiffPatch in various scenarios. Object detectors powered by deep neural networks (DNNs) are integral to modern computer vision systems, enabling critical applications such as autonomous driving Chen et al. 2015 and medical diagnostics Miotto et al. 2018 . However, DNNs are highly susceptible to adversarial attacks ( Carlini and Wagner 2017 ; Xie et al. 2017 ; Eykholt et al. 2018 ; Wei et al. 2018 ) , which manipulate their outputs in ways that defy logic or compromise safety. These attacks can cause object detectors to misclassify or fail to detect critical objects. This vulnerability poses significant risks, particularly in safety-critical domains. Adversarial attacks on vision models can be broadly categorized into digital attacks ( Goodfellow et al. 2014b ; Carlini and Wagner 2017 ) and physical attacks ( Evtimov et al. 2017 ; Kurakin et al. 2018 ; Liu et al. 2022 ; Li et al. 2024 ; Wei et al. 2025 ) . Unlike digital attacks, which introduce imperceptible perturbations in the digital domain, physical attacks manipulate real-world objects or environments to deceive vision models. Common physical attack strategies include applying adversarial patches Thys et al. 2019 , altering lighting conditions Zhu et al. 2021 , or introducing distracting elements Liu et al. 2025 . While adversarial patches have shown promise in physical attacks, most existing methods ( Thys et al. 2019 ; Xu et al. 2020 ) prioritize attack effectiveness at the expense of stealthiness, resulting in patches that appear unnatural and are easily detectable. Although techniques like total variation loss ( Wu et al. 2020 ; Huang et al. 2020 ) have been employed to smooth patches, they fail to fully address this issue. Recent advances in image generation ( Goodfellow et al. 2014a ; Ho et al. 2020 ) have enabled the creation of more naturalistic patches Hu et al. 2021 ; Lin et al. 2023 , but these methods often sacrifice attack effectiveness and lack customization capabilities. For instance, recent work Chen et al. 2024 leverages image-latent mapping for adversarial sample generation, but its large perturbations in physical settings lead to poor naturalness or effectiveness. To address these limitations, we propose DiffPatch , a novel adversarial patch generation framework based on Stable Diffusion Rombach et al. 2022 . DiffPatch generates naturalistic and customizable adversarial patches from a reference image specified by the adversary. Our approach first maps the reference image into the latent space and optimizes noise latent vectors to create adversarial patches. We employ Null-text inversion Mokady et al. 2023 to ensure near-perfect reconstruction, preserving the original semantics. Additionally, we introduce an Incomplete Diffusion Optimization (IDO) strategy to balance natural appearance and attack performance, along with an IoU-Detection loss to accelerate convergence. To generate irregular-shaped patches, we use masks to replace the background and suppress gradients in non-essential areas, reducing interference during optimization. Using DiffPatch, we construct AdvT-shirt-1K , a dataset of over a thousand images featuring customizable adversarial patches in diverse physical-world scenarios. This dataset demonstrates the practicality of our approach and provides a valuable resource for future research. In summary, our key contributions are as follows: • We propose a novel diffusion-based method called DiffPatch for generating naturalistic adversarial patches based on a reference image. DiffPatch integrates Null-text inversion, Incomplete Diffusion Optimization (IDO) , masked customization, and an IoU-Detection loss to generate stylized and customizable adversarial patches. • Experimental results demonstrate that our DiffPatch achieves the highest attack success rate (ASR), significantly reducing the mean average precision (mAP) of various object detectors. Its performance rivals or even surpasses state-of-the-art methods for generating unnatural adversarial patches. • We construct AdvT-shirt-1K , a dataset of physical-world adversarial patches printed on T-shirts. It includes 1,131 annotated images featuring 9 unique adversarial designs across diverse scenarios, including indoor and outdoor environments, as well as individual and group photos, providing a valuable resource for future defense research. 2 Related Work 2.1 Physical Adversarial Attacks Early research on adversarial attacks primarily focused on digital domains, with methods like Basic Iterative Method (BIM) Kurakin et al. 2018 and Projected Gradient Descent (PGD) Madry et al. 2018 generating perturbations that remain effective even after physical transformations. To address real-world challenges, the Expectation Over Transformation (EOT) approach Athalye et al. 2018 was introduced, simulating environmental variations to enhance robustness. The Digital-to-Physical (D2P) method Jan et al. 2019 further improved attack effectiveness by modeling the transition between digital and physical domains. These advancements laid the groundwork for physical adversarial attacks, such as AdvPatch Brown et al. 2017 , which demonstrated robust adversarial patches effective across diverse conditions. Subsequent work, including AdvYOLO Thys et al. 2019 , extended these ideas to evade object detectors using clothing-based patches. Other physical attack strategies have leveraged accessories like glasses Sharif et al. 2016 , manipulated lighting Zhu et al. 2021 , and exploited shadows Zhong et al. 2022 , highlighting the expanding scope of real-world adversarial threats. 2.2 Naturalistic Adversarial Patch Drawing inspiration from the imperceptibility constraints in digital attacks, Universal Physical Camouflage Attacks (UPC) Huang et al. 2020 use the L_{\infty} norm to bound perturbations, ensuring patches retain a natural appearance. Unlike traditional iterative methods that optimize patches directly on images, the Naturalistic Adversarial Patch (NAP) Hu et al. 2021 adopts a generative approach. NAP employs GANs to optimize patches indirectly by modifying initial latent vectors, producing patches that are both natural-looking and effective. This approach has spurred the development of several GAN-based methods ( Doan et al. 2022 ; Lapid and Sipper 2023 ) . More recently, diffusion models have emerged as a powerful tool in the vision domain ( Chen et al. 2023 ; Lin et al. 2023 ; Xue et al. 2024 ) , leveraging their generative capabilities to create even more naturalistic and effective physical adversarial patches. These advancements highlight the growing potential of generative models