Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation

Yuhao He, Jinyu Tian, Haiwei Wu, Jianqing Li · 2025

Read Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation by Yuhao He, Jinyu Tian, Haiwei Wu, Jianqing Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19425.txt.

Introduction

Yuhao He and co-authors build, in a 2025 paper, a shield against malicious AI inpainting, where an attacker erases or replaces parts of someone's photo to fake content. Noise-based protections fail when editing is guided by a mask, because only the masked region gets regenerated.

Their Structure Disruption Attack perturbs the queries inside the model's self-attention at the very first denoising step. Since self-attention sketches a scene's contours, scrambling it breaks the structure of anything the model tries to invent. Visual checks and broad experiments show top-tier protection that holds up under countermeasures, letting users post photos with less fear of forgery.

Catalog record

CREATORYuhao He, Jinyu Tian, Haiwei Wu, Jianqing Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19425

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation Yuhao He, Jinyu Tian, Haiwei Wu, Jianqing Li Abstract The rapid advancement of diffusion models has enhanced their image inpainting and editing capabilities but also introduced significant societal risks. Adversaries can exploit user images from social media to generate misleading or harmful content. While adversarial perturbations can disrupt inpainting, global perturbation-based methods fail in mask-guided editing tasks due to spatial constraints. To address these challenges, we propose Structure Disruption Attack (SDA) , a powerful protection framework for safeguarding sensitive image regions against inpainting-based editing. Building upon the contour-focused nature of self-attention mechanisms of diffusion models, SDA optimizes perturbations by disrupting queries in self-attention during the initial denoising step to destroy the contour generation process. This targeted interference directly disrupts the structural generation capability of diffusion models, effectively preventing them from producing coherent images. We validate our motivation through visualization techniques and extensive experiments on public datasets, demonstrating that SDA achieves state-of-the-art (SOTA) protection performance while maintaining strong robustness. 1 Introduction The rapid advancement of diffusion models has revolutionized image synthesis, facilitating the efficient generation of photorealistic and high-fidelity images [ 1 , 2 , 3 , 4 ] . In conditional generation tasks, these models can be fine-tuned on limited exemplars to capture intricate stylistic attributes [ 5 , 6 , 7 , 8 ] or reproduce specific objects and identities with remarkable precision [ 9 , 10 , 11 , 12 ] . For inpainting applications [ 13 , 14 , 15 , 16 ] , diffusion models leverage spatially constrained mask-guided synthesis, enabling targeted content generation within user-defined regions while maintaining coherence with surrounding context through text-guided conditioning. However, the rapid advancement of diffusion models has raised significant ethical and legal concerns [ 17 , 18 ] . For instance, unauthorized generation of images mimicking a specific artist’s style may constitute copyright infringement, and malicious actors can fine-tune these models to synthesize highly realistic yet fabricated portraits in diverse contexts, facilitating the creation of deepfake misinformation. Furthermore, the advancement of inpainting has lowered the technical barrier to image manipulation, potentially leading to the rampant spread of misinformation. As illustrated in the red region of Figure 1 , attackers can extract user photos from social media, employ a mask to preserve facial features, and combine them with negative prompts to generate defamatory or misleading imagery, severely violating personal reputation rights. Figure 1: Protected vs. Unprotected Image Resistance: (Top) Malicious inpainting alters contextual elements (e.g. naked and the White House) while preserving key features (e.g. human face). (Bottom) SDA-protected images demonstrate robust resistance to such editings through sensitive region encryption, effectively neutralizing unauthorized edits. To mitigate these risks, previous research has explored the use of global adversarial perturbations to disrupt the denoising process of diffusion models, demonstrating promising protection efficacy in fine-tuning-based text-to-image generation tasks [ 19 , 20 , 21 , 22 ] . However, in mask-guided image editing (e.g., inpainting), only the masked regions interact with the model, rendering global perturbations ineffective. Currently, inpainting-specific protection methods remain underexplored, often exhibiting suboptimal and unstable performance [ 23 , 24 ] . In this work, we present Structure Disruption Attack (SDA) , an innovative and efficient image protection method designed to prevent malicious editing of sensitive image region via diffusion model-based inpainting. The core motivation of SDA stems from the observation that diffusion models typically generate images through a coarse-to-fine process [ 25 ] : Early denoising timesteps establish object contours, while later timesteps progressively refine details. Therefore, we propose to disrupt contour structure of the protected contents (e.g. human faces), during the initial diffusion phase to prevent the synthesis of a complete image. Since the self-attention mechanism primarily governs structural coherence [ 26 ] (e.g., texture consistency and spatial relationships) by focusing on these contours, we implement this disruption through self-attention interference during the initial denoising step. As illustrated in Figure 1 (blue region), this disruption prevents the model from reconstructing primary object structures, ultimately leading to incomplete image generation. Extensive experiments in the last section would not only validate our design rationale but also demonstrate SDA’s remarkable performance. In summary, our contributions are as follows: • We propose structural disruption as a novel defense mechanism to prevent the misuse of diffusion-based inpainting, offering targeted protection for sensitive regions. • We reveal a phenomenon by visualization analysis that perturbing self-attention queries in early diffusion steps triggers a cascade failure: the model not only loses the ability to capture object contours but also breaks semantic alignment with text prompts, leading to complete generation collapse. • Our method achieves state-of-the-art performance in countering diffusion-model-based inpainting misuse and demonstrates effective performance across defenses and model versions. We further validate the practical effectiveness of the proposed method in real-world scenarios simulated through mask augmentation [ 24 ] . 2 Related work Adversarial examples against diffusion models The growing adoption of diffusion models in content creation has raised critical security concerns, particularly regarding unauthorized image synthesis. Researchers have developed specialized adversarial attacks that exploit these models’ noise prediction mechanisms to prevent malicious usage. AdvDM [ 20 ] optimizes perturbations by directly maximizing the diffusion loss, while Anti-Dreambooth [ 19 ] alternately maximizes this loss and minimizes the Dreambooth [ 9 ] training loss through joint perturbation-model optimization. CAAT [ 27 ] enhances this approach by fine-tuning cross-attention blocks during noise prediction maximization. Alternative protection strategies manipulate latent space encodings through encoder-targeted optimization. Mist [ 28 ] jointly optimizes noise prediction maximization and latent alignment minimization, while SDST [ 21 ] improves optimization efficiency through score distillation sampling. Adversarial examples against inpainting Unlike standard image generation tasks that process complete inputs, inpainting models operate under constrained conditions where only specific masked regions of the image are modifiable. This partial accessibility requirement fundamentally alters the threat model, as conventional global adversarial protection methods cannot function effectively when models inherently ignore unmasked areas. To address this challenge, researchers have developed localized attack strategies. Photoguard [ 23 ] proposes two localized strategies: EncoderAttack for latent space manipulation and DiffusionAttack for full-process interference. DiffusionGuard [ 24 ] introduces mask augmentation to strengthen local perturbations while attacking initial denoising steps. Advanced approaches disrupt semantic consistency through latent centroid deviation (DDD [ 29 ] ) or attention mechanism perturbation (AdvPaint [ 30 ] ). Our analysis reveals that targeted self-attention interference provides superior protection by directly preventing coherent image generation. 3 Methodology In this section, we introduce the SDA, an efficient image protection method that prevents images from being maliciously inpainted through Stable Diffusion models. Our approach operates by directly disrupting the contour structure during the initial phase of denoising to prevent the model from reconstructing primary object structures, this intervention fundamentally compromises the model’s capacity to synthesize structurally coherent imagery. Before discussing technical details, we present the threat model that governs our security analysis. Threat model We assume that adversaries could strategically mask the sensitive image region (e.g., human face) and exploit diffusion-powered inpainting through arbitrary malicious prompts to synthesize reputationally damaging content. This threat formulation aligns with real-world adversarial patterns: sensitive regions like facial areas in portraits or primary subjects in other image types are prioritized targets due to

Read the complete text