You Don’t Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

Kairan Zhao, Eleni Triantafillou, Peter Triantafillou · 2026

Read You Don’t Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models by Kairan Zhao, Eleni Triantafillou, Peter Triantafillou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00133.txt.

Introduction

Zhao, Triantafillou and Triantafillou introduce GUARD, a framework that curbs memorization in text-to-image diffusion models by steering generation away from training images. It adjusts denoising so outputs stay distinct from training data yet aligned with the prompt, without harming image quality. Its steering target comes from cross-attention attenuation, using a statistical test that spots which prompt positions need dampening. The result is a surgical, per-prompt inference-time method that proves most robust across two architectures for both verbatim and template memorization.

Catalog record

CREATORKairan Zhao, Eleni Triantafillou, Peter Triantafillou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70487
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00133

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

You Don’t Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models Kairan Zhao, Eleni Triantafillou, Peter Triantafillou Abstract Generative models have been shown to “memorize” certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality. 1 Introduction Researchers have recently observed that generative models can reproduce verbatim or near-verbatim copies of certain training examples ( 3 ; 4 ; 27 ) ; a phenomenon referred to as “memorization”. This is problematic for various reasons, ranging from revealing sensitive or private data to potential copyright concerns, and numerous research efforts have attempted to address the issue of memorization either preventatively or post-hoc. We study this important problem in the context of text-to-image (T2I) diffusion models. Several approaches have been proposed to address memorization, including “training time” methods that act during the original training phase, to prevent memorization from occurring in the first place ( 21 ; 30 ) , and methods operating at post hoc, “finetuning time” that aim to post-process a model to remove the memorization of certain examples (“unlearning”) ( 10 ; 17 ; 25 ; 8 ; 33 ; 1 ) . However, these methods have various drawbacks: in modern pipelines, practitioners often build on pretrained models and lack control of the original pretraining phase, making training-time interventions unrealistic. Even setting that unrealistic assumption aside, training-time interventions that can at best guess which examples will become memorized, rely on heuristics and may have unwanted side-effects such as deteriorating the model’s utility. Finetuning-based unlearning methods, on the other hand, are both computationally inefficient and often lack robustness ( 34 ; 26 ) . Given the above shortcomings of training- and finetuning- time methods, and the desiderata of computational efficiency and surgical precision of memorization reduction without compromising utility, we turn our attention to a third category of inference-time mitigation: rather than seeking a model that has no memorized information in its weights, we accept that memorization may have occurred, but devise specialized inference-time procedures that mask memorized information so that it doesn’t affect the generated content. In other words, while some memorized information may be present in the model weights, we seek a specialized forward pass at inference-time that: (i) guards against generating any memorized training example; (ii) does so in a manner that does not adversely affect the high-quality of generated images; and (iii) with minimal impact to efficiency. Our first contribution is a novel inference-time mitigation framework for T2I diffusion models called Guidance Using Attractive-Repulsive Dynamics (GUARD). We build on the classic “classifier-free guidance” formula, that sets the predicted noise for a given timestep of denoising to be a combination of the predicted noise obtained from an empty prompt (the “unconditional prediction”) and the predicted noise obtained from the given text prompt (the “text-conditional prediction”, obtained via cross-attention with the text prompt embeddings). We modify this guidance to add a negative weight to the text-conditional prediction (the “repulsion” term), and we add an additional “attraction” term with a positive weight. Effectively performing arithmetic of the predicted noise, our revised “contrastive guidance” is governed by two forces. The “repulsive” term encourages moving away from the predicted noise that the standard forward pass would have produced on the given prompt. This is a key driver of memorization mitigation, since that standard forward pass applied on a memorized prompt would steer the generation towards reconstructing the training image associated with that prompt. At the same time, the “attraction term” provides a better target to steer towards. This term not only helps with mitigating memorization but acts as a quality booster, as steering aggressively away from the memorized prompt can otherwise cause fidelity collapse, where generated images lose structural coherence or semantic relevance to the prompt. Next, we propose a concrete instantiation of the GUARD framework via a novel cross-attention adjustment method playing the role of the “positive target” that GUARD steers towards. We arrived at this method after a careful empirical analysis of the distribution of cross-attention at particular tokens, building and expanding on past work that found that specific “trigger” tokens are responsible for regurgitating training images ( 27 ; 30 ) . Compared to prior work that uses hard-coded strategies to redistribute the (cross)-attention ( 21 ) , we develop a dynamic on-the-fly and per-prompt trigger-token detection strategy to inform attention redistribution. We demonstrate that this approach improves upon the prior state of the art on its own, but even by a larger margin when integrated into GUARD as the positive target. We summarize our key contributions below. • Framework: We propose GUARD, a contrastive guidance framework for inference-time memorization mitigation that combines repulsion from memorized directions with attraction toward a safe target. • Empirical Analysis. We analyze the distribution of cross attention at tokens of memorized and non-memorized prompts, across model architectures and for verbatim and template memorization, revealing more nuanced desiderata for memorization mitigation. • Detection: Guided by our analysis, we design a per-prompt cross-attention spike detector that identifies memorization-critical positions \mathcal{S}(p) for a prompt p based on on-the-fly statistical outlier analysis. • Instantiation: We instantiate GUARD with a surgical cross-attention-logit attenuation mechanism that attenuates attention at positions in \mathcal{S}(p) . • Evaluation: We conduct a comprehensive evaluation of GUARD and its instantiations versus several prior state-of-the-art methods. The results show that our method outperforms prior work across model architecture, memorization types (verbatim vs template) and according to both memorization and quality metrics. 2 Background Diffusion Models . In image diffusion models, a “forward diffusion” process first adds Gaussian noise to an image. This happens gradually over T steps, with each step adding a predetermined amount of noise. The image x_{t} at step t of this forward process can be computed in closed form as: x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon, (1) where x_{0} is the original image (before any noise addition), and \bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{t}) , where \beta_{t}\in(0,1) is the scheduled variance at step t , and \epsilon\sim\mathcal{N}(0,I) . Then, the “reverse diffusion” process gradually removes the noise, aiming to obtain the original clean image. This happens via a trainable “noise predictor” model \epsilon_{\theta} , parameterized by \theta , that takes as input a noised image and predicts the noise that was added to it. The training objective

Read the complete text