Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability

Rohan Asthana, Vasileios Belagiannis · 2026

Read Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability by Rohan Asthana, Vasileios Belagiannis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20642.txt.

Introduction

Rohan Asthana and Vasileios Belagiannis detect and mitigate memorization in diffusion image models through the anisotropy of the log-probability. Diffusion models make high-fidelity images by iterative denoising, but they can memorize, reproducing exact copies or parts of training images. Earlier detectors mostly rely on the norm of a score difference as the indicator.

The authors use direction as well as size: how unevenly the log-probability varies across directions. Memorization is a legal and privacy risk, since a model may hand back someone's artwork or photo. A sharper detector is the first step to filtering it out. A copy that slips past the guard can be sold, shared, or used to frame the true owner of the work.

Catalog record

CREATORRohan Asthana, Vasileios Belagiannis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability Rohan Asthana, Vasileios Belagiannis Abstract Diffusion-based image generative models produce high-fidelity images through iterative denoising but remain vulnerable to memorization, where they unintentionally reproduce exact copies or parts of training images. Recent memorization detection methods are primarily based on the norm of score difference as indicators of memorization. We prove that such norm-based metrics are mainly effective under the assumption of isotropic log-probability distributions, which generally holds at high or medium noise levels. In contrast, analyzing the anisotropic regime reveals that memorized samples exhibit strong angular alignment between the guidance vector and unconditional scores in the low-noise setting. Through these insights, we develop a memorization detection metric by integrating isotropic norm and anisotropic alignment. Our detection metric can be computed directly on pure noise inputs via two conditional and unconditional forward passes, eliminating the need for costly denoising steps. Detection experiments on Stable Diffusion v1.4 and v2 show that our metric outperforms existing denoising-free detection methods while being at least approximately 5x faster than the previous best approach. Finally, we demonstrate the effectiveness of our approach by utilizing a mitigation strategy that adapts memorized prompts based on our developed metric. The code is available at https://github.com/rohanasthana/memorization-anisotropy . 1 Introduction Recent advances in diffusion models ( 9 ; 11 ; 24 ) , especially score-based models ( 33 ) , have positioned them as the dominant class of generative models, synthesizing various types of data, for instance, images ( 24 ; 26 ) , videos ( 10 ; 8 ) , graphs ( 35 ; 18 ) , and even neural network architectures ( 2 ) . Score-based diffusion models progressively corrupt data with Gaussian noise and then learn score estimates that approximate the gradients of the perturbed log-density to transform noise back into data. This enables high-quality generation in complex data domains. However, despite the tremendous success of these models, they are susceptible to memorization, where the model generates an exact or near replica of training images. This phenomenon is similar to overfitting in artificial neural networks and has important implications related to data privacy, copyright issues, bias of evaluation benchmarks, and assumptions about generalization. Hence, detection and mitigation of memorization in these high-fidelity generative models has been a growing body of research ( 31 ; 37 ; 6 ; 14 ; 13 ; 25 ) . Since the denoising process explicitly interpolates between Gaussian noise and the data manifold, the associated score estimates help to understand the underlying dynamics of diffusion models. This perspective motivates the use of score estimates as a powerful tool to characterize memorization in diffusion models. Inspired by this perspective, 37 characterized memorization in text-to-image diffusion models utilizing the norm of the difference of the score between the unconditional and conditional estimates. This characterization was supported by the observation that memorized samples exhibit stronger text-driven guidance, which guides the generation towards specific memorized images. Later, multiple works adopted this perspective and developed novel methods for detecting and mitigating memorization. For instance, 14 improved the metric from 37 by incorporating the Hessian of the log-probability. Moreover, 13 utilized Wen’s metric to deploy opposite guidance until the denoising trajectory steers away from the memorized sample. In this work, we demonstrate that such norm-based metrics are effective only when the underlying log-probability is isotropic, which is typically satisfied at high or mid noise levels. This is because the norm of the score function encodes information about the overall curvature of the log-probability. While the overall curvature is a strong signal for memorization in isotropic distributions, it fails in the anisotropic case, as the curvature is different across different directions. This issue is relevant in scenarios where the access to anisotropic diffusion regime is easier/more efficient compared to the isotropic regime. For example, in image level memorization task ( 15 ) , the aim is to detect memorized images (or parts of them) only through access to generated images. These generated images are far from the high-noise isotropic regime and are at the data manifold. Thus, utilizing a metric that works under the anisotropic regime should be preferable in this case. To overcome this issue, we additionally account for the anisotropy of the log-probability in the low-noise regime and leverage a more informative distribution that better captures memorization than isotropic metrics alone. We explore the anisotropic low-noise regime and reveal that memorization manifests as a stronger alignment between the guidance vector and the unconditional score estimate in the anisotropic low-noise regime. We utilize this phenomenon to develop a simple yet effective memorization detection metric, which is the weighted sum of a) cosine similarity between the guidance vector and the unconditional score estimate in the anisotropic regime, and b) norm of the guidance vector in the isotropic regime. Importantly, our developed metric is denoising-free (meaning that it does not require costly denoising steps) and utilizes only two conditional and unconditional forward passes, hence detecting memorization at a rapid pace. We finally integrate our detection metric into a prompt augmentation scheme to mitigate memorization effectively during inference. To evaluate the performance and efficiency of our developed metric, we follow the standard evaluation protocol ( 37 ; 14 ) and conduct experiments on Stable Diffusion v1.4 and v2.0 ( 24 ) . We show that our method outperforms existing denoising-free metrics while being at least approximately 5x faster than the previous best method. Moreover, our mitigation experiments on MemBench ( 12 ) showcase that our developed metric makes the generations highly dissimilar to the memorized training sample, while exhibiting strong text-image alignment and high aesthetic quality. Lastly, we conduct three ablation studies in the Appendix Section A.2 , comparing alternative formulations of our proposed metric, analyzing the contribution of each component of our metric, and ablating normalization and timestep choices. In summary, our contributions are as follows: • We identify that norm-based memorization detection metrics are effective only under the assumption of isotropic log-probability distributions. To rectify this, we consider both the isotropic (high-noise) and anisotropic (low-noise) regimes of the log-probability for memorization detection. • We show that in the anisotropic regime, memorization manifests as a strong alignment between the guidance vector and the unconditional score estimate. Thus, we develop our denoising-free detection metric by combining the score-based alignment in anisotropy with the norm of the score difference in isotropy. • Through our experiments, we demonstrate that our metric achieves superior denoising-free detection performance compared to existing metrics while being efficient. We further validate the performance of our developed metric by performing inference-time memorization mitigation through the benchmark MemBench ( 12 ) . 2 Related Work Memorization in Diffusion Models Detecting and mitigating memorization in diffusion models has gained substantial interest in recent years ( 31 ; 37 ; 6 ; 14 ; 13 ; 25 ) . In text-to-image diffusion models, this phenomenon was firstly identified in the works by 30 , where they show that factors such as training set size affect the scale of memorization. Concurrently, 4 demonstrated the extraction of training data (which were essentially memorized samples) from diffusion models. Subsequent work investigated dataset properties that drive memorization ( 16 ; 19 ) , while others

Read the complete text