SolidMark: Evaluating Image Memorization in Generative Models

Nicky Kriplani, Minh Pham, Gowthami Somepalli, Chinmay Hegde, Niv Cohen · 2025

Read SolidMark: Evaluating Image Memorization in Generative Models by Nicky Kriplani, Minh Pham, Gowthami Somepalli, Chinmay Hegde, Niv Cohen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00592.txt.

Introduction

Nicky Kriplani and co-authors introduce SolidMark, a per-image test for memorization in image generators, in a 2025 paper. Diffusion models can regurgitate training images, a legal and privacy worry. Existing scores for that risk carry dataset-dependent biases and cannot judge one specific image well.

The paper first maps the flaws in current metrics, then offers a method that scores memorization image by image, down to pixel-level detail. Known defenses against memorization are re-scored under the new lens. Models are released for others to use. Sharper measurement is the first step to generators that learn styles without storing pictures.

Catalog record

CREATORNicky Kriplani, Minh Pham, Gowthami Somepalli, Chinmay Hegde, Niv Cohen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SolidMark: Evaluating Image Memorization in Generative Models Nicky Kriplani, Minh Pham, Gowthami Somepalli, Chinmay Hegde, Niv Cohen Abstract Recent works have shown that diffusion models are able to memorize training images and emit them at generation time. However, the metrics used to evaluate memorization and its mitigation techniques suffer from dataset-dependent biases and struggle to detect whether a given specific image has been memorized or not. This paper begins with a comprehensive exploration of issues surrounding memorization metrics in diffusion models. Then, to mitigate these issues, we introduce SolidMark , a novel evaluation method that provides a per-image memorization score. We then re-evaluate existing memorization mitigation techniques. We also show that SolidMark is capable of evaluating fine-grained pixel-level memorization. Finally, we release a variety of models based on SolidMark to facilitate further research for understanding memorization phenomena in generative models. All of our code is available at https://github.com/NickyDCFP/SolidMark . 1 Introduction Diffusion models ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Rombach et al., 2022 ) have gained prominence because of their ability to generate remarkably photorealistic images. However, they have also been subject to scrutiny and litigation ( Saveri and Butterick, 2023 ) owing to their ability to regurgitate potentially copyrighted training images. Additionally, commonly used datasets ( Schuhmann et al., 2021 ) have been shown to contain sensitive materials such as clinical images of medical patients, whose exact recreation poses incredibly intrusive privacy concerns. As a result, recent works ( Somepalli et al., 2023a ; Somepalli et al., 2023b ; Carlini et al., 2023 ; Wen et al., 2024 ; Ren et al., 2024 ; Kumari et al., 2023b ) have looked to quantify, explain, and mitigate memorization in diffusion models. Crucially, reliable and effective quantification of memorization requires sound metrics. Although a few proposed metrics serve as powerful memorization indicators, there exist disagreements in terms of how they should be applied ( Chen et al., 2024 ) . The typical way in which a given image is declared to be memorized is if it is produced in a pixel-exact manner at inference time. However, such a generation can be challenging to induce, even if the training prompt is known, due to inherent stochasticity present in diffusion model inference. This problem is even harder in unconditional models, where there are no knobs to guide the generation towards a given target image. If such a generation is not randomly observed, the user is not provided with any strong indication on whether the model has knowledge of the image. Table 1: Use Cases of Different Metrics. Metric Type of Memorization Per-Image Evaluation Caveat SSCD Similarity Reconstructive ✗ Out-of-Distribution Datasets \bar{\ell}_{2} Distance Pixel-Level ✗ Monochromatic Images SolidMark Pixel-Level ✓ Excessive Duplication Figure 1: An overview of SolidMark . We begin by augmenting training images with random scalar keys in the form of grayscale borders. Next, we inject these keys into the model by training it on these augmented images. To query for a key, we ask the model to outpaint a training image’s border using the training caption as the text prompt. We retrieve its prediction at the key by averaging the outpainted border. Finally, we report the distance between the predicted key and the true value. Existing memorization metrics usually consist of (i) some distance measure \ell between a model generation and its training dataset 1 1 1 Other works ( Somepalli et al., 2023a ; Chen et al., 2024 ) use a similarity \sigma instead, but flipping signs makes these interchangeable, so we will use the most natural measure in each case. and (ii) some scoring function that takes in a large number of these distance values (from many generations) and outputs a scalar metric. For example, a commonly used metric for memorization is the 95th percentile (scoring function) of SSCD similarities ( Pizzi et al., 2022 ; Somepalli et al., 2023b ; Chen et al., 2024 ) , an embedding-based distance between each generation and its nearest training image. In this paper, we propose SolidMark , an approach that allows for the precise quantification of pixel-level memorization. The basic idea is simple: SolidMark augments each image with a grayscale border of random intensity (see Fig. 1 ). At evaluation time, we prompt the model to fill in only the image’s border in a task we call outpainting (as an analogy to inpainting). Since the pattern is randomized independently for each image, a correct reconstruction of the pattern’s color indicates strong memorization of the sample. The idea of using this pattern is closely related to watermarking as it is reflective of the source of an image generation, but there are also some key differences that distinguish it: (i) a watermark should be difficult to remove or forge, whereas our pattern is easily removable; (ii) a watermark only needs to be detectable, but our pattern needs to be precisely reconstructed to provide a continuous metric for quantifying memorization; (iii) the value of the key should be unrelated from the content of the image, which is not required for a watermark. SolidMark can be easily incorporated into the images during training. Additionally, since the image’s border can be easily cropped out when displaying generated images, SolidMark does not affect the core performance of the model. To encourage further exploration, we release a Stable Diffusion (SD) 2.1 model injected with SolidMark ’s patterns during pretraining. Subsequently, we re-evaluate existing memorization mitigation techniques with SolidMark . We demonstrate the method’s ability to evaluate fine-grained pixel-level memorization and its universal compatibility, testing it on five different datasets in a variety of settings. We provide in Table 1 a summary of the strengths and weaknesses of SolidMark compared to existing evaluation methods in the field. Our main contributions are the following: • An in-depth exploration of existing memorization metrics, • SolidMark , a new method for precise evaluation of pixel-level memorization, • A variety of models trained specially for evaluating memorization. 2 Background and Related Work Detecting Memorization in Diffusion Models. Many works have aimed to detect memorization in diffusion models ( Somepalli et al., 2023a ; Carlini et al., 2023 ; Kumari et al., 2023b ) . A generative model that memorizes data might be especially vulnerable to membership inference attacks, in which the goal is to determine whether an image belongs to the original training set ( Carlini et al., 2022 ; Hu and Pang, 2021 ; Wen et al., 2023 ) . One notable example of a membership inference attack is an inpainting attack from Carlini et al. (2023) , who show that a diffusion model’s performance on the inpainting task significantly increases for memorized images. Mitigating Unwanted Generations. A number of works ( Somepalli et al., 2023b ; Chen et al., 2024 ; Wen et al., 2024 ; Ren et al., 2024 ) have introduced methods to mitigate memorization in diffusion models. These methods either perturb training data to decrease memorization as the model trains or perturb inputs at test time to decrease the model’s chances of recalling memorized information. Although most mitigation techniques usually involve augmenting data with some type of noise ( Somepalli et al., 2023b ) , other works attempt to alter generation trajectories using

Read the complete text