DMin: Scalable Training Data Influence Estimation for Diffusion Models
Huawei Lin, Yingjie Lao, Weijie Zhao · 2024
Read DMin: Scalable Training Data Influence Estimation for Diffusion Models by Huawei Lin, Yingjie Lao, Weijie Zhao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08637.txt.
Introduction
Huawei Lin, Yingjie Lao, and Weijie Zhao built DMin in two thousand twenty-four, a tool to find which training images most shaped a picture made by a diffusion model. Past influence methods only worked on small or lightly tuned models, because the computing cost was huge. DMin claims to be the first that scales to models with billions of settings.
By compressing gradient data, DMin shrinks storage from hundreds of terabytes to mere megabytes. It then pulls the top matching training samples in under a second, with no loss in quality. Tests show it spots the truly influential samples well. Such tools help explain and audit image generators.
Catalog record
| CREATOR | Huawei Lin, Yingjie Lao, Weijie Zhao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DMin: Scalable Training Data Influence Estimation for Diffusion Models Huawei Lin, Yingjie Lao, Weijie Zhao Abstract Identifying the training data samples that most influence a generated image is a critical task in understanding diffusion models (DMs), yet existing influence estimation methods are constrained to small-scale or LoRA-tuned models due to computational limitations. To address this challenge, we propose DMin ( D iffusion M odel in fluence), a scalable framework for estimating the influence of each training data sample on a given generated image. To the best of our knowledge, it is the first method capable of influence estimation for DMs with billions of parameters. Leveraging efficient gradient compression, DMin reduces storage requirements from hundreds of TBs to mere MBs or even KBs, and retrieves the top- k k most influential training samples in under 1 second, all while maintaining performance. Our empirical results demonstrate that DMin is both effective in identifying influential training samples and efficient in terms of computational and storage requirements. Figure 1 : Examples of influential training samples, with prompts displayed below generated image. (SD 3 Medium with LoRA, v=2^{16} ). 1 Introduction Diffusion models have emerged as powerful generative models, capable of producing high-quality images and media across various applications [ 8 , 41 , 25 , 43 , 22 ] . Despite their impressive performance, the datasets used for training are often sourced broadly from the internet [ 38 , 40 , 39 , 31 , 23 ] . This vast dataset diversity allows diffusion models to generate an extensive range of content, enhancing their versatility and adaptability across multiple domains [ 19 , 5 ] . However, it also means that these models may inadvertently generate unexpected or even harmful content, reflecting biases or inaccuracies present in the training data. This raises an important question: given a generated image, can we estimate the influence of each training data sample on this image? Such an estimation is crucial for various applications, such as understanding potential biases [ 17 , 26 ] and improving model transparency by tracing the origins of specific generated outputs [ 16 , 7 , 13 ] . Recently, many studies have explored influence estimation in diffusion models [ 28 , 18 , 28 , 30 , 11 ] . These methods assign an influence score to each training data sample relative to a generated image, quantifying the extent to which each sample impacts the generation process. For instance, DataInf [ 18 ] and K-FAC [ 28 ] are influence approximation techniques tailored for diffusion models. However, they are both second-order methods that require the inversion of the Hessian matrix. To approximate this inversion, they must load all the gradients of training data samples across several predefined timesteps. Notably, in the case of the full-precision Stable Diffusion 3 Medium model [ 10 ] , the gradient of the entire model requires approximately 8 GB of storage. Collecting gradients for one training sample over 10 timesteps would consume 8\times 10=80 GB. Scaling this requirement to a training dataset of 10,000 samples results in a storage demand of around 800 TB – far exceeding the capacity of typical memory or even hard drives. Given that diffusion models are often trained on datasets with millions of samples, this storage demand becomes impractical. Consequently, these methods are limited to LoRA-tuned models or small diffusion models [ 15 , 36 ] . Although some prior works have applied gradient compression, such as SVD [ 13 ] and quantization [ 28 ] , the achieved compression rates are insufficient to maintain performance at this scale. Alternatively, Journey-TRAK [ 11 ] and D-TRAK [ 30 ] are first-order methods for influence estimation on diffusion models, which are extended from TRAK [ 32 ] on deep learning models. Both approaches utilize random projection to reduce the dimensionality of gradients. However, for large diffusion models, such as the full-precision Stable Diffusion 3 Medium model, the gradient dimensionality exceeds 2 billion parameters. Using the suggested projection dimension of 32,768 in D-TRAK, storing such a 2\text{B}\times 32,768 projection matrix requires more than 238 TB of storage. Even if the projection matrix is dynamically generated during computation, the scale of these operations substantially slows down the overall process. As a result, they are only feasible for small models or adapter-tuned models. Challenges. Although these approaches have demonstrated superior performance on certain diffusion models, several key challenges remain: (1) Scalability on Model Size: Existing methods either require computing second-order Hessian inversion or handling a massive projection matrix, both of which restrict their applicability to large diffusion models. (2) Scalability on Dataset Size: Diffusion models frequently rely on datasets containing millions of samples, making the computation of a Hessian inversion for the entire training dataset impractical. Additionally, storing the full gradients for all training data samples presents a significant challenge. (3) Fragility of Influence Estimation: Previous studies have demonstrated the fragility of influence estimation in extremely deep models [ 24 , 2 , 9 , 12 ] . Similarly, we observed this fragility in large diffusion models, regardless of whether they use U-Net or transformer. To address these challenges, in this paper, we propose DMin , a scalable influence estimation framework for diffusion models. Unlike existing approaches that are limited to small models or LoRA-tuned models, the proposed DMin scales effectively to larger diffusion models with billions of parameters. For each data sample, DMin first computes and collects gradients at each timestep, then compresses these gradients to MBs or KBs while maintaining performance. Following this compression, DMin can accurately estimate the influence of each training data sample on a given generated image or retrieve the top- k most influential samples on-the-fly using K-nearest neighbors (KNN) search, enabling further speedup based on the specific task. Contributions. The main contributions of this paper are: • We introduce DMin , a scalable influence estimation framework for DMs, compatible with various architectures, from small models and LoRA-tuned models to large-scale models with billions of parameters. • To overcome storage and computational limitations, DMin employs a gradient compression technique, reducing storage from around 40 GB to 80 KB per sample while maintaining accuracy, enabling feasible influence estimation on large models and datasets. • DMin utilizes KNN to retrieve the top- k most influential training samples for a generated image on-the-fly. • Our experimental results demonstrate DMin ’s effectiveness and efficiency in influence estimation. • We provide an open-source PyTorch implementation with multiprocessing support 1 1 1 We will release the GitHub link after acceptance. . Figure 2 : Overview of the proposed DMin . (a) In gradient computation, given a training data sample (a pair of prompt p^{i} and image x^{i} ) and a timestep t , the data passes through the diffusion model in the same manner as during training. After the backward pass, the gradients g^{i}_{t} at timestep t can be obtained. (b) For the full model, gradients are collected from the UNet or transformer, whereas for models with adapters, such as LoRA, gradients are collected only from the adapter. (c) For a prompt p^{s} and the corresponding generated image x^{s} , the gradients are obtained in the same way as in Gradient Computation. The influence \mathcal{I}_{\theta}(X^{s},X^{i}) is then