How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion.
Giannis Daras, Yeshwanth Cherapanamjeri *, Costantinos Daskalakis · 2024
Read How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion. by Giannis Daras, Yeshwanth Cherapanamjeri *, Costantinos Daskalakis (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02780.txt.
Description
Abstract The quality of generative models depends on the quality of the data they are trained on. Creating large-scale, high-quality datasets is often expensive and sometimes impossible, e.g. in certain scientific applications where there is no access to clean data due to physical or instrumentation constraints.
Catalog record
| CREATOR | Giannis Daras, Yeshwanth Cherapanamjeri *, Costantinos Daskalakis |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion. Giannis Daras, Yeshwanth Cherapanamjeri *, Costantinos Daskalakis Abstract The quality of generative models depends on the quality of the data they are trained on. Creating large-scale, high-quality datasets is often expensive and sometimes impossible, e.g. in certain scientific applications where there is no access to clean data due to physical or instrumentation constraints. Ambient Diffusion and related frameworks train diffusion models with solely corrupted data (which are usually cheaper to acquire) but ambient models significantly underperform models trained on clean data. We study this phenomenon at scale by training more than 80 80 models on data with different corruption levels across three datasets ranging from 30,000 30,000 to ≈ 1.3 \approx 1.3 M samples. We show that it is impossible, at these sample sizes, to match the performance of models trained on clean data when only training on noisy data. Yet, a combination of a small set of clean data (e.g. 10 % 10\% of the total dataset) and a large set of highly noisy data suffices to reach the performance of models trained solely on similar-size datasets of clean data, and in particular to achieve near state-of-the-art performance. We provide theoretical evidence for our findings by developing novel sample complexity bounds for learning from Gaussian Mixtures with heterogeneous variances. Our theoretical model suggests that, for large enough datasets, the effective marginal utility of a noisy sample is exponentially worse that of a clean sample. Providing a small set of clean samples can significantly reduce the sample size requirements for noisy data, as we also observe in our experiments. 1 Introduction A key factor behind the remarkable success of modern generative models, from image Diffusion Models (DMs) to Large Language Models (LLMs), is the curation of large scale datasets ( Gadre et al., 2023 ; Li et al., 2024 ) . However, in certain applications, access to high-quality data is scarce, expensive, or impossible. For example, in Magnetic Resonance Imaging (MRI) the quality of the data is proportional to the time spent in the scanner ( Jalal et al., 2021 ) and, in black-hole imaging, it is never possible to get full measurements from the object of interest ( Lin et al., 2024 ) . Constructing a (copyright-free) large-scale dataset of high-quality general domain images is also an expensive and complex process. Enterprise text-to-image image DMs rely on proprietary datasets, often acquired from third-party vendors at significant cost ( Betker et al., 2023 ; Imagen-Team-Google et al., 2024 ) . State-of-the-art open-source DMs are typically trained by crawling a large pool of images from the Web and filtering them for quality with a pipeline that deems each sample as suitable or non-suitable for training ( Gadre et al., 2023 ) . However, this binary treatment of samples is problematic because often low-quality images still contain useful information. For example, a blurry image might get dismissed from the filtering pipeline to avoid blurry generations at inference time, yet the image might still contain important information about the world, such as the type of objects present at the scene. Recently, there has been a growing interest in developing frameworks for training generative models using corrupted data, e.g. from blurry or noisy images ( Bora et al., 2018 ; Daras et al., 2023b ; Kawar et al., 2023 ; Aali et al., 2023 ; Daras et al., 2024 ; Bai et al., 2024 ; Rozet et al., 2024 ; Wang et al., 2024 ; Kelkar et al., 2023 ) . Most frameworks that are applicable to diffusion models introduce training or sampling approximations, which significantly hurt performance. A recent work, Consistent Diffusion Meets Tweedie ( Daras et al., 2024 ) , proposes a method that guarantees exact sampling, at least asymptotically as the number of noisy samples tends to infinity. For realistic sample sizes, however, models trained on noisy data using this method (or any other proposed method) are significantly inferior to models trained on clean data. How do we overcome this challenge? Figure 1: ImageNet FID performance (lower is better) for models trained with different amounts of clean and noisy data. Performance of models trained with only clean data (red curve) reduces as we decrease the amount of data used. Training with purely noisy data (right-most points) also gives poor performance – even if 100\% of the dataset is available. Training with a mix of noisy and clean data strikes an interesting balance: a model trained with 90\% noisy and 10\% clean data is almost as good as a model trained with 100\% clean data. A fundamental limitation facing existing approaches for leveraging corrupted data is that, so far, they have been studied under the assumption that no access to clean samples is available. As we show in this work, it is theoretically and experimentally impossible to compensate for the lack of clean data without a significant hit (that is exponential in the number of classes in our theoretical analysis) in the number of noisy datapoints. On the other hand, we show that a combination of a small set of clean images (e.g. 10% of the total dataset) and a large set of noisy images (e.g. 90% of the total dataset) suffices to maintain state-of-the-art performance. Models trained on this data mixture significantly outperform two natural baselines: i) training only on the small set of clean images; and ii) training only on the large set of noisy images (see also Figure 1 and Table 1 ). We validate our finding by training more than 80 models across various noise scales and corruption ratios on ImageNet, CelebA-HQ, and CIFAR-10. Available Data CIFAR-10 CelebA ImageNet 100% clean 1.99\scriptstyle{\pm 0.02} 2.40\scriptstyle{\pm 0.07} 1.41\scriptstyle{\pm 0.03} 100% noisy 11.93\scriptstyle{\pm 0.09} 12.97\scriptstyle{\pm 0.11} 5.32\scriptstyle{\pm 0.08} 10% clean (smaller dataset) 17.30\scriptstyle{\pm 0.14} 11.92\scriptstyle{\pm 0.05} 10.57\scriptstyle{\pm 0.06} 90% noisy + 10% clean 2.81\scriptstyle{\pm 0.02} 2.75\scriptstyle{\pm 0.02} 1.68\scriptstyle{\pm 0.02} Table 1: FID scores for models trained on datasets with different percentages of noisy and clean data. Training with a mix of 90% noisy and 10% clean data achieves much better performance than training the same number of purely noisy samples, or just the clean samples. The standard deviation of the additive Gaussian noise for this experiment is \sigma=0.2 , which corresponds to SNR \approx 14.15\mathrm{dB} . Theoretically, we consider a novel variant of the classical problem of learning Gaussian Mixture Models (GMMs) with heterogeneous noise levels. In contrast to the classical setting, each sample is generated with a (potentially) different noise level reflective of how clean or noisy the data point is. We derive minimax optimal estimation rates, which showcase the differing utility of clean and noisy data. Concretely, having lots of noisy data enables identification of the right low-dimensional structure for learning the mixture components, while a more moderate number of clean data allows fine-grained differentiation of the components in low dimensions. In particular, effective learning is possible even when the number of clean samples is substantially smaller than the dimension. At the same time, relying solely on noisy data requires exponentially (in the number of components) noisy samples than clean ones. This theoretical result could justify why training on a mixture of clean and noisy data