Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimation
Michele De Vita, Vasileios Belagiannis · 2024
Read Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimation by Michele De Vita, Vasileios Belagiannis (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00205.txt.
Introduction
Michele De Vita and Vasileios Belagiannis add an uncertainty gauge to diffusion image generation. Their twenty twenty-four paper estimates doubt for each pixel while a sample is being made. Diffusion models had lacked a numeric way to judge their own output quality.
The gauge reads the spread of denoising scores under a tailored perturbation. That spread ties to the curvature of the noise distribution in theory. Tests on two image sets show the gauge can flag low-quality samples for removal. Guiding sampling by the gauge also lifts overall quality. Builders of image systems gain a self-check that can catch bad outputs before users see them.
Catalog record
| CREATOR | Michele De Vita, Vasileios Belagiannis |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimation Michele De Vita, Vasileios Belagiannis Abstract Despite the remarkable progress in generative modelling, current diffusion models lack a quantitative approach to assess image quality. To address this limitation, we propose to estimate the pixel-wise aleatoric uncertainty during the sampling phase of diffusion models and utilise the uncertainty to improve the sample generation quality. The uncertainty is computed as the variance of the denoising scores with a perturbation scheme that is specifically designed for diffusion models. We then show that the aleatoric uncertainty estimates are related to the second-order derivative of the diffusion noise distribution. We evaluate our uncertainty estimation algorithm and the uncertainty-guided sampling on the ImageNet and CIFAR-10 datasets. In our comparisons with the related work, we demonstrate promising results in filtering out low quality samples. Furthermore, we show that our guided approach leads to better sample generation in terms of FID scores. 1 Introduction Recently, diffusion models have made significant progress in producing synthetic images that appear realistic [ 10 , 44 , 21 ] . However, the quality of the generated images is not always consistent, and the models may produce artefacts or low-quality samples. Therefore, understanding and quantifying the uncertainty associated with the generated samples is crucial for ensuring the quality of the data, especially in safety-critical applications such as medical imaging [ 18 , 6 ] or autonomous driving [ 37 , 13 ] . While for established generative models, such as Generative Adversarial Networks (GANs) [ 15 ] and Variational auto-encoders (VAEs) [ 29 ] , there are already a few approaches to obtain uncertainty estimates [ 42 , 39 , 5 ] , diffusion models remain mostly unexplored. Although it is possible to rely on common uncertainty estimation methods, such as Monte Carlo dropout [ 14 ] or ensemble methods [ 32 ] , these approaches are computationally expensive and not easily applicable to diffusion models. For instance, MC Dropout requires a diffusion model to be trained with dropout, which is quite uncommon and sampling needs to be performed several times. Furthermore, ensemble methods require multiple models to be trained and it is pretty expensive in terms of time budget and computational resources. The only method to estimate pixel-wise predictive uncertainty for diffusion models is the recently proposed BayesDiff [ 30 ] . Building on the limitations of the aforementioned uncertainty estimation methods, BayesDiff provides an efficient ad-hoc formulation to estimate uncertainty for image generations based on the Last Layer Laplace Approximation (LLLA) [ 8 ] . However, BayesDiff still requires a significant amount of Number of Function Evaluations (NFEs) and does not leverage uncertainty to steer the sampling process. Unlike BayesDiff, we present an approach that is not only computationally more efficient, but more importantly makes use of the uncertainty to guide the generation process towards regions of better sample quality, as illustrated in Fig. 1 . Figure 1 : Visual Results I. We provide qualitative samples of uncertainty guidance applied to Stable Diffusion 3 [ 11 ] (first two columns) and 1.5 [ 41 ] (last two columns). In the upper row we present images produced without the uncertainty guidance while the bottom row features images generated with the uncertainty guidance. We can observe that the images with uncertainty guidance present fewer artefacts and more faithful generation We propose a training-free and computationally efficient approach to estimate the aleatoric pixel-wise uncertainty during the sampling phase of diffusion models. Our method 1 1 1 Our code is available at https://github.com/Michedev/diffusion-uncertainty . estimates the uncertainty as the sensitivity [ 35 ] of multiple data points with the same denoising process. Then, we theoretically show that the proposed uncertainty measure is connected to the second derivative of the noising distribution, providing a solid understanding for our approach. Given our uncertainty estimates, pixels with high second-order derivatives are more susceptible to changes during sampling, representing features or details that are more challenging for the model to reconstruct consistently. By directing the sampling process towards high-uncertainty regions, we achieve superior image quality from the same initial conditions. Note that our approach is designed to measure data uncertainty, and thus provides aleatoric pixel-wise uncertainty estimates. We show the effectiveness of our approach by filtering out low-quality samples in ImageNet [ 9 ] and CIFAR-10 [ 31 ] datasets. Our approach outperforms existing uncertainty estimation methods in terms of both sample quality and function evaluations on ImageNet. In addition, we demonstrate the generalisation capabilities of our approach by evaluating it on different samplers and neural network architectures. Overall, our contributions are summarised as: • We propose a training-free, pixel-wise uncertainty estimation approach for diffusion models. During each sampling step, our algorithm estimates the uncertainty as the variance of multiple generated samples with the same denoising process. • We show that the uncertainty estimates gives second-order information about the noising distribution. Given this fact, we present an algorithm to guide the sampling phase based on the per-pixel uncertainty estimates. • Our experiments demonstrate state-of-the-art performance compared to previous work on ImageNet and CIFAR-10. Also, we show that our method improves the quality of generated samples by guiding the diffusion model to focus on areas with low uncertainty. 2 Related Work We discuss below the related work on uncertainty estimation, focusing on generative and diffusion models. 2.1 Traditional Uncertainty Estimation Methods Variational Bayesian Neural Networks (BNNs) [ 5 ] have been developed to approximate posterior distributions over weights by providing better-calibrated uncertainties and improving the model generalisation, as shown by Wilson et al . [ 51 ] . However, BNNs can be difficult to train compared to standard neural networks due to optimisation challenges and computational cost. For these reasons, recent approaches have aimed to approximate BNNs more efficiently [ 22 , 36 , 49 , 24 , 50 ] . For instance, Morales-Álvarez et al . [ 36 ] proposed modelling uncertainty in neural networks by using Gaussian process priors on the activation functions rather than on the weights. Teye et al . [ 49 ] approximate the uncertainty efficiently using Batch Normalisation [ 27 ] , which is equivalent to approximate inference in Bayesian models. Another uncertainty estimation method is Monte Carlo dropout (MC-Dropout), proposed by Gal et al. (2016) [ 14 ] , which leverages dropout at test time to obtain an approximation of a Bayesian neural network. However, MC-Dropout requires a model trained with dropout and multiple forward passes at test time. Deep ensembles, proposed by Lakshminarayanan et al. (2017), [ 32 ] provide a simpler approach by training an ensemble of neural networks with different random initialisations. At test time, the predictions are averaged to obtain the ensemble prediction and variance for uncertainty estimates. Deep ensembles have a higher computational cost due to the training of multiple models, but are easier to optimise compared to BNNs. Snapshot Ensembles, proposed by Huang et al . [ 25 ] , is a method to train an ensemble of neural network models at no additional cost compared to training a single model. The approach relies on the ability of the optimisers to escape local minima using a cyclic learning rate to save several snapshots of the models. Although the