Enhancing and Accelerating Diffusion-Based Inverse Problem Solving through Measurements Optimization

Tianyu Chen, Zhendong Wang, Mingyuan Zhou · 2024

Read Enhancing and Accelerating Diffusion-Based Inverse Problem Solving through Measurements Optimization by Tianyu Chen, Zhendong Wang, Mingyuan Zhou (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03941.txt.

Introduction

Tianyu Chen, Zhendong Wang and Mingyuan Zhou speed up diffusion-based solving of inverse problems in their 2024 arXiv paper. Diffusion models can rebuild images from partial measurements, but they usually need thousands of steps because each step uses little measurement information. Their module, Measurements Optimization, injects more of that information at every step.

Tested on eight linear and nonlinear tasks across two image datasets, the module set new state-of-the-art results on several tasks while using at most one hundred evaluations, except for one hard case. It plugs into existing solvers without redesign. In one pairing, it matched quality that rivals needed ten to forty times more evaluations to reach.

Catalog record

CREATORTianyu Chen, Zhendong Wang, Mingyuan Zhou
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing and Accelerating Diffusion-Based Inverse Problem Solving through Measurements Optimization Tianyu Chen, Zhendong Wang, Mingyuan Zhou Abstract Diffusion models have recently demonstrated notable success in solving inverse problems. However, current diffusion model-based solutions typically require a large number of function evaluations (NFEs) to generate high-quality images conditioned on measurements, as they incorporate only limited information at each step. To accelerate the diffusion-based inverse problem-solving process, we introduce M easurements O ptimization (MO), a more efficient plug-and-play module for integrating measurement information at each step of the inverse problem-solving process. This method is comprehensively evaluated across eight diverse linear and nonlinear tasks on the FFHQ and ImageNet datasets. By using MO, we establish state-of-the-art (SOTA) performance across multiple tasks, with key advantages: (1) it operates with no more than 100 NFEs, with phase retrieval on ImageNet being the sole exception; (2) it achieves SOTA or near-SOTA results even at low NFE counts; and (3) it can be seamlessly integrated into existing diffusion model-based solutions for inverse problems, such as DPS [ 5 ] and Red-diff [ 15 ] . For example, DPS-MO attains a peak signal-to-noise ratio (PSNR) of 28.71 dB on the FFHQ 256 dataset for high dynamic range imaging, setting a new SOTA benchmark with only 100 NFEs, whereas current methods require between 1000 and 4000 NFEs for comparable performance. 1 Introduction Inverse problems, which aim to recover the true signal \bm{x_{0}} from a corrupted and noisy measurement \bm{y} , have broad applications across various fields, including image restoration [ 23 , 12 , 30 , 28 , 5 , 15 ] and physics [ 19 , 1 ] . The measurement process often functions as a many-to-one mapping from \bm{x_{0}} to \bm{y} , resulting in non-unique solutions. One approach to addressing inverse problems is through a Bayesian framework, where we aim to sample \bm{x_{0}} from the posterior p(\bm{x_{0}}|\bm{y})\propto p(\bm{y}|\bm{x_{0}})p(\bm{x_{0}}) , with p(\bm{y}|\bm{x_{0}}) representing the known measurement process and p(\bm{x_{0}}) serving as a prior for \bm{x_{0}} . In cases where \bm{x_{0}} represents an image, the inverse problem aligns with low-level vision tasks, specifically targeting image restoration tasks such as inpainting, deblurring, and super-resolution. In this setting, an effective prior p(\bm{x_{0}}) would be a deep image generation model [ 22 ] . A more accurate image prior enhances solutions to inverse problems; thus, given the SOTA performance of diffusion models in image generation [ 6 , 23 , 11 ] , it is natural to consider how diffusion models can serve as a prior. Figure 1 : DPS-MO examples of restoration. We integrate the MO module into DPS framework and present the recovered images with measurements and ground truth to demonstrate the performance of our method. All results are achieved with 100 NFEs. Existing methods for solving inverse problems using diffusion models generally fall into two categories. The first is sampling-based methods [ 5 , 28 , 23 , 20 , 23 ] , which aim to inject measurement information during inference and sample images starting from Gaussian noise. For example, DPS [ 5 ] , approximate \nabla_{\bm{x}_{t}}\log p(\bm{x}_{t}|\bm{y}) during inference. The second category is training-based methods [ 15 , 27 ] , such as Red-diff , which directly optimize the images \bm{x_{0}} guided by both the diffusion model and the measurement \bm{y} . While methods in both categories have demonstrated strong performance in solving inverse problems, they suffer from high NFE requirements—typically around 1000 NFEs—to maintain high-quality image generation. Additionally, they exhibit limited performance on nonlinear problems, particularly in tasks like phase retrieval, even though some of the methods are designed to solve the general inverse problems. The most recent advance in improving performance on nonlinear tasks is DAPS [ 30 ] , which samples from p(\bm{x_{0}}|\bm{y},\bm{x_{t}}) and achieves strong results in nonlinear tasks but requires as many as 4000 NFEs. To achieve desirable performance on nonlinear tasks while minimizing NFEs, we introduce a novel module, M easurement O ptimization (MO). MO combines iterative Stochastic Gradient Langevin Dynamics with querying the pretrained diffusion prior, efficiently integrating measurement information at each step. This approach reduces NFE requirements to as few as 50–100 NFEs for most tasks, while maintaining or achieving state-of-the-art performance across multiple benchmarks. MO is user-friendly and can be seamlessly integrated into existing diffusion-based solutions, such as DPS and Red-diff, enhancing both performance and efficiency. Qualitative examples are shown in Figure 1 . 2 Background Since this paper is focused on diffusion-based solutions for inverse problems, we begin by introducing some preliminary knowledge on diffusion-based generative models and inverse problems in this section. We also briefly discuss why previous solutions require high NFEs and face difficulties in solving nonlinear tasks. 2.1 Diffusion Models Diffusion models generate images by initially sampling from Gaussian noise and then applying a trained score function to solve a reverse stochastic differential equation (SDE) or ordinary differential equation (ODE) to produce images. Using the notation from EDM [ 11 ] , we define a forward process that adds Gaussian noise to a clean image \bm{x_{0}} via a perturbation kernel: \bm{x_{t}}\sim\mathcal{N}(s(t)\bm{x_{0}},s(t)^{2}\sigma(t)^{2}\bm{I}) , where s(t) and \sigma(t) are predefined time-dependent scaling and variance functions. This forward process corresponds to a reverse ODE with the same marginal distribution p(\bm{x_{t}}) for all t as the forward process, given by \displaystyle\text{d}\bm{x}=\left[\frac{\dot{s}(t)}{s(t)}\bm{x}-s(t)^{2}\dot{\sigma}(t)\sigma(t)\nabla_{\bm{x}}\log p\left(\frac{\bm{x}}{s(t)};\sigma(t)\right)\right]\text{d}t, (1) The functions s(t) , \sigma(t) are predefined scale and variance schedules, which may vary depending on the specific diffusion model design, such as VP [ 23 ] , VE [ 22 ] , DDIM [ 24 ] , or EDM [ 11 ] . Here, \dot{s}(t) and \dot{\sigma}(t) represent the time derivatives of s(t) and \sigma(t) , respectively. The score function \nabla_{\bm{x}}\log p\left(\frac{\bm{x}}{s(t)};\sigma(t)\right) can be approximated by a score model trained using diffusion loss. Then to sample from the data distribution p(\bm{x_{0}}) , we first draw initial samples from N(\bm{0},s(t_{\text{max}})^{2}\sigma(t_{\text{max}})^{2}\bm{I}) and then solve the reverse ODE in Eq. 1 . Note that in this paper, we use \bm{x} and \bm{x_{t}} interchangeably when there is no ambiguity, for clarity of illustration and simplicity of notation. 3 Inverse Problem under the Bayesian Framework The inverse problem aims to recover the true signal \bm{x_{0}} from a corrupted and noisy measurement \bm{y} . Formally, we assume the measurement \bm{y} is generated by the process \displaystyle\bm{y}=\mathcal{A}(\bm{x_{0}})+\bm{n}, (2) where \mathcal{A} is a known forward operator, which can be either linear or nonlinear, and typically represents a many-to-one mapping. The term \bm{n} represents noise in the observation process, following a distribution \bm{n}\sim N(0,\sigma_{\bm{n}}^{2}\bm{I}) . Our goal is to recover \bm{x_{0}} from the measurement \bm{y} . Since \mathcal{A} is a many-to-one and potentially nonlinear operator, directly inverting it is challenging. Moreover, because the solution is not unique, additional prior information about \bm{x_{0}} is needed. One approach to address the non-uniqueness is to introduce a prior distribution for \bm{x_{0}} . This leads us to a Bayesian framework, where we aim to sample \bm{x_{0}} from the posterior p(\bm{x_{0}}|\bm{y})\propto p(\bm{y}|\bm{x_{0}})p(\bm{x_{0}}) . In our setting, we focus on image restoration tasks, where the prior is a diffusion model trained on a corresponding dataset. When using a diffusion generative model as the prior, we seek to sample \bm{x_{0}} conditioned on \bm{y} . This modifies the unconditional score function \nabla_{\bm{x_{t}}}\log p(\bm{x_{t}}) to a conditional score: \displaystyle\nabla_{\bm{x_{t}}}\log p(\bm{x_{t}}|\bm{y})=\nabla_{\bm{x_{t}}}\log p(\bm{x_{t}})+\eta\nabla_{\bm{x_{t}}}\log p(\bm{y}|\bm{x_{t}}), where

Read the complete text