Diffusion priors for Bayesian 3D reconstruction from incomplete measurements

Julian L. Möbius & Michael Habeck · 2024

Read Diffusion priors for Bayesian 3D reconstruction from incomplete measurements by Julian L. Möbius & Michael Habeck (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14897.txt.

Introduction

Julian Möbius and Michael Habeck rebuild 3D shapes from broken data using learned priors in a 2024 paper. Many measurement problems are ill-posed: too little data, too many possible shapes. Classic Bayesian fixes prefer smooth or sparse answers, which look nothing like real objects.

Their priors come from diffusion models trained on 3D point clouds. Those models know what plausible objects look like at medium detail. Folded into Bayesian sampling, the prior steers guesses toward realistic shapes. The test case is cryo-electron microscopy, rebuilding protein assemblies from sparse, low-resolution views. Posterior sampling recovered structures from very partial observations. The authors pitch learned priors as a general upgrade for inverse problems.

Catalog record

CREATORJulian L. Möbius & Michael Habeck
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusion priors for Bayesian 3D reconstruction from incomplete measurements Julian L. Möbius & Michael Habeck Abstract Many inverse problems are ill-posed and need to be complemented by prior information that restricts the class of admissible models. Bayesian approaches encode this information as prior distributions that impose generic properties on the model such as sparsity, non-negativity or smoothness. However, in case of complex structured models such as images, graphs or three-dimensional (3D) objects, generic prior distributions tend to favor models that differ largely from those observed in the real world. Here we explore the use of diffusion models as priors that are combined with experimental data within a Bayesian framework. We use 3D point clouds to represent 3D objects such as household items or biomolecular complexes formed from proteins and nucleic acids. We train diffusion models that generate coarse-grained 3D structures at a medium resolution and integrate these with incomplete and noisy experimental data. To demonstrate the power of our approach, we focus on the reconstruction of biomolecular assemblies from cryo-electron microscopy (cryo-EM) images, which is an important inverse problem in structural biology. We find that posterior sampling with diffusion model priors allows for 3D reconstruction from very sparse, low-resolution and partial observations. 1 Introduction Inverse problems are encountered in many different scientific fields. The basic setting is that we observe noisy and incomplete data {\bm{y}} and seek to find a model {\bm{x}} that predicts mock data via a forward model \mathcal{A} such that {\bm{y}}\approx\mathcal{A}({\bm{x}}) . An important subclass are linear models where \mathcal{A} is a linear operator. Well-known inverse problems are deconvolution or tomography. The challenge in solving inverse problems stems from the fact that they tend to be ill-posed meaning that many models can produce highly similar data and/or the reconstructed model can be very sensitive to noise. The remedy is to combine a reconstruction loss with a regularizer. Well-studied regularizers are Tikhonov regularization (aka ridge regression), sparsity, and non-negativity. Bayesian inference offers a powerful framework to tackle inverse problems. The conditional probability p({{\bm{y}}}\,|\,{{\bm{x}}}) , the likelihood, relates the data {\bm{y}} to the mock data \mathcal{A}({\bm{x}}) via a noise model. A common assumption is independent Gaussian noise resulting in the likelihood p({{\bm{y}}}\,|\,{{\bm{x}}})\propto\exp\left(-\frac{\|{\bm{y}}-\mathcal{A}({\bm{x}})\|^{2}}{2\sigma^{2}}\right)\,. (1) Maximizing the likelihood is then equivalent to standard least-squares fitting. The prior probability p({\bm{x}}) encodes data-independent knowledge about a particular model {\bm{x}} ; its negative logarithm -\log p({\bm{x}}) can be viewed as a regularizer. The posterior of the model is p({{\bm{x}}}\,|\,{{\bm{y}}})=\frac{p({{\bm{y}}}\,|\,{{\bm{x}}})\,p({\bm{x}})}{p({\bm{y}})} (2) with model evidence p({\bm{y}})=\int p({{\bm{y}}}\,|\,{{\bm{x}}})\,p({\bm{x}})\,d{\bm{x}} . In case of a Gaussian likelihood, maximization of \log p({{\bm{x}}}\,|\,{{\bm{y}}}) is equivalent to regularized least-squares fitting. Often detailed knowledge about reasonable solutions is available but difficult to capture by the standard priors that are typically used to tackle inverse problems. For example, cryo-electron microscopy (cryo-EM) aims to reconstruct the three-dimensional structure of macromolecular complexes from two-dimensional (2D) projections. Cryo-EM images are typically very noisy with signal-to-noise ratios (SNR) far below one. On the other hand, a large body of knowledge has been accumulated over the past six decades, including hundreds of thousands of experimentally determined biomolecular structures that are stored in the Protein Data Bank (PDB) ( Berman et al., 2000 ) . Experimentally determined structures exhibit recurrent features such as alpha-helices and beta-strands and preferences for the proximity and packing of amino acids and entire subunits. This detailed information is not captured by standard priors used in cryo-EM reconstruction packages such as CryoDRGN ( Zhong et al., 2021a ; Zhong et al., 2021b ) , cryoSPARC ( Punjani et al., 2017 ) or RELION ( Scheres, 2012 ) . These approaches represent the structure as voxel grid and use generic priors enforcing non-negativity or penalizing high-frequency contributions. If one were to sample volumes from the corresponding prior, the sampled structures would not resemble any of the known biomolecular structures. Here we try to encode the rich knowledge available in the PDB as a diffusion model prior. We test 3D reconstruction from sparse, low-resolution and partial measurements by posterior sampling with diffusion models as priors. 1.1 Contribution Our contributions are as follows: We propose a method to reconstruct 3D structures from 2D projections that utilizes diffusion models as priors. Using diffusion priors has previously not been explored to solve the 2D to 3D reconstruction problem in cryo-EM. The combination of the diffusion model prior with a likelihood allows us to reconstruct 3D structures from very sparse observations such as 2D projections, low-resolution structures and known structures of subunits. This is achieved with diffusion-based posterior sampling (DPS) ( Chung et al., 2023 ) a method that has not yet been investigated in the context of 3D point cloud data. We combine DPS with optimized diffusion schedules and second-order correction steps with adaptable noise injection ( Karras et al., 2022 ) to improve sample quality and runtime. We demonstrate the fidelity and flexibility of our method on highly complex and diverse datasets of 3D point clouds from ShapeNet and the PDB. We emphasize that the reconstruction problem which we solve differs from the problem of reconstructing a 3D surface from a 2D surface color image, which is tackled by, for example, Point-E ( Nichol et al., 2022 ) , Shape-E ( Jun & Nichol, 2023 ) , \text{PC}^{2} ( Melas-Kyriazi et al., 2023 ) , One-2-3-45 ( Liu et al., 2023 ) and BDM ( Xu et al., 2024 ) . The main difference is that in our work the 2D observations are projections that provide information about the density across the full volume rather than only information about the surface. In addition, our approach is also capable of conditioning on coarse-grained or partial observations of the 3D structure. Moreover, the reconstruction problem in cryo-EM aims to reconstruct the internal structure not only the surface. It is important to note that diffusion models have been used in prior work in the context of cryo-EM but have not been used for the 2D to 3D reconstruction problem. DiffModeler ( Wang et al., 2024 ) uses a diffusion model as part of their pipeline to trace the backbone of protein chains within already reconstructed 3D cryo-EM maps to model large protein complexes. Zhang et al. (2024) utilized a diffusion model to denoise and restore 2D cryo-EM images of single particles. Kreis et al. (2022) trained diffusion models on the CryoDRGN latent conformational space of single proteins to tackle the prior hole problem. In Ingraham et al. (2023) diffusion models were used to generate protein complexes at the atomic level and also showed generation conditioned on shape and symmetry constraints. So far they have not used their diffusion model for the 2D to 3D reconstruction task. 2 Background on Diffusion Models Diffusion models have gained wide recognition in the field of generative modeling ( Sohl-Dickstein et al., 2015 ; Song & Ermon, 2019 ; Ho et al., 2020 ; Song et al., 2021 ) , particularly in image synthesis, where diffusion models have demonstrated their capability by surpassing former leading models in key metrics ( Dhariwal & Nichol, 2021 ) and continue to set new records ( Karras et al., 2024 ) . In generative modeling, the main goal is to learn a sampler for an unknown distribution p_{0} from i.i.d. samples {\bm{x}}(0)_{i}\sim

Read the complete text