When, why, and how do diffusion posterior samplers fail? A finite-sample lens

Benjamin A. Burns, Sara Fridovich-Keil · 2026

Read When, why, and how do diffusion posterior samplers fail? A finite-sample lens by Benjamin A. Burns, Sara Fridovich-Keil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30330.txt.

Introduction

Benjamin A. Burns and Sara Fridovich-Keil ask when, why, and how diffusion posterior samplers fail, via a finite-sample lens. They give a finite-sample view of the posterior as training set size grows. With this lens, approximations tend to misjudge posterior spread at middle steps. This causes early stopping errors, mode weighting errors, and hallucination. They find these errors need neither a nonlinear model nor a multimodal posterior, arising from a multimodal prior and bad posterior spread.

Catalog record

CREATORBenjamin A. Burns, Sara Fridovich-Keil
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE141810
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When, why, and how do diffusion posterior samplers fail? A finite-sample lens Benjamin A. Burns, Sara Fridovich-Keil Abstract Diffusion models have excellent capacity to model complex distributions of natural data, which has made them a popular and effective choice for posterior sampling in imaging inverse problems. Existing methods can incorporate any measurement model at inference time but must use an inexact approximation for the likelihood at intermediate timesteps for computational tractability. Although these approximations can often work well empirically, their downstream effect on the sampled posterior is poorly understood and can result in unexplained failures. To understand when, why, and how these likelihood approximations propagate to erroneous posterior distributions, we introduce a finite-sample perspective on posterior sampling that approximates the posterior to arbitrary precision as training set size tends towards infinity, for any forward model and prior distribution. Using this finite-sample lens, we observe that popular posterior sampling approximations tend to under- or over-estimate the spread of the posterior at intermediate timesteps, causing downstream consequences including sensitivity to early stopping time, inaccurate relative weighting of posterior modes, and hallucination, both of prior modes that are not in the posterior and likelihood modes that are not supported by the prior. Moreover, we find that the cause of these posterior errors requires neither a nonlinear measurement model nor a multimodal posterior, but can arise solely due to a multimodal prior and inaccurate posterior spread at intermediate sampling times. Our finite-sample posterior sampling approach is agnostic to the type of likelihood approximation and the type of (linear or nonlinear) forward model, and can thus serve as a drop-in diagnostic to evaluate the accuracy and failure modes of existing and future posterior samplers. All code for experiments is available at: https://github.com/voilalab/diagnosing-posterior-sampling . 1 Introduction Computational imaging has become a staple of modern science, medicine, security, and industry, allowing us to computationally visualize inside solid objects and at scales invisible to the human eye. Computational imaging works by first collecting indirect measurements through a known physical forward model , and then computationally inverting that forward model to recover an interpretable image of the target object. In cases where the forward model is linear, and we have prior knowledge that constrains the target image to lie in a convex set, the rich theory of compressive sensing guarantees that the true image can be recovered exactly from a minimal number of measurements [ 1 ] . However, in many computational imaging problems of modern interest, the forward model may be nonlinear, our prior knowledge may be nonconvex or statistical rather than structural, and we may have measurements that are too few or too noisy to uniquely determine the true image. In this context, the posterior distribution is the primary object of interest, as it allows us to (i) produce candidate reconstructions via sampling and (ii) quantify confidence of candidate reconstructions via density evaluation, both of which have downstream applications in uncertainty quantification [ 2 , 3 ] , active measurement acquisition [ 4 , 5 , 6 ] , and risk-aware decision-making [ 7 , 8 ] . A key challenge in posterior sampling is to accurately capture events or posterior modes which have low probability but high impact, such as observing a tumor in medical imaging or a small dog or child in autonomous driving. Diffusion models [ 9 , 10 , 11 ] have become an increasingly popular technique for learning a complex distribution from a dataset of samples, and thus an increasingly popular statistical prior for computational imaging [ 12 ] . In particular, score-based methods are an appealing option for posterior sampling because the posterior score, the object required for sampling, decomposes exactly as a sum of a prior score plus a likelihood score. The prior score is learnable from clean image examples, while the likelihood can be written in marginal form given the prior, noise schedule, and forward model. However, these so-called plug-and-play [ 13 ] or zero-shot methods [ 14 ] cannot directly use this marginal for posterior sampling as evaluating the integral for each sample at each denoising step is computationally tractable. Moment-matching methods replace the true marginal with a tractable surrogate, specifically by selecting a surrogate for the denoising process of the diffusion model. Although the denoiser is unknown, its moments are accessible through the noise schedule and learned prior score via Tweedie’s formula [ 15 ] . However, satisfying both conditions is restrictive. Dirac approximations such as Diffusion Posterior Sampling (DPS) [ 16 ] analytically simplify for all measurement models, but capture no spread information about the data nor denoising schedule. Gaussian approximations [ 17 , 18 ] capture additional information, yet only analytically simplify for linear inverse problems with Gaussian measurement noise. The posterior samples generated under these approximations are often high quality, but can fail dramatically in relatively common imaging contexts such as quantized sensing [ 19 ] . The root causes of both (i) the unexpectedly strong general performance and (ii) sporadic failure of these approximations are poorly understood. Thus, despite the increasing prevalence of diffusion posterior samplers, relatively little is known about when such methods will fail, why these approximations only fail in isolated examples, and what the characteristic nature of their failure is. Contributions: To study when, why, and how these existing approaches fail, we introduce a finite-sample perspective on posterior sampling that approximates the posterior to arbitrary precision as training set size tends towards infinity, for any forward model and prior. We provide algorithmic analysis and finite-sample rates which inform when the finite-sample perspective gives rise to a tractable posterior sampling method, and leverage this perspective as a lens into the nature of the approximations proposed in prior work and their downstream consequences. Through this finite-sample lens, we observe that Gaussian approximations [ 17 , 18 ] tend to commit to prior modes too early in the sampling process, which can lead them to place inaccurate relative weight on different posterior modes as well as hallucinate samples from prior modes that are not in the posterior. For Dirac approximations (DPS) [ 16 ] , our finite-sample lens reveals that the choice of parameter \zeta causes DPS to over- or under-weight the likelihood relative to the prior at intermediate timesteps during posterior sampling, which can cause erroneous posterior variance at intermediate timesteps, hallucination of prior modes that are inconsistent with the measurement, and/or measurement-consistent modes that are inconsistent with the prior. These effects can arise even under linear measurement models, Gaussian measurement noise, and unimodal posteriors, as long as the prior distribution is multimodal. 2 Related work We briefly discuss key related work here, but defer extended discussion to Sections A.1 , A.2 and A.3 . Generative posterior samplers. Our goal is to analyze the effect of popular moment-matching likelihood (and hence likelihood score) approximations [ 16 , 17 , 18 ] on posterior sampling. Alternative methods which employ either learning-based likelihood or posterior scores [ 5 ] or exact likelihood sampling (e.g. via variable splitting) [ 19 ] exist. However, we focus on moment-matching methods [ 16 , 17 , 18 ] due to two of their key benefits: (i) they can reuse pre-trained prior scores, which are often pre-trained with significantly

Read the complete text