Guess & Guide: Gradient-Free Zero-Shot Diffusion Guidance
Abduragim Shtanchaev, Albina Ilina, Yazid Janati, Arip Asadulaev, Martin Takáč, Eric Moulines · 2026
Read Guess & Guide: Gradient-Free Zero-Shot Diffusion Guidance by Abduragim Shtanchaev, Albina Ilina, Yazid Janati, Arip Asadulaev, Martin Takáč, Eric Moulines (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07860.txt.
Introduction
Shtanchaev and colleagues speed up posterior sampling for inverse problems with pretrained diffusion models. Full guidance through the whole reverse path is costly. Their solver, Spin, starts from a mid-path posterior state instead of pure noise. It applies light fixes only at set steps where the denoiser can still clean up flaws. Data fit is then enforced in pixel space without backprop through the denoiser. Tests on face and image sets show rival quality at far lower time and memory cost.
Catalog record
| CREATOR | Abduragim Shtanchaev, Albina Ilina, Yazid Janati, Arip Asadulaev, Martin Takáč, Eric Moulines |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88291 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.07860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Guess & Guide: Gradient-Free Zero-Shot Diffusion Guidance Abduragim Shtanchaev, Albina Ilina, Yazid Janati, Arip Asadulaev, Martin Takáč, Eric Moulines Abstract Pretrained diffusion models serve as effective priors for Bayesian inverse problems. These priors enable zero-shot generation by sampling from the conditional distribution, which avoids the need for task-specific retraining. However, a major limitation of existing methods is their reliance on surrogate likelihoods that require vector-Jacobian products at each denoising step, creating a substantial computational burden. To address this, we introduce a lightweight likelihood surrogate that eliminates the need to calculate gradients through the denoiser network. This enables us to handle diverse inverse problems without backpropagation overhead. Experiments confirm that using our method, the inference cost drops dramatically. At the same time, our approach delivers the highest results in multiple tasks. Broadly speaking, we propose the fastest and Pareto optimal method for Bayesian inverse problems. Figure 1 : FFHQ and ImageNet Qualitative results. Comparison against baseline methods over different tasks: Gaussian Deblurring and Center Inpainting on FFHQ ( top and middle rows respectively ), and Motion Deblurring on ImageNet ( bottom row ). Guess and Guide (ours) achieves more accurate and detailed restoration compared to existing approaches. See Appendix H / J for more samples. 1 Introduction Zero-shot generative modeling refers to leveraging a pre-trained generative model of the data distribution p(\mathbf{x}) and adapting it without task-specific retraining to solve downstream tasks it has never seen during training ( Moufad et al. 2025 ) . In this setting, the only new ingredient at test time is a description of the task (e.g., a measurement operator or constraint), and the generative prior is used to produce realistic solutions that satisfy these new conditions. A convenient probabilistic formalization of many such zero-shot tasks is through Bayesian inverse problems . Inverse problems arise when a signal of interest must be recovered from incomplete, corrupted, or indirect observations. Formally, this involves retrieving an original signal \mathbf{x} from a measurement \mathbf{y}=\mathcal{A}(\mathbf{x})+n , where \mathcal{A} is a degradation operator (such as a blur kernel or masking) and n is noise. Because information is usually lost during this process, the problem is ill-posed and requires strong prior knowledge to solve. The Bayesian framework offers a principled approach to these challenges by combining a data likelihood with a learned or structured prior distribution. A prominent strategy is to employ a strong generative prior at inference time to solve arbitrary inverse problems. These methods function as training-free posterior sampling frameworks, enabling plug-and-play adaptation across diverse observation models. In this setup, the diffusion model ( Sohl-Dickstein et al. 2015 ; Ho et al. 2020 ; Song & Ermon 2019 ; Song et al. 2020b ) serves as a gradient field that iteratively denoises the candidate solution, guiding it toward the manifold of realistic data while ensuring it remains consistent with the observed measurements. Recent methods achieved impressive results in such inverse problems as inpainting, deblurring, super-resolution, and source separation ( Lugmayr et al. 2022 ; Kawar et al. 2021 ; Kawar et al. 2022 ; Saharia et al. 2022 ) At the core of diffusion-based posterior sampling is a sequential denoising process that approximates intermediate posterior distributions. These distributions combine the learned prior with progressively refined likelihoods at each noise level. However, the exact posterior denoiser involves an intractable likelihood gradient, leading most methods to rely on approximations that require computing vector-Jacobian products through the denoiser network at every step ( Chung et al. 2022a ; Song et al. 2023b ) . While effective, this introduces substantial computational and memory overhead, limiting scalability and practical deployment. To avoid these problems, we present the Guess and Guide (G&G) framework for backpropagation-free zero-shot generative modeling. Our approach introduces a lightweight optimization procedure for Bayesian Inverse Inference that avoids costly vector-Jacobian products at each sampling step. First, an initial guess phase rapidly obtains a high-quality estimate through iterative optimization. Next, a guided denoising phase refines this estimate, using a flexible custom scheduling strategy. We show that our simple approach significantly reduces memory usage and inference time. Importantly, our method achieves high reconstruction quality across diverse tasks, including super-resolution, inpainting, deblurring, phase retrieval, and HDR reconstruction. Together, we propose a fast, general, and gradient-free framework to solve diffusion-based inverse problems. 2 Background Denoising diffusion models (DDMs) ( Sohl-Dickstein et al. 2015 ; Song & Ermon 2019 ; Ho et al. 2020 ) generate samples from a target distribution p_{0} by constructing a distribution interpolation path \{p_{t}\}_{t\in[0,1]} that connects the data distribution p_{0} to the standard Gaussian base distribution p_{1}:=\mathcal{N}(0,I_{d}) . This interpolation is defined through a noising process where each intermediate distribution p_{t}=\text{Law}(X_{t}) arises from the mixture X_{t}=\alpha_{t}X_{0}+\sigma_{t}X_{1},\quad X_{0}\sim p_{0},\quad X_{1}\sim p_{1}, (1) with independent random variables X_{0}\sim p_{0} and X_{1}\sim p_{1} . The coefficients (\alpha_{t})_{t\in[0,1]} and (\sigma_{t})_{t\in[0,1]} are deterministic schedules that satisfy monotonicity constraints (non-increasing and non-decreasing, respectively) and boundary conditions (\alpha_{0},\sigma_{0}):=(1,0) and (\alpha_{1},\sigma_{1}):=(0,1) . Common instantiations include the variance-preserving schedule where \alpha_{t}^{2}+\sigma_{t}^{2}=1 ( Ho et al. 2020 ; Dhariwal & Nichol 2021 ) , and the linear schedule with (\alpha_{t},\sigma_{t})=(1-t,t) ( Lipman et al. 2022 ; Esser et al. 2024 ; Gao et al. 2024 ) . To sample from p_{0} , DDIM ( Song et al. 2020a ) starts from a sample drawn from p_{1} at time t_{K}=1 . It then performs a sequence of deterministic reverse transitions, progressively denoising each sample \hat{X}_{t_{k+1}} into a less noisy sample \hat{X}_{t_{k}} , until reaching the clean distribution at t_{0}=0 . The DDIM transition from t_{k+1} to t_{k} is defined as \hat{X}_{t_{k}}=\alpha_{t_{k}}\hat{\mathbf{x}}^{\theta}_{0}(\hat{X}_{t_{k+1}},{t_{k+1}})+\sigma_{t_{k}}\hat{\mathbf{x}}^{\theta}_{1}(\hat{X}_{t_{k+1}},{t_{k+1}}), where \hat{\mathbf{x}}^{\theta}_{0}(\cdot,{t_{k+1}}) and \hat{\mathbf{x}}^{\theta}_{1}(\cdot,{t_{k+1}}) are respectively neural network approximations of the denoiser \mathbb{E}[X_{0}|X_{t}] and noise prediction \mathbb{E}[X_{1}|X_{t}] under the model ( 1 ). Since it holds, following the same model, that \mathbb{E}[X_{1}|X_{t}]=(X_{t}-\alpha_{t}\mathbb{E}[X_{0}|X_{t}])/\sigma_{t} , only the denoisers (\hat{\mathbf{x}}^{\theta}_{0}(\cdot,t))_{t\in[0,1]} are trained by minimizing an L_{2} denoising loss and \hat{\mathbf{x}}^{\theta}_{1}(\mathbf{x},t) is set to (\mathbf{x}-\alpha_{t}\hat{\mathbf{x}}^{\theta}_{0}(\mathbf{x},t))/\sigma_{t} . The DDIM framework also allows for stochastic updates. In this case, \hat{X}_{t_{k}}=\alpha_{t_{k}}\hat{\mathbf{x}}^{\theta}_{0}(\hat{X}_{t_{k+1}},{t_{k+1}})\\ +\sqrt{\sigma^{2}_{t_{k}}-\eta^{2}_{t_{k}}}\hat{\mathbf{x}}^{\theta}_{1}(\hat{X}_{t_{k+1}},{t_{k+1}})+\eta_{t_{k}}W_{k}, (2) where W_{k}\sim\mathcal{N}(0,\mathbf{I}) and 0\leq\eta_{t_{k}}\leq\sigma_{t_{k}} . In what follows we write p^{\theta}_{{t_{k}}\mathrel{\raisebox{0.45206pt}{$\scriptscriptstyle\mid$}}{t_{k+1}}}(\cdot|\mathbf{x}_{t_{k+1}}) to refer to the associated conditional density (the dependence on \eta is left implicit). In practice, most diffusion models are implemented in the latent space ( Rombach et al. 2022 ; Esser et al. 2024 ) , assuming an encoder-decoder pair (\mathcal{E},\mathcal{D}) . The model is trained on Z_{0}=\mathcal{E}(X_{0}) where X_{0}\sim p_{0} . At inference, reverse diffusion occurs in the latent space, and the final sample is obtained via decoding. We adopt this framework, assuming p_{0} and the process (Z_{t})_{t} (analogous to ( 1 )) are defined in the latent space. However, our approach applies equally to pixel space by treating \mathcal{E} and \mathcal{D} as identity mappings. Inverse problems. In this work, we investigate a broad class of image restoration and reconstruction problems formulated as inverse tasks. We consider an unknown clean image \mathbf{x}_{*}\in\mathbb{R}^{d} that is observed through a (possibly nonlinear) measurement process described by an operator \mathcal{A}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m} . The observed data are modeled as \mathbf{y}=\mathcal{A}(\mathbf{x}_{*})+\sigma_{y}W,\qquad W\sim\mathcal{N}(0,\mathbf{I}_{m}), (3) where \sigma_{y}>0 denotes the noise level. The task is to recover a reconstruction \hat{\mathbf{x}} that explains the observation (\mathcal{A}(\hat{\mathbf{x}})\approx\mathbf{y}) while remaining consistent with the statistical properties of natural images. Within a Bayesian framework, the prior distribution p_{0} captures encoded natural-image statistics, and the measurement process is modeled by