Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models

Siddarth Venkatraman, Mohsin Hasan, Minsu Kim, Luca Scimeca, Marcin Sendera, Yoshua Bengio, Glen Berseth, Nikolay Malkin · 2025

Read Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models by Siddarth Venkatraman, Mohsin Hasan, Minsu Kim, Luca Scimeca, Marcin Sendera, Yoshua Bengio, Glen Berseth, Nikolay Malkin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06999.txt.

Introduction

Siddarth Venkatraman and co-authors propose outsourced diffusion sampling in a 2025 paper on posterior inference in the latent spaces of generative models. Any tidy generative model can be seen as a fixed map from Gaussian noise to data.

Sampling plain data is easy, but sampling data that must meet a side condition is usually hard. The method trains a diffusion sampler in the noise space instead. Reinforcement learning shapes it so mapped samples match the wanted posterior. Noise space posteriors are often smoother than data space ones, easing the task. Tests span conditional images, feedback learning, and protein shapes. The sampler works with several prior types and beats rival inference routes.

Catalog record

CREATORSiddarth Venkatraman, Mohsin Hasan, Minsu Kim, Luca Scimeca, Marcin Sendera, Yoshua Bengio, Glen Berseth, Nikolay Malkin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-06999

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models Siddarth Venkatraman, Mohsin Hasan, Minsu Kim, Luca Scimeca, Marcin Sendera, Yoshua Bengio, Glen Berseth, Nikolay Malkin Abstract Any well-behaved generative model over a variable 𝐱 \mathbf{x} can be expressed as a deterministic transformation of an exogenous ( ‘outsourced’ ) Gaussian noise variable 𝐳 \mathbf{z} : 𝐱 = f θ ​ ( 𝐳 ) \mathbf{x}=f_{\theta}(\mathbf{z}) . In such a model ( e.g. , a VAE, GAN, or continuous-time flow-based model), sampling of the target variable 𝐱 ∼ p θ ​ ( 𝐱 ) \mathbf{x}\sim p_{\theta}(\mathbf{x}) is straightforward, but sampling from a posterior distribution of the form p ⁡ ( 𝐱 ∣ 𝐲 ) ∝ p θ ​ ( 𝐱 ) ​ r ​ ( 𝐱 , 𝐲 ) p(\mathbf{x}\mid\mathbf{y})\propto p_{\theta}(\mathbf{x})r(\mathbf{x},\mathbf{y}) , where r r is a constraint function depending on an auxiliary variable 𝐲 \mathbf{y} , is generally intractable. We propose to amortize the cost of sampling from such posterior distributions with diffusion models that sample a distribution in the noise space ( 𝐳 \mathbf{z} ). These diffusion samplers are trained by reinforcement learning algorithms to enforce that the transformed samples f θ ​ ( 𝐳 ) f_{\theta}(\mathbf{z}) are distributed according to the posterior in the data space ( 𝐱 \mathbf{x} ). For many models and constraints, the posterior in noise space is smoother than in data space, making it more suitable for amortized inference. Our method enables conditional sampling under unconditional GAN, (H)VAE, and flow-based priors, comparing favorably with other inference methods. We demonstrate the proposed outsourced diffusion sampling in several experiments with large pretrained prior models: conditional image generation, reinforcement learning with human feedback, and protein structure generation. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Left: Top row: Marginal densities of a CNF that transforms a Gaussian distribution ( t=0 ) to a Swiss roll ( t=1 ). Middle row: The constraint function – a mixture of two Gaussians centered an an observation \mathbf{y} ( \bullet ) and its reflection through the origin – pulled back to \mathbf{x}_{t} . Bottom row: Posterior densities at \mathbf{x}_{t} , proportional to the product of the first two rows. The rightmost column shows samples in the data space. Right: The same objects shown in noise and data space for a GAN that transforms noise ( \mathbf{z} ) to data ( x ). Outsourced diffusion samplers approximate p(\mathbf{x}_{0}\mid\mathbf{y}) or p(\mathbf{z}\mid\mathbf{y}) , which are smoother than p(\mathbf{x}\mid\mathbf{y}) (see Fig. 2 ). Table 1 : Common families of generative models can be expressed as deterministic transformations of noise. Here, d_{\rm latent} refers to the dimension of the latent variable as typically understood in the model; typically, d_{\rm latent}\ll d_{\rm data} ( § 2 ). Posteriors under these generative model priors are generally intractable, but some models can be fine-tuned by asymptotically unbiased variational objectives to sample intractable posteriors ( § 3.1 ). Outsourced diffusion sampling approximates posteriors in the noise space ( § 3.2 and 4 ). Model Noise dim. d_{\rm noise} Invertible? Variational tuning? (H)VAE d_{\rm latent}\cdot N+d_{\rm data} × ✓ GAN d_{\rm latent} × × NF / CNF d_{\rm data} ✓ ✓/× Diffusion d_{\rm data}\cdot(T+1) × ✓ Latent diffusion d_{\rm latent}\cdot(T+1)+d_{\rm data} × ✓ Generative models, trained on a dataset to maximize likelihood or related quantities, can become priors for Bayesian inference problems. The aim is to approximate or sample from the product of the modeled distribution over a data space with an observation likelihood or other constraint function. While such diverse applications as conditional generation ( Dhariwal & Nichol, 2021 ; Ho & Salimans, 2022 ) , inverse problems ( Song et al., 2022 ; Chung et al., 2023 ; Venkatraman et al., 2024 ) , and constrained improvement from human feedback ( Korbak et al., 2022 ; Fan et al., 2023 ) can be cast as posterior inference tasks, sampling from such posterior distributions when no unbiased target data is available is generally intractable. For some model families, approximate solutions, such as MCMC, approximate guidance, and variational inference, may be possible. Each of those methods has limitations, such as high cost to reach convergence for multimodal posteriors, intractability of accurate density estimation, and reliance on techniques specialized to the model and constraint. Fundamentally, generative models are probabilistic programs that produce samples from the distributions they define by a combination of deterministic computation and injection of random noise. 1 1 1 We refer here to models that produce samples in a bounded number of operations, not to objects such as deep energy-based models, for which sampling is intractable. This paper argues that the noise space of generative models – the space where the noise injected during generation resides – is an effective target for posterior inference. To be precise, we consider a generative model that expresses data as a deterministic transformation of noise, \mathbf{x}=f_{\theta}(\mathbf{z}) , where the noise variable \mathbf{z} follows a known distribution \mathbf{z}\sim p_{\mathbf{z}}(\mathbf{z}) and \theta are the model parameters. This model defines a distribution over \mathbf{x} – the pushforward [f_{\theta}]_{*}p_{\mathbf{z}} of the noise distribution by the deterministic transformation – with density p(\mathbf{x}) . A constraint function r(\mathbf{x},\mathbf{y}) in the data space, depending on an auxiliary variable \mathbf{y} , defines a posterior distribution p(\mathbf{x}\mid\mathbf{y})\propto p(\mathbf{x})r(\mathbf{x},\mathbf{y}) . This posterior can be sampled by inferring a distribution over the noise variable \mathbf{z} that, when transformed by f_{\theta} , aligns with the posterior in the data space. The posterior in the noise space is often smoother (and lower-dimensional) than the corresponding posterior in the data space ( Fig. 1 ), making it more amenable to efficient sampling ( Fig. 2 ). While posterior sampling in noise space is still intractable, it can be addressed by methods of black-box variational inference. Recent advances in diffusion samplers – diffusion models trained not on a dataset, but to match a given unnormalized density ( Zhang & Chen, 2022 ; Vargas et al., 2023 ; Richter & Berner, 2024 ; Sendera et al., 2024 ) – open an opportunity to model complex posteriors in noise space. We call such amortized posterior inference in the noise space outsourced diffusion sampling . Our exposition and experiments support three claims: (1) Outsourced diffusion sampling is agnostic to the form of the mapping f_{\theta} and applicable to a wide range of prior models, including VAEs, GANs, normalizing flows, and continuous-time flow-based models ( Table 1 ). (2) Outsourced diffusion sampling is an effective posterior inference method under large pretrained generative model priors in a variety of domains: conditional image generation, reinforcement learning with human feedback, discriminator-adjusted GAN sampling, and protein structure generation ( Table 2 ). (3) Outsourced diffusion sampling is more efficient than amortized inference methods that fit a model to sample the data space posterior directly and than non-amortized methods like MCMC, illustrating the flexibility of diffusion sampling in outsourced noise spaces for sampling complex posteriors ( § 5 ). Figure 2 : Marginal densities of a diffusion sampler of the posteriors from the CNF example in Fig. 1 in data space and noise space. The data space posterior (top row) has well-separated modes and is harder to sample from than the outsourced posterior (bottom row). 2 Outsourcing Noise in Generative Models Consider a probabilistic

Read the complete text