Latent Stochastic Interpolants

Saurabh Singh, Dmitry Lagun · 2025

Read Latent Stochastic Interpolants by Saurabh Singh, Dmitry Lagun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02276.txt.

Introduction

Saurabh Singh and Dmitry Lagun move a generative modeling technique into latent space in this two thousand twenty-five paper. Stochastic interpolants bend one probability distribution into the next in a free way, powering today's generative models. To use them in jointly trained latent variable models was unexplored. The method needs straight access to samples from both distributions. Latent Stochastic Interpolants solves this, learning the encoder, decoder, and the latent interpolant as one, end to end.

The training goal is an evidence lower bound set out straight in continuous time. Joint tuning lets the model learn latent forms that are of use. The interpolant turns a free prior into the encoder's aggregated posterior. This dodges the tight simple priors of normal diffusion models. It dodges the heavy cost of working in high-dimensional data space too. Tests on the large-scale ImageNet generation test show the way works at scale.

Catalog record

CREATORSaurabh Singh, Dmitry Lagun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Stochastic Interpolants Saurabh Singh, Dmitry Lagun Abstract Stochastic Interpolants (SI) are a powerful framework for generative modeling, capable of flexibly transforming between two probability distributions. However, their use in jointly optimized latent variable models remains unexplored as they require direct access to the samples from the two distributions. This work presents Latent Stochastic Interpolants (LSI) enabling joint learning in a latent space with end-to-end optimized encoder, decoder and latent SI models. We achieve this by developing a principled Evidence Lower Bound (ELBO) objective derived directly in continuous time. The joint optimization allows LSI to learn effective latent representations along with a generative process that transforms an arbitrary prior distribution into the encoder-defined aggregated posterior. LSI sidesteps the simple priors of the normal diffusion models and mitigates the computational demands of applying SI directly in high-dimensional observation spaces, while preserving the generative flexibility of the SI framework. We demonstrate the efficacy of LSI through comprehensive experiments on the standard large scale ImageNet generation benchmark. 1 Introduction Diffusion models have achieved remarkable success in modeling complex, high-dimensional data distributions across various domains. These models learn to transform a simple “prior” distribution p_{0} , such as a standard Gaussian, into a complex data distribution p_{1} . While early formulations were constrained to use specific prior distributions that are Lévy Stable, recent advancements, particularly Stochastic Interpolants (SI) ( Albergo et al. 2023 ) offer a powerful, unifying framework capable of bridging arbitrary probability distributions. However, SI assumes that both the prior p_{0} and the target p_{1} distributions are fixed and the samples from both are directly observed . This requirement limits their use in jointly learned latent variable models where the generative model is learned, along with an encoder and a decoder, in a latent unobserved space. Further, the latent space, often lower dimensional, evolves as the encoder and decoder are jointly optimized. Lack of support for joint optimization implies that arbitrary fixed latent representations may not be optimally aligned with the generative process resulting in inefficiencies. To address this, we present Latent Stochastic Interpolants (LSI), a novel framework for end-to-end learning of a generative model in an unobserved latent space. Our key innovation lies in deriving a principled, flexible and scalable training objective as an Evidence Lower Bound (ELBO) directly in continuous time. This objective, like SI, provides data log-likelihood control, while enabling scalable end-to-end training of the three components: an encoder mapping high-dimensional observations to a latent space, a decoder reconstructing observations from latent representations, and a latent SI model operating entirely within the learned latent space. Our approach allows transforming arbitrary prior distributions into the encoder-defined aggregated posterior, simultaneously aligning data representations with a high-fidelity generative process using that representation. LSI’s single ELBO objective provides a unified, scalable framework that avoids the need for simple priors of the normal diffusion models, mitigates the computational demands of applying SI directly in high-dimensional observation spaces and offers an alternative to ad-hoc multi-stage training. Our formulation admits simulation-free training analogous to observation-space diffusion and SI models, while preserving the flexibility of SI framework. We empirically validate LSI’s strengths through comprehensive experiments on the challenging ImageNet generation benchmark, demonstrating competitive generative performance and highlighting its advantages in efficiency. Our key contributions are: 1) Latent stochastic interpolants (LSI): a novel and flexible framework for scalable training of a latent variable generative model with continuous time dynamic latent variables, where the encoder, decoder and latent generative model are jointly trained, 2) Unifying perspective: a novel perspective on integrating flexible continuous-time formulation of SI within latent variable models, leveraging insights from continuous time stochastic processes, 3) Principled ELBO objective: a new ELBO as a principled training objective that retains strengths of SI – simple simulation free training and flexible prior choice – while enabling the benefits of joint training in a latent space. 2 Background Notation. We use small letters x,y,t etc. to represent scalar and vector variables, f,g etc. to represent functions, Greek letters \beta,\theta etc. to represent (hyper-)parameters. Lower case letters x are used to represent both the random variable and a particular value x\sim p(x) . Dependence on an argument t is indicated as a subscript u_{t} or argument u(t) interchangeably. Our work builds upon two key results briefly reviewed below. The first result ( Li et al. 2020 ) states an Evidence Lower Bound (ELBO) for models using continuous time dynamic latent variables. The second result is a well known method for constructing a stochastic mapping between two distributions. We exploit it to construct a variational approximation in the latent space. 2.1 Variational Lower Bound using Dynamic Latent Variables As in Li et al. 2020 , consider two SDEs, starting with the same starting point z_{0}\sim p_{0}(z_{0}) at t=0 . \displaystyle d\tilde{z}_{t} \displaystyle=h_{\theta}(\tilde{z}_{t},t)dt+\sigma(\tilde{z}_{t},t)dw_{t}, (model) (1) \displaystyle dz_{t} \displaystyle=h_{\phi}(z_{t},t)dt+\sigma(z_{t},t)dw_{t}, (variational posterior) (2) Where w_{t} is the Wiener process. The first equation can be viewed as the latent dynamics under the model h_{\theta} we are interested in learning and the second as the latent dynamics under some variational approximation to the posterior that can be used to produce samples z_{t} . The dispersion coefficient \sigma(\cdot,\cdot) is assumed to be common and known. Further, let x_{t_{i}} be observations at time t_{i} that are assumed to only depend on the corresponding unobserved latent state z_{t_{i}} , then the ELBO can be written as \displaystyle\ln p_{\theta}(x_{t_{1}},\dots,x_{t_{n}})\geq\mathbb{E}_{z_{t}}\left[\sum_{i=1}^{n}\ln p_{\theta}(x_{t_{i}}|z_{t_{i}})-\int_{0}^{T}\frac{1}{2}\lVert u(z_{t},t)\rVert^{2}dt\right] (3) Where u satisfies \displaystyle\sigma(z,t)u(z,t)=h_{\phi}(z,t)-h_{\theta}(z,t) (4) We refer the reader to Li et al. 2020 for additional details and proof. Similar to the ELBO for the VAEs ( Kingma et al. 2013 ) , the first term in eq. 3 can be viewed as a reconstruction term and the second term as approximating the posterior resulting from h_{\theta} with the variational approximation h_{\phi} . 2.2 Diffusion Bridge Given two arbitrary points z_{0} and z_{1} , a diffusion bridge between the two is a random process constrained to start and end at the two given end points. A diffusion bridge can be used to specify the stochastic dynamics of a particle that starts at z_{0} at t=0 and is constrained to land at z_{1} at t=1 . Consider a stochastic process starting at z_{0} with the dynamics specified by eq. 2 . Using Doob’s h-transform, the SDE for the end point conditioned diffusion bridge, constrained to end at z_{1} at time t=1 can be written as \displaystyle dz_{t} \displaystyle=[h_{\phi}(z_{t},t)+\sigma(z_{t},t)\sigma(z_{t},t)^{T}\nabla_{z_{t}}\ln p(z_{1}|z_{t})]dt+\sigma(z_{t},t)dw_{t} (5) where p(z_{1}|z_{t}) is the conditional density for z_{1} under the original dynamics in eq. 2 and depends on h_{\phi} . Note that a Brownian bridge is a special case of a Diffusion bridge where the dynamics are specified by the standard Brownian motion. Diffusion bridges can be used to construct a stochastic mapping between two distributions by considering the end points z_{0}\sim p_{0}(z_{0}) and z_{1}\sim p_{1}(z_{1}) to be sampled from the two distributions of interest. 3 Latent Stochastic Interpolants Stochastic Interpolants (SI) and their limitation: Let x_{1}\sim p(x_{1}) be an observation from the data distribution p(x_{1}) that we want to model. In SI framework, another distribution p_{0}(x_{0}) is chosen as a prior with samples x_{0}\sim p_{0}(x_{0}) . A stochastic interpolant x_{t} is then constructed with the requirement that the marginal distribution p_{t}(x_{t}) of x_{t} equals p_{0} at t=0 and p_{1}

Read the complete text