Schödinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders
Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama · 2024
Read Schödinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders by Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18237.txt.
Introduction
Diffusion models of a Schrödinger bridge type are recast by Kentaro Kaba and co-authors in a 2024 paper. Such models use forward and backward stochastic equations to link data to a prior distribution. Older models learn only the backward process, while bridge models also learn the forward one, which makes them more flexible but harder to grasp.
The team offers a unified frame that views bridge models as an extension of variational autoencoders. In that view, the data processing inequality plays a central role. The training goal then splits into two clear parts, a prior loss and a drift matching part. Readers who know autoencoders gain a more intuitive path into a model class whose math often feels opaque.
Catalog record
| CREATOR | Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Schödinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama Abstract Generative diffusion models use time-forward and backward stochastic differential equations to connect the data and prior distributions. While conventional diffusion models (e.g., score-based models) only learn the backward process, more flexible frameworks have been proposed to also learn the forward process by employing the Schrödinger bridge (SB). However, due to the complexity of the mathematical structure behind SB-type models, we can not easily give an intuitive understanding of their objective function. In this work, we propose a unified framework to construct diffusion models by reinterpreting the SB-type models as an extension of variational autoencoders. In this context, the data processing inequality plays a crucial role. As a result, we find that the objective function consists of the prior loss and drift matching parts. Introduction.— Generative diffusion model [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] is one of the likelihood-based generative models that have successfully synthesized high-quality samples such as image [ 9 , 10 ] and audio [ 11 , 12 ] . In the framework of the diffusion model, the model distribution that mimics the training dataset is defined by a distribution transported from a simple prior through a stochastic differential equation (SDE). This framework enables us to generate new samples by learning the transport dynamics via neural networks (NNs). A prominent example of generative diffusion models is the score-based model (SBM) [ 3 , 13 ] . In SBMs, the data distribution is gradually perturbed by the addition of noise and eventually converges to a normal distribution that does not have any information. This procedure is commonly referred to as the noising process. Subsequently, the reversal of the noising process is learned to establish a generative process, which synthesizes new samples by transporting the prior distribution (i.e., the normal distribution). Although SBMs are used in many situations [ 9 , 10 , 11 , 12 , 14 ] , they have two main limitations. First, since learning of the reverse process requires computation of the score function (the gradient of the log probability distribution), the noising process must be restricted to a simple form (e.g., linear or degenerate), where the score function can be evaluated analytically. Second, when the data distribution far deviates from the prior, the time required for the noising process becomes longer, which results in substantial delays in sample generation. To overcome these limitations, a more flexible model has recently been proposed using the Schrödinger bridge (SB) [ 4 , 5 , 6 , 7 , 8 ] . The SB is a stochastic optimization technique that derives an SDE to connect any two arbitrary probability distributions over a given time horizon [ 15 , 16 ] . This technique allows us to use highly structured SDEs with NNs for the noising process and acceletates the generation of new samples. However, the mathematical structure underlying SB-type diffusion models is complicated, because it involves mapping partial differential equations that solve SB problems to SDEs of noising and generative processes by using the non-linear Feynman-Kac formula [ 7 , 17 ] . This complicated structure inhibits our intuitive understanding of the objective function used in the training of SB-type models. In this work, we attempt to reinterpret SB-type models as a relatively straightforward extension of variational autoencoders (VAEs) [ 18 , 19 ] . VAEs are generative models that learn two mappings between data and latent spaces: the encoder that maps the data into the latent variables and the decoder that represents its inverse. Recent work [ 20 ] has rederived SBMs in the VAE framework by associating the noising process with the encoder and the generative process with the decoder, respectively. However, the approach in that work has omitted the training of the encoder and has limited the scope to the derivation of SBMs. In this letter, taking into account the training of the encoder, we propose a method to construct the objective function for SB-type models. As a result, it is revealed that the objective function is composed of the prior loss and drift matching parts, which characterize the training of NNs in the encoder and the decoder, respectively. Variational Autoencoders.— We begin with a brief introduction to variational autoencoders. Let \{x_{i}\}_{i=1}^{n}\in\mathcal{X}^{n} be a dataset and define its empirical distribution in \mathcal{X} as \mu(x)=(1/n)\sum_{i}\delta(x-x_{i}) . q_{\theta}(x) denotes a generative model with a parameter \theta . In addition, the training of the model based on the data distribution \mu(x) is performed by solving \displaystyle\theta^{*} \displaystyle=\argmin_{\theta}D_{\text{KL}}(\mu(x)\|q_{\theta}(x)), (1) which represents the minimization of the Kullback-Leibler (KL) divergence: D_{\text{KL}}(\mu\|q_{\theta}):=\mathbb{E}_{\mu}[\log(\mu/q_{\theta})] . That is equivalent to the maximum log-likelihood training [ 21 ] . In VAEs, the model q_{\theta}(x) is defined by the marginal distribution of Q_{\theta}(x,z) as \displaystyle q_{\theta}(x):=\int dz\,Q_{\theta}(x,z)=\int dz\,Q_{\theta}(x|z)\pi(z). (2) Here, \pi(z) is a known distribution of a latent variable z\in\mathcal{Z} , which is called the prior 1 1 1 We often choose that \dim\mathcal{Z}\leq\dim\mathcal{X} and the prior is a normal distribution. ; Q_{\theta}(x|z) is a conditional distribution of x given z . Since this distribution carries information from the latent space \mathcal{Z} to the data space \mathcal{X} , it is called the decoder. In VAEs, instead of solving Eq. ( 1 ) directly, we consider the optimization problem of its upper bound by employing the data processing inequality 2 2 2 This inequality usually describes the relationships among mutual informations, but in this paper, we apply it for the KL divergences. (Appendix A ): \displaystyle D_{\text{KL}}(\mu(x)\|q_{\theta}(x))\leq D_{\text{KL}}(P_{\phi}(x,z)\|Q_{\theta}(x,z)). (3) That minimizes the KL divergence on the joint space \mathcal{X}\times\mathcal{Z} . Here, P_{\phi}(x,z) is the joint distribution extended from \mu(x) with a conditional distribution P_{\phi}(z|x) : \displaystyle P_{\phi}(x,z)=P_{\phi}(z|x)\mu(x). (4) In contrast to the decoder Q_{\theta}(x|z) , since P_{\phi}(z|x) sends the information from \mathcal{X} to \mathcal{Z} with the parameter \phi , it is called the encoder. The training of VAEs is to find the optimal parameters (\phi^{*},\theta^{*}) : \displaystyle(\phi^{*},\theta^{*})=\argmin_{\phi,\theta}D_{\text{KL}}(P_{\phi}(x,z)\|Q_{\theta}(x,z)). (5) By taking Eqs. ( 2 ) and ( 4 ) into account, the objective function of VAEs is derived as \displaystyle\begin{split}&D_{\text{KL}}(P_{\phi}(x,z)\|Q_{\theta}(x,z))=\mathbb{E}_{\mu(x)}[\log\mu(x)]\\ &+\mathbb{E}_{\mu(x)}[D_{\text{KL}}({P}_{\phi}(z|x)\|\pi(z))]-\mathbb{E}_{\mu(x){P}_{\phi}(z|x)}\!\left[\log Q_{\theta}(x|z)\right]\!.\end{split} (6) Its derivation is shown in Appendix B . The negative sign of the second and third terms is collectively referred to as the evidence lower bound [ 19 ] . In practical training, the encoder P_{\phi}(z|x) and the decoder Q_{\theta}(x|z) are usually modeled by a normal distribution whose mean and variance are given by neural networks (NNs). After training, we can generate new samples using the optimized parameter \theta^{*} as follows: (i) The latent variable z is sampled from a known prior \pi(z) . (ii) A new sample x can be generated by the optimal decoder Q_{\theta^{*}}(x|z) conditioned on z . Here, we note that \phi^{*} is not used in the generating process but is used for the training of \theta . To clarify the scheme of this training, we summarize the role of each parameter. On the one hand, training of \theta adjusts the model distribution q_{\theta} closer to the data distribution \mu , which is the primary goal of all generative models. On the other hand, training for \phi does not directly contribute to