The Diffusion Encoder

Akhil Premkumar, Sarah Lucioni · 2026

Read The Diffusion Encoder by Akhil Premkumar, Sarah Lucioni (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13399.txt.

Introduction

Akhil Premkumar and Sarah Lucioni construct a diffusion encoder by replacing a variational autoencoder's encoder with a diffusion model. In that standard model, encoder and decoder jointly agree on a latent code. The reparameterisation trick that makes training easy restricts the encoder to simple distributions. Swapping in a diffusion model breaks that restriction. Yet encoder and decoder then tend to update their latent estimates in opposing directions. The authors solve this with alternating training inspired by expectation-maximisation, which syncs the two sides.

Catalog record

CREATORAkhil Premkumar, Sarah Lucioni
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81492
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13399

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Diffusion Encoder Akhil Premkumar, Sarah Lucioni Abstract We construct a new kind of encoder, leveraging the expressive power of diffusion models. In a traditional variational autoencoder, the encoder and decoder jointly negotiate a latent representation of the input. This is made possible by the reparameterization trick, which simplifies training at the cost of restricting the encoder to a simple family of distributions. Replacing this encoder with a diffusion model requires rethinking how the decoder pressure can be transmitted back to the encoder, given that they tend to update their internal estimates of the latent in opposing directions. We solve this problem with an alternating training scheme, inspired by the expectation-maximization algorithm. Our method enables more reliable synchronization between encoder and decoder, while preserving the simple and efficient training objective of standard diffusion models. 1 Introduction Can a diffusion model be used as an encoder? After all, diffusion models are highly expressive probabilistic models, which makes them a natural candidate for parameterizing complex distributions over latent variables. The challenge is that these models are typically applied to cases where the target distribution is fixed, whereas a latent distribution must necessarily evolve during training as the encoder and decoder negotiate an optimal representation of the input signal. In variational autoencoders (VAEs) [ 19 ] , the encoder is a Gaussian distribution with a diagonal covariance. Although this choice limits representational flexibility, it enables joint training of the encoder and decoder through the reparameterization trick. That is, it is straightforward to express the latents in terms of the encoder parameters, which allows the latter to be directly coupled to the decoder pressure. The same is not true for a diffusion model, since the latents are generated through many more iterations over the underlying score network, which makes their functional dependence on the network parameters more complicated [ 3 ] . Hence, we need to resort to a training algorithm that alternates between encoder and decoder updates, similar to techniques commonly used in maximum entropy reinforcement learning [ 12 , 25 ] . Sacrificing the convenience of joint training may still be worthwhile if it yields a more flexible latent representation. The limited expressiveness of a rigid encoder family can hinder the autoencoder’s effectiveness in the following ways: (1) the optimal encoder may not lie within the chosen family, (2) an overly smooth encoder may map semantically distinct inputs to overlapping neighborhoods in latent space, and (3) an inexpressive encoder can throttle the reconstruction fidelity of the decoder [ 5 ] . These effects are discussed in greater detail in section 2 . Part of the expressive power of a diffusion model comes from its iterative generative process, which allows the model to gradually build up complex internal correlations in the generated sample (see section 3 ). Normalizing flows [ 34 ] and neural ODEs [ 4 ] also share this feature, but diffusion models benefit from a considerably simpler and more robust training objective (see section 5 ). Flow matching models [ 27 ] also inherit this property. However, diffusion models remain differentiated by their firm grounding in information theory, specifically, their ability to efficiently estimate quantities like mutual information and neural entropy [ 11 , 21 , 22 , 32 ] , which are otherwise difficult to compute in high-dimensions. This is particularly relevant to autoencoders, which are, at their core, realizations of an information bottleneck [ 39 , 35 , 1 ] . In this paper, we develop a framework to leverage a diffusion model as an encoder. The key innovation is a training scheme that updates the encoder and decoder in alternation, while keeping them synchronized with one another (see section 4 ). This procedure is directly inspired by the expectation-maximization algorithm [ 6 ] , where the E-step updates a diffusion model with samples from the posterior, and the M-step updates the decoder. The posterior samples are generated from a short Langevin chain that explores the decoder landscape, while an affine drift term keeps the samples regularized. Lifting the regularization pressure from the encoder side subdues the tendency for the encoder and decoder to fall out of alignment with one another during their independent updates. The diffusion encoder takes on a more passive role in this arrangement, which means it can be trained with the standard, well-established denoising objective. Figure 1 : Input/reconstructed images (top/bottom) from a diffusion encoder + convolutional decoder. Finally, it is important to be clear about the scope of this work. Our goal is not to surpass the state-of-the-art in encoder design; over a decade of refinement has made the Gaussian encoder a highly competitive baseline. Rather, we seek to motivate a new application of diffusion models that draws on their representational capacity and training simplicity [ 15 , 8 , 37 ] . Therefore, the experiments in section 6 serve to demonstrate proof of concept—showing that the proposed setup trains stably and produces coherent reconstructions—rather than optimizing reconstruction quality at scale. We will be transparent about the limitations of our framework as they arise, and suggest potential avenues for improvement throughout. We hope these ideas spur further research on the diffusion encoder. 2 Information Bottlenecks and Variational Autoencoders We begin with an information-theoretic view of unsupervised representation learning, starting from the information bottleneck principle [ 39 , 35 ] . Consider two random variables {\bm{X}} and {\bm{Y}} that are correlated with one another (see appendix F for notation). We want to predict {\bm{Y}} given some value of {\bm{X}} . This can be done by extracting from {\bm{X}} only the information that is relevant to {\bm{Y}} . Such information can be encoded in a third random variable {\bm{Z}} , called the latent , which compresses from {\bm{X}} the details pertinent to {\bm{Y}} . The optimal assignment from {\bm{X}} to {\bm{Z}} is obtained by finding the encoder q({\bm{z}}|{\bm{x}}) that solves (with \gamma>0 ) \min_{q({\bm{z}}|{\bm{x}})}I({\bm{X}};{\bm{Z}})-\gamma^{-1}I({\bm{Z}};{\bm{Y}}). (1) The mutual information (MI) between two random variables quantifies how much we learn about the value of one of them, given a measurement of the other (see App. A of [ 33 ] for a review). Minimizing I({\bm{X}};{\bm{Z}}) while maximizing I({\bm{Z}};{\bm{Y}}) squeezes into {\bm{Z}} only the {\bm{Y}} -specific content of {\bm{X}} . Implementing eq. 1 in practice requires estimating the MI terms, which is challenging when the random variables are high-dimensional. One workaround is to choose a simple form of the encoder, q_{\bm{\phi}}({\bm{z}}|{\bm{x}})={\mathcal{N}}({\bm{z}};\mu_{\bm{\phi}}({\bm{x}}),\textrm{diag}(\sigma_{\bm{\phi}}({\bm{x}})^{2})), (2) and the objective \min_{{\bm{\phi}},{\bm{\psi}}}\mathbb{E}_{{\bm{X}},{\bm{Y}}}\mathbb{E}_{q_{\bm{\phi}}({\bm{z}}|{\bm{x}})}[-\log{p}_{\bm{\psi}}({\bm{y}}|{\bm{z}})]+\gamma\mathbb{E}_{\bm{X}}{D_{\textrm{KL}}}(q_{\bm{\phi}}({\bm{z}}|{\bm{x}})\,\|\,p({\bm{z}})){\,=:\,}\min_{{\bm{\phi}},{\bm{\psi}}}\mathbb{E}_{{\bm{X}},{\bm{Y}}}{\mathcal{L}}_{\rm VI}. (3) The first term is the reconstruction term, which also trains the decoder {p}_{\bm{\psi}}({\bm{x}}|{\bm{z}}) . It follows from \min_{{\bm{\phi}},{\bm{\psi}}}-I({\bm{Z}};{\bm{Y}})=-S({\bm{Y}})+\min_{{\bm{\phi}},{\bm{\psi}}}S({\bm{Y}}|{\bm{Z}})\equiv\min_{{\bm{\phi}},{\bm{\psi}}}\mathbb{E}_{{\bm{Y}},{\bm{Z}}}[-\log{p}_{\bm{\psi}}({\bm{y}}|{\bm{z}})],{\,=:\,}\min_{{\bm{\phi}},{\bm{\psi}}}{\mathcal{L}}_{\rm rec} (4) since the entropy of {\bm{Y}} is fixed by the dataset, and is independent of the parameters ({\bm{\phi}},{\bm{\psi}}) . If we choose {\bm{Y}}={\bm{X}} , we recognize eq. 3 as the negative of the evidence lower bound (ELBO) of a variational auto encoder [ 19 ] (see appendix C ). The second term in eq. 3 , called the regularization term, incentivizes the encoder to be as uninformative as possible, by pushing q_{\bm{\phi}}({\bm{z}}|{\bm{x}}) to be as close to the prior p({\bm{z}})\equiv{\mathcal{N}}(0,I) . It is straightforward to show that {\mathcal{L}}_{\rm reg}{\,:=\,}\mathbb{E}_{\bm{X}}{D_{\textrm{KL}}}(q_{\bm{\phi}}({\bm{z}}|{\bm{x}})\,\|\,p({\bm{z}}))=I({\bm{Z}};{\bm{X}})+{D_{\textrm{KL}}}(q_{\bm{\phi}}({\bm{z}})\|p({\bm{z}})). (5) Intuitively, the aggregate q_{\bm{\phi}}({\bm{z}}){\,:=\,}\mathbb{E}_{\bm{X}}[q_{\bm{\phi}}({\bm{z}}|{\bm{x}})] is the distribution of all available ‘latent codes,’ and q_{\bm{\phi}}({\bm{z}}|{\bm{x}}) is concentrated on the subset that encodes

Read the complete text