Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling
Tianyu Xie, Shuchen Xue, Zijin Feng, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Cheng Zhang · 2025
Read Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling by Tianyu Xie, Shuchen Xue, Zijin Feng, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Cheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17384.txt.
Introduction
Xie and co-authors propose in 2025 a framework that strengthens discrete diffusion models by adding latent variable modeling. Masked diffusion models build data by unmasking parts step by step, with a good speed and quality trade. With few steps, weak links among dimensions hurt quality.
Their method, VADD, adds a helper recognition model so correlations among dimensions are captured in a hidden space. Training stays stable through a variational bound, and generation keeps the speed of masked models. Tests on toy data, images, and text beat masked baselines at low step counts. The gain is largest where past models degrade most.
Catalog record
| CREATOR | Tianyu Xie, Shuchen Xue, Zijin Feng, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Cheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling Tianyu Xie, Shuchen Xue, Zijin Feng, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Cheng Zhang Abstract Discrete diffusion models have recently shown great promise for modeling complex discrete data, with masked diffusion models (MDMs) offering a compelling trade-off between quality and generation speed. MDMs denoise by progressively unmasking multiple dimensions from an all-masked input, but their performance can degrade when using few denoising steps due to limited modeling of inter-dimensional dependencies. In this paper, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion with latent variable modeling to implicitly capture correlations among dimensions. By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds maximization and amortized inference over the training set. Our approach retains the efficiency of traditional MDMs while significantly improving sample quality, especially when the number of denoising steps is small. Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines. 1 Introduction Diffusion models ( Ho et al. 2020 ; Song et al. 2020 ) have shown their remarkable successes in generative modeling of continuous objects, e.g., image ( Rombach et al. 2021 ; Ramesh et al. 2022 ) , audio ( Kong et al. 2021 ; Liu et al. 2023 ) , and video ( Ho et al. 2022 ; Blattmann et al. 2023 ) . Recently, diffusion models have also been extended to the discrete state space ( Austin et al. 2021 ; Campbell et al. 2022 ; Sun et al. 2023 ; Lou et al. 2024 ) , achieving competitive or even superior performance to autoregressive models in tasks such as Sudoku solving and code generation. A notable example is the masked diffusion model (MDM) ( Sahoo et al. 2024 ; Shi et al. 2024 ; Ou et al. 2025 ) , which works by progressively masking the dimensions of the data point towards an all-masked state in the forward process, and gradually unmasking (i.e., predicting the distributions of) multiple dimensions simultaneously in the backward process. This parallel prediction capability allows MDMs to generate samples much faster than autoregressive models, which typically predict one dimension at a time. Despite the inference efficiency of MDMs, their denoising distribution at each backward step is typically modeled as a product of independent categorical distributions across dimensions which may fail to capture complex inter-dimensional dependencies commonly present in real-world data. This issue becomes particularly pronounced when using a small number of denoising steps, where many dimensions must be unmasked simultaneously, amplifying the impact of independence assumptions. While recent efforts ( Xu et al. 2024 ; Liu et al. 2024 ) have attempted to mitigate this limitation, they require inner-loop sampling steps under guidance from pre-trained autoregressive models or correlation models, introducing additional computation costs. In this work, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion models by incorporating a latent variable structure into the denoising distribution. This structure enables the model to implicitly capture inter-dimensional correlations, thereby improving its approximation capacity. To address the intractability of the resulting marginal distributions, we adopt the variational autoencoding framework ( Kingma & Welling 2014 ) , jointly optimizing the denoising model and an auxiliary recognition model via a variational lower bound. Beyond standard latent-variable sampling, we introduce a consistency sampler that fixes the latent variable throughout the entire backward process, empirically leading to higher-quality samples. We further design a specialized transformer-based architecture for VADD that retains the fast inference efficiency of traditional MDMs. Through experiments on 2D toy datasets, pixel-level image generation, and text generation, we show that VADD consistently outperforms MDM baselines—particularly in terms of sample quality—highlighting the benefits of modeling dimensional dependencies. 2 Background Masked diffusion models Let \bm{x}_{0} be a categorical sample with N dimensions and \bm{x}_{0}^{i}\in\{1,\ldots,V\} be the i -th dimension of \bm{x}_{0} . We further augment the state space with a special mask state \texttt{[M]}=V+1 . Let \delta_{a} denote a one-hot vector of length V+1 with the value 1 at position a . The forward process of masked diffusion models (MDMs) is defined as q(\bm{x}_{t}|\bm{x}_{s})=\prod_{i=1}^{N}q(\bm{x}_{t}^{i}|\bm{x}_{s}^{i}) and q(\bm{x}_{t}^{i}|\bm{x}_{s}^{i})=\mathrm{Cat}(\bm{x}_{t}^{i};\frac{\alpha_{t}}{\alpha_{s}}\delta_{\bm{x}_{s}^{i}}+\frac{\alpha_{s}-\alpha_{t}}{\alpha_{s}}\delta_{\texttt{[M]}}) for s<t , where the mask schedule \alpha_{t} is a strictly decreasing function in t\in[0,1] with \alpha_{0}\approx 1 and \alpha_{1}\approx 0 . Particularly, we have q(\bm{x}_{t}^{i}|\bm{x}_{0}^{i})=\mathrm{Cat}(\bm{x}_{t}^{i};\alpha_{t}\delta_{\bm{x}_{s}^{i}}+(1-\alpha_{t})\delta_{\texttt{[M]}}) that will diffuse to an all-masked state \texttt{[M]}^{N} at t=1 . Given \bm{x}_{0} , the posterior distribution of \bm{x}_{s} takes the form q(\bm{x}_{s}|\bm{x}_{t},\bm{x}_{0})=\prod_{i=1}^{N}q(\bm{x}_{s}^{i}|\bm{x}_{t}^{i},\bm{x}_{0}^{i}) where q(\bm{x}_{s}^{i}|\bm{x}_{t}^{i},\bm{x}_{0}^{i})=\left\{\begin{array}[]{ll}\mathrm{Cat}(\bm{x}_{s}^{i};\delta_{\bm{x}_{t}^{i}}),&\bm{x}_{t}^{i}\neq\texttt{[M]},\\ \mathrm{Cat}\left(\bm{x}_{s}^{i};\frac{1-\alpha_{s}}{1-\alpha_{t}}\delta_{\texttt{[M]}}+\frac{\alpha_{s}-\alpha_{t}}{1-\alpha_{t}}\delta_{\bm{x}_{0}^{i}}\right),&\bm{x}_{t}^{i}=\texttt{[M]}.\\ \end{array}\right. (1) Equation ( 1 ) inspires parametrizing the backward transitions as p_{\bm{\theta}}(\bm{x}_{s}|\bm{x}_{t})=q(\bm{x}_{s}|\bm{x}_{t},\bm{x}_{0}=\bm{\mu}_{\bm{\theta}}(\bm{x}_{t},t))=\prod_{i=1}^{N}q(\bm{x}_{s}^{i}|\bm{x}_{t}^{i},\bm{x}_{0}^{i}=\bm{\mu}^{i}_{\bm{\theta}}(\bm{x}_{t},t)), (2) where the denoising distribution \bm{\mu}_{\bm{\theta}}(\bm{x}_{t},t)\in\mathbb{R}^{N\times(V+1)} with the constraint \sum_{j=1}^{V}\bm{\mu}^{i,j}_{\bm{\theta}}(\bm{x}_{t},t)=1 and \bm{\mu}^{i,\texttt{[M]}}_{\bm{\theta}}(\bm{x}_{t},t)=0 is expected to match the posterior of the clean data q(\bm{x}_{0}|\bm{x}_{t}) . The \bm{\mu}_{\bm{\theta}}(\bm{x}_{t},t) is explicit and often trained by maximizing the continuous-time evidence lower bound (ELBO) ( Shi et al. 2024 ; Sahoo et al. 2024 ) \mathcal{L}(\bm{x}_{0};\bm{\theta})=\int_{0}^{1}\mathbb{E}_{q(\bm{x}_{t}|\bm{x}_{0})}\frac{-\alpha_{t}^{\prime}}{1-\alpha_{t}}\log p_{\bm{\theta}}(\bm{x}_{0}|\bm{x}_{t})\mathrm{d}t\leq\log p_{\bm{\theta}}(\bm{x}_{0}) (3) for all \bm{x}_{0} in the training set, closely related to any-order autoregressive models ( Ou et al. 2025 ) . Note that the backward transition p_{\bm{\theta}}(\bm{x}_{s}|\bm{x}_{t}) in equation ( 2 ) is factorizable over N dimensions which helps reducing the modeling complexity of state space. However, this conditional independence structure fails to capture the inter-dimensional correlations and would inevitably introduce cumulative approximation errors. As the number of dimensions unmasked simultaneously during a backward transition is proportional to (\alpha_{s}-\alpha_{t}) , this drawback can be more severe under a large step size. Variational autoencoders Consider the generative modeling task on a dataset \{\bm{y}_{1},\ldots,\bm{y}_{M}\} , where \bm{y}_{i} is an N -dimensional continuous or discrete variable. The variational autoencoder (VAE) ( Kingma & Welling 2014 ) assumes a generative model p_{\bm{\theta}}(\bm{y},\bm{z})=p_{\bm{\theta}}(\bm{y}|\bm{z})p(\bm{z}) , where \bm{z}\in\mathbb{R}^{d} is a latent variable with a prior distribution p(\bm{z}) , and a recognition model r_{\bm{\phi}}(\bm{z}|\bm{y}) as an approximation for the intractable posterior p_{\bm{\theta}}(\bm{z}|\bm{y}) . The generative model and recognition model can be jointly learned by maximizing the following evidence lower bound (ELBO) L(\bm{y};\bm{\theta},\bm{\phi})=\mathbb{E}_{r_{\bm{\phi}}(\bm{z}|\bm{y})}\log\left(\frac{p_{\bm{\theta}}(\bm{y},\bm{z})}{r_{\bm{\phi}}(\bm{z}|\bm{y})}\right)=\log p_{\bm{\theta}}(\bm{y})-D_{\mathrm{KL}}\left(r_{\bm{\phi}}(\bm{z}|\bm{y})\|p_{\bm{\theta}}(\bm{z}|\bm{y})\right)\leq\log p_{\bm{\theta}}(\bm{y}) (4) for all data points \{\bm{y}_{1},\ldots,\bm{y}_{M}\} . Here, p_{\bm{\theta}}(\bm{y})=\int_{\mathbb{R}^{d}}p_{\bm{\theta}}(\bm{y},\bm{z})\mathrm{d}\bm{z} is the marginal likelihood of \bm{y} and D_{\mathrm{KL}} is the Kullback-Leibler (KL) divergence. A mean-field structure is often assumed for the conditional distribution p_{\bm{\theta}}(\bm{y}|\bm{z}) , i.e., the dimensions of \bm{y} are independently distributed conditioned on \bm{z} . As a comprehensible example, Kingma & Welling 2014 assumes the following distribution over a continuous image sample \bm{y} p_{\bm{\theta}}(\bm{y}|\bm{z})=\mathcal{N}\left(\bm{m}_{\bm{\theta}}(\bm{z}),\mathrm{diag}\{\bm{\sigma}^{2}_{\bm{\theta}}(\bm{z})\}\right), (5) where \bm{m}_{\bm{\theta}},\bm{\sigma}_{\bm{\theta}}:\mathbb{R}^{d}\to\mathbb{R}^{N} are two learnable neural networks. As a latent variable model, the marginal distribution p_{\bm{\theta}}(\bm{y}) is still capable of modeling the complex dimensional correlations by integrating out the latent variable \bm{z} . 3 Methodology In this section, we propose Variational Autoencoding Discrete Diffusion (VADD), which extends the dimension-independent denoising distribution in MDMs by introducing a latent variable structure. This design enables VADD to capture complex dependencies across dimensions. Both the denoising distribution and an auxiliary recognition model are jointly optimized under the variational autoencoding framework. The basic idea of VADD can also be transferred to discrete diffusion models with other noise schedules (e.g., uniform distribution as the noise), which is an interesting future direction. 3.1 Denoising model in VADD Instead of parametrizing the backward transition p_{\bm{\theta}}(\bm{x}_{s}|\bm{x}_{t}) as an explicit distribution, we define it as a latent variable model: p_{\bm{\theta}}(\bm{x}_{s}|\bm{x}_{t})=\int_{\mathbb{R}^{d}}p_{\bm{\theta}}(\bm{x}_{s}|\bm{x}_{t},\bm{z})p(\bm{z})\mathrm{d}\bm{z}, (6) where p(\bm{z}) is the prior distribution