Information-Theoretic Discrete Diffusion

Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No, [0.5em] Department of Artificial Intelligence, Yonsei University · 2025

Read Information-Theoretic Discrete Diffusion by Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No, [0.5em] Department of Artificial Intelligence, Yonsei University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24088.txt.

Introduction

Moongyu Jeon and three co-authors build a theory that links diffusion models for discrete data to information theory. Discrete diffusion adds noise step by step to data like text, then learns to remove it. The authors adapt a known identity from the continuous case to this discrete setting.

They derive two relations that tie mutual information between data and its noisy form to the best possible training losses. One covers general discrete diffusion, the other masked diffusion. Together they split the log likelihood of the data into a sum over time of best losses. The result shows common losses are not just bounds but exact parts of the likelihood.

Catalog record

CREATORMoongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No, [0.5em] Department of Artificial Intelligence, Yonsei University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24088

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Information-Theoretic Discrete Diffusion Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No, [0.5em] Department of Artificial Intelligence, Yonsei University Abstract We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results for the discrete setting. Specifically, we introduce the Information-Minimum Denoising Score Entropy (I-MDSE) relation, which links mutual information between data and its diffused version to the minimum denoising score entropy (DSE) loss. We extend this theory to masked diffusion and establish the Information-Minimum Denoising Cross-Entropy (I-MDCE) relation, connecting cross-entropy losses to mutual information in discrete masked processes. These results provide a time-integral decomposition of the log-likelihood of the data in terms of optimal score-based losses, showing that commonly used losses such as DSE and DCE are not merely variational bounds but tight and principled estimators of log-likelihood. The I-MDCE decomposition further enables practical extensions, including time-free formula, conditional likelihood estimation in prompt-response tasks, and coupled Monte Carlo estimation of likelihood ratios. Experiments on synthetic and real-world data confirm the accuracy, variance stability, and utility of our estimators. The code is publicly available at https://github.com/Dongjae0324/infodis . 1 Introduction Diffusion models have emerged as a powerful framework for generative modeling, enabling state-of-the-art performance in continuous domains such as image and audio generation ( Sohl-Dickstein et al. 2015 ; Ho et al. 2020 ; Chen et al. 2021 ; Kong et al. 2021 ; Saharia et al. 2022 ) . Central to these models is the idea of gradually corrupting data through a forward noising process, and learning to reverse this corruption via score-based loss ( Hyvärinen and Dayan 2005 ; Vincent 2011 ; Song and Ermon 2019 ; Song et al. 2021a ; Song et al. 2021b ) . Recent works have extended diffusion models to discrete domains, proposing models designed for categorical data ( Hoogeboom et al. 2021 ; Austin et al. 2021 ; Campbell et al. 2022 ; Meng et al. 2022 ; Sun et al. 2023 ; Lou et al. 2024 ; Sahoo et al. 2024 ; Shi et al. 2024 ) . These models offer a promising alternative to traditional autoregressive approaches ( Radford et al. 2018 ; Radford et al. 2019 ; Brown et al. 2020 ) , particularly for sequence modeling tasks that involve text and other symbolic structures ( Li et al. 2022 ; Nie et al. 2025 ) . Continuous diffusion models benefit from a well-established information-theoretic foundation ( Kong et al. 2023 ; Kong et al. 2024 ) . In the Gaussian setting, the I-MMSE identity ( Guo et al. 2005 ; Venkat and Weissman 2012 ) connects the mutual information between clean and noisy variables to the minimum mean squared error (MMSE), offering both theoretical insight and a basis for likelihood estimation. A pointwise generalization of this identity yields closed-form decompositions of the data log-likelihood in terms of estimation losses ( Kong et al. 2023 ) . However, the discrete case has not yet been investigated from an information-theoretic perspective. In this work, we extend these ideas to discrete diffusion models, developing an information-theoretic framework that rigorously characterizes the relationship between mutual information and score-based loss in discrete settings. We first establish the Information–Minimum Denoising Score Entropy (I-MDSE) identity, which connects mutual information decay in the forward process to the minimum of the denoising score entropy (DSE) loss. This leads to a closed-form decomposition of the negative log-likelihood (NLL) into a time-integral of the minimum DSE, showing that the DSE loss, previously viewed as a variational bound, actually constitutes a principled estimator for likelihood estimation. We then turn to masked (absorbing) diffusion models, where we formulate the pointwise denoising cross-entropy (DCE) loss and prove its equivalence to the DSE loss under certain time reparameterization. Leveraging this equivalence, we derive the Information–Minimum Denoising Cross-Entropy (I-MDCE) identity, which mirrors the I-MDSE relation but in the masked setting. The I-MDCE identity provides a parallel time-integral decomposition of the NLL along the minimum DCE trajectory, revealing that the DCE loss, like DSE, serves as a theoretically grounded training objective that enables exact estimation of the data likelihood. Building on this decomposition via I-MDCE, we derive a time-free reformulation of the log-likelihood, expressed as an expectation over randomly selected unmasked token subsets. This formulation enables efficient Monte Carlo estimation of the NLL without diffusion-time integration and extends naturally to conditional likelihoods of the form p_{0}(\mathbf{x}^{\text{target}}|\mathbf{x}^{\text{context}}) , allowing estimation in structured generative settings such as prompt–response modeling. Furthermore, by coupling the sampling paths of two sequences, our framework provides a principled Monte Carlo estimator for likelihood ratios, achieving unbiased and low-variance estimation compared to independently sampled baselines. We validate our framework through experiments on both synthetic and real-world datasets. First, using synthetic datasets with known ground-truth distributions, we show that our estimators accurately recover both unconditional and conditional log-likelihoods. Next, we verify the variance reduction properties of our time-free likelihood estimator and the coupled likelihood ratio estimator against their respective baselines. Finally, we demonstrate the practical utility of our approach through auditing experiments on real-world data, where conditional likelihood estimates detect out-of-distribution inputs and reveal distributional shifts in LLaDA ( Nie et al. 2025 ) . These results confirm that our information-theoretic framework not only offers theoretical insight but also enables accurate and interpretable likelihood estimation in discrete generative models. 2 Preliminaries 2.1 Discrete Diffusion Models and Score Matching Given data x_{0}\sim p_{0} , the forward diffusion process is modeled as a continuous-time Markov chain (CTMC), governed by a linear ODE ( Anderson 2012 ; Campbell et al. 2022 ) : \frac{dp_{t}}{dt}=Q_{t}p_{t},\quad p_{0}=p_{\text{data}} (1) where Q_{t}\in\mathbb{R}^{N\times N} is the time-dependent transition rate matrix, with N=|\mathcal{X}| possible states. As t\rightarrow\infty , the marginal p_{t} converges to a stationary distribution \pi . For tractability, it is common to assume a factored form Q_{t}=\sigma(t)Q , where Q is a fixed matrix and \sigma(t) is a positive scalar function. The reverse process is also governed by another CTMC, described by the following ODE ( Kelly 1980 ; Sun et al. 2023 ; Lou et al. 2024 ) : \frac{dp_{T-t}}{dt}=\overline{Q}_{T-t}p_{T-t},\quad\overline{Q}_{t}(y,x)=\begin{cases}\frac{p_{t}(y)}{p_{t}(x)}Q_{t}(x,y)&\text{if }x\neq y,\\ -\sum_{\tilde{y}\neq x}\overline{Q}_{t}(\tilde{y},x)&\text{if }x=y.\end{cases} (2) To simulate the reverse process, one typically initializes at p_{T}^{\theta}=\pi and replaces the marginal ratio \frac{p_{t}(y)}{p_{t}(x)} in Eq. 2 with a learned approximation, yielding a parameterized family \{p_{t}^{\theta}\}_{t=T}^{0} . Early methods ( Austin et al. 2021 ; Campbell et al. 2022 ) modeled the reverse conditional p_{0|t} directly, but suffered from combinatorial scalability issues. Meng et al. 2022 instead used an \ell^{2} regression loss to approximate the marginal ratio \frac{p_{t}(y)}{p_{t}(x)} , which proved unstable in practice. Subsequently, Lou et al. 2024 introduced the denoising score entropy (DSE) loss, resolving these scalability and stability issues. Specifically, they use a score network s^{\theta}:\mathcal{X}\times[0,T]\rightarrow\mathbb{R}^{N} to estimate the marginal ratio, where each output s^{\theta}(x,t)_{y} corresponds to \frac{p_{t}(y)}{p_{t}(x)} . The network is trained by minimizing the DSE loss, defined pointwise as: \ell_{\mathrm{DSE}}(x_{0},x,t,s_{t}):=\sum_{y\neq x}Q_{t}(x,y)\left(s_{t}(x)_{y}-\frac{p_{t|0}(y|x_{0})}{p_{t|0}(x|x_{0})}\log s_{t}(x)_{y}+K\!\left(\frac{p_{t|0}(y|x_{0})}{p_{t|0}(x|x_{0})}\right)\right), (3) where K(a)=a(\log a-1) . This loss is minimized when the score network recovers the true score s_{t}^{\star} , where s_{t}^{\star}(x)_{y}=\frac{p_{t}(y)}{p_{t}(x)} : s_{t}^{\star}=\mathop{\mathrm{arg\,min}}_{s_{t}}\,\mathbb{E}_{p(x_{0},x_{t})}\left[\ell_{\mathrm{DSE}}(x_{0},x_{t},t,s_{t})\right]. (4) Aggregating this loss over time yields the time-integrated DSE training objective: \mathcal{L}_{\mathrm{DSE}}^{T}(x_{0}):=\int_{0}^{T}\mathbb{E}_{p_{t|0}(x_{t}|x_{0})}\left[\ell_{\mathrm{DSE}}(x_{0},x_{t},t,s^{\theta}_{t})\right]dt, where s^{\theta}_{t}=s^{\theta}(\cdot,t) is the learned score at time t . Importantly, this

Read the complete text