Interpolating Discrete Diffusion Models with Controllable Resampling

Marcel Kollovieh, Sirine Ayadi, Stephan Günnemann · 2026

Read Interpolating Discrete Diffusion Models with Controllable Resampling by Marcel Kollovieh, Sirine Ayadi, Stephan Günnemann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17310.txt.

Introduction

Kollovieh, Ayadi, and Günnemann introduce IDDM, a discrete diffusion model that generates text and molecular graphs by partly resetting probability toward a prior distribution. The reset curbs errors that pile up when masked models unmask too early, while keeping the self-correction that uniform models offer. Its steps blend staying put, resampling, and moving toward the target, and tests show competitive results against leading discrete diffusion models.

Catalog record

CREATORMarcel Kollovieh, Sirine Ayadi, Stephan Günnemann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56565
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17310

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpolating Discrete Diffusion Models with Controllable Resampling Marcel Kollovieh, Sirine Ayadi, Stephan Günnemann Abstract Discrete diffusion models form a powerful class of generative models across diverse domains, including text and graphs. However, existing approaches face fundamental limitations. Masked diffusion models suffer from irreversible errors due to early unmasking, while uniform diffusion models, despite enabling self-correction, often yield low-quality samples due to their strong reliance on intermediate latent states. We introduce IDDM , an Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states. Central to IDDM is a controllable resampling mechanism that partially resets probability mass to the marginal distribution, mitigating error accumulation and enabling more effective token corrections. IDDM specifies a generative process whose transitions interpolate between staying at the current state, resampling from a prior, and flipping toward the target state, while enforcing marginal consistency and fully decoupling training from inference. We benchmark our model against state-of-the-art discrete diffusion models across molecular graph generation as well as text generation tasks, demonstrating competitive performance. † † affiliationnotice: Equal contribution 1 Introduction Discrete diffusion models ( Austin et al. 2021 ) have emerged as a powerful approach for generating structures across domains such as proteins ( Gruver et al. 2023 ; Nisonoff et al. 2024 ) , DNA sequences ( Sarkar et al. 2024 ) , and molecules ( Vignac et al. 2022 ; Campbell et al. 2022 ) , and are starting to catch up with autoregressive models in language generation ( Lou et al. 2023 ; Sahoo et al. 2025 ; Schiff et al. 2024 ) . As with their continuous counterparts, the core idea is to define Markov transitions that gradually corrupt the data, and then learn a model to denoise it by parametrizing the reverse process. Common forward processes include uniform transitions over the state space or masking, also known as absorbing-state diffusion, in which the state either remains unchanged or transitions to a mask token ( Austin et al. 2021 ; Sahoo et al. 2024 ; Shi et al. 2024 ) . While these frameworks work analogously to the continuous case, they introduce considerable limitations. Masked diffusion models, for example, do not support refinement: once a token is unmasked, it remains fixed throughout the entire generation, restricting the model to correct earlier mistakes and leading to error accumulation ( Sahoo et al. 2024 ; Wang et al. 2025 ; Shi et al. 2024 ; Boget 2025 ) . Uniform transitions, on the other hand, allow flips back to the prior, but they can still suffer from error accumulation ( Boget 2025 ) , and are typically outperformed by absorbing-state diffusion approaches ( Austin et al. 2021 ; Lou et al. 2023 ) . In this work, we propose Interpolating Discrete Diffusion Model ( IDDM ), a generative model for discrete data that derives transition probabilities from a different perspective. Instead of defining a Markovian forward transition to reach a terminal distribution, we directly define the reverse, i.e., the generative distribution. To this end, our transitions make use of three different actions: (i) staying in the current state, (ii) transitioning to the goal state, and (iii) resampling from the prior distribution, see Figure 1 . Figure 1 : Graphical model of IDDM. In each step, our model consists of three actions: (i) stay in the current state, (ii) flip to the goal state, and (iii) sample from the prior distribution. The model can be trained without the need to define a forward process. By constraining the marginal distribution to match an interpolation from prior to the data distribution, we obtain a family of posterior transitions. The posteriors interpolate between a fully local update without random flips and a state-independent update that discards the current state, enabling us to trade off preservation of the current knowledge and stochasticity. To demonstrate IDDM’s capability, we evaluate it on molecule generation and language tasks. Figure 2 : Overview of the generative process of IDDM. Initial samples are drawn from the prior distribution \mathbf{q}_{1} . In each denoising step \mathbf{x}_{s}\mid\mathbf{x}_{t} , our model constructs the posterior p_{\lambda}^{\theta}(\mathbf{x}_{s}\mid\mathbf{x}_{t}) . Increasing \lambda increases the stochasticity and state transitions by shifting probability mass from the current to the predicted and random states, as visualized in the bar plot. Contributions. Our contributions can be summarized as: • Interpolating reverse dynamics. We propose the Interpolating Discrete Diffusion Model (IDDM) , a simple discrete diffusion model consisting of three different actions, i.e., stay in the current state, transition to the goal state, resample from the prior. • Marginal-consistent posterior family. By enforcing that the marginals adhere to an interpolation between the prior and the data distribution, we derive a family of posterior transitions that trades off preserving the current state against injecting stochasticity. • Forward-process interpretation. We show that the resulting transitions correspond, in general, to a non-Markovian forward process. • Empirical evaluation. We demonstrate that our model achieves competitive performance on well-established benchmarks for molecule generation and language modeling. 2 Discrete Diffusion Models Notations. For clarity and ease of presentation, we use \mathbf{x} to denote a single discrete random variable taking values in \{1,\dots,K\} . In sequence modeling, \mathbf{x} represents a single token, while in graph modeling, it can represent a single node or a single edge in the graph. We denote the corresponding one-hot encoding by \mathbf{e}_{\mathbf{x}}\in\Delta^{K-1} , where \Delta^{K-1} is the simplex over K categories. We denote by \Cat(\cdot;\mathbf{y}) the categorical distribution with probability vector \mathbf{y}\in\Delta^{K-1} . Diffusion models learn a Markov chain that involves a forward process to perturb the data \mathbf{x}\sim\mathbf{q}_{0}(\mathbf{x}) to increasingly noisy latent variables \mathbf{x}_{t} , for t\in[0,1] , and a trained reverse process. The forward process p(\mathbf{x}_{t}\mid\mathbf{x}_{s}) is expressed under the Markovian assumption, and the induced marginal for every latent variable \mathbf{x}_{t} can be defined as an interpolation between the clean data and a noisy prior \mathbf{q}_{1} ( Sahoo et al. 2024 ) : p(\mathbf{x}_{t}\mid\mathbf{x})\;=\;\Cat(\mathbf{x}_{t};(1\;-\;\alpha_{t})\mathbf{q}_{1}\;+\;\alpha_{t}\mathbf{e}_{\mathbf{x}})\;, (1) where \alpha_{t}\in[0,1] is a strictly decreasing function in t , and unless stated otherwise, \mathbf{x}_{t} is represented in one-hot form. The parametrized reverse process p_{\theta}(\mathbf{x}_{s}\mid\mathbf{x}_{t},\mathbf{x}),s<t is trained to optimize a variational lower bound on the log-likelihood: \displaystyle\log\ \displaystyle p_{\theta}(\mathbf{x})\geq \displaystyle\mathbb{E}_{p(\mathbf{x}_{t(1)},\dots,\mathbf{x}_{t(T)}\mid\mathbf{x})}\left[\log\frac{p_{\theta}(\mathbf{x},\mathbf{x}_{t(1)},\dots,\mathbf{x}_{t(T)})}{p(\mathbf{x}_{t(1)},\dots,\mathbf{x}_{t(T)}\mid\mathbf{x})}\right], (2) where t(i)=i/T , T is the number of diffusion steps, and the reverse Markov process \displaystyle p_{\theta}(\mathbf{x},\mathbf{x}_{t(1:T)})= \displaystyle\qquad p(\mathbf{x}_{t(T)})\,\prod_{i=2}^{T}p_{\theta}(\mathbf{x}_{t(i-1)}\mid\mathbf{x}_{t(i)})\,p_{\theta}(\mathbf{x}\mid\mathbf{x}_{t(1)}), (3) gradually denoises the latent states towards the data distribution. There exist two popular variants for interpolating discrete diffusion models: uniform diffusion models with a stationary distribution \mathbf{q}_{1}=\bm{1}/K , where \bm{1} is a K- dimensional column vector of all ones, and masked diffusion models, also known as absorbing state diffusion, where the space is augmented with a special mask token, and the prior distribution is set to \mathbf{q}_{1}=\mathbf{m} . Marginal distributions over node and edge attributes as noise have proven to be more suitable for graph generation. Standard discrete diffusion models tightly couple the reverse process to a fixed forward corruption mechanism. The forward process is defined as a Markov chain p(\mathbf{x}_{t}\mid\mathbf{x}_{s}) , and the reverse transitions can be determined via Bayes’ rule: p(\mathbf{x}_{s}\mid\mathbf{x}_{t},\mathbf{x})=\frac{p(\mathbf{x}_{t}\mid\mathbf{x}_{s})p(\mathbf{x}_{s}\mid\mathbf{x})}{p(\mathbf{x}_{t}\mid\mathbf{x})}\;, where p(\mathbf{x}_{t}\mid\mathbf{x}_{s})=p(\mathbf{x}_{t}\mid\mathbf{x}_{s},\mathbf{x}) follows from the Markov property. As a result, the reverse dynamics are largely determined once the forward kernel and the noise schedule are fixed. This can limit the extent to which reverse updates

Read the complete text