Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes

Bocheng Li, Zhujin Gao, Linli Xu · 2025

Read Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes by Bocheng Li, Zhujin Gao, Linli Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22165.txt.

Introduction

Text diffusion models no longer need to choose between two kinds, in a 2025 paper by Bocheng Li, Zhujin Gao, and Linli Xu. Discrete models corrupt each token on its own schedule but control coarsely; continuous models control finely but force every token to march in step.

Their model, NeoDiff, runs a Poisson diffusion process that lets tokens move at their own pace with fine-grained noise, and a time predictor that adjusts each token's denoising by its meaning. An optimized inference schedule tightens noise control further. The design unifies the two families in one theory, and experiments show stronger generation than either parent approach. The work gives text diffusion the flexibility its image cousins already enjoy.

Catalog record

CREATORBocheng Li, Zhujin Gao, Linli Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22165

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes Bocheng Li, Zhujin Gao, Linli Xu Abstract Diffusion models have emerged as a promising approach for text generation, with recent works falling into two main categories: discrete and continuous diffusion models. Discrete diffusion models apply token corruption independently using categorical distributions, allowing for different diffusion progress across tokens but lacking fine-grained control. Continuous diffusion models map tokens to continuous spaces and apply fine-grained noise, but the diffusion progress is uniform across tokens, limiting their ability to capture semantic nuances. To address these limitations, we propose N on-simultan e ous C o ntinuous Diff usion Models (NeoDiff), a novel diffusion model that integrates the strengths of both discrete and continuous approaches. NeoDiff introduces a Poisson diffusion process for the forward process, enabling a flexible and fine-grained noising paradigm, and employs a time predictor for the reverse process to adaptively modulate the denoising progress based on token semantics. Furthermore, NeoDiff utilizes an optimized schedule for inference to ensure more precise noise control and improved performance. Our approach unifies the theories of discrete and continuous diffusion models, offering a more principled and effective framework for text generation. Experimental results on several text generation tasks demonstrate NeoDiff’s superior performance compared to baselines of non-autoregressive continuous and discrete diffusion models, iterative-based methods and autoregressive diffusion-based methods. These results highlight NeoDiff’s potential as a powerful tool for generating high-quality text and advancing the field of diffusion-based text generation. 1 Introduction Diffusion models have demonstrated remarkable success in generating high-quality samples in various domains, including vision ( Dhariwal and Nichol, 2021 ; Nichol and Dhariwal, 2021 ; Ho and Salimans, 2021 ; Rombach et al., 2022 ) and audio ( Chen et al., 2020 ; Kong et al., 2020 ) . Inspired by their achievements, there has been a growing interest in applying diffusion models to text generation tasks ( Li et al., 2022 ; Gong et al., 2022 ; Gao et al., 2024 ; Zheng et al., 2023 ) . The core idea behind diffusion models is to corrupt the data through a forward process and then learn to reverse this process to generate new samples. In text generation, existing diffusion models can be broadly categorized into two classes: discrete and continuous diffusion models. Discrete diffusion models treat tokens as discrete random variables and perform state transitions independently for each token using a categorical distribution. While straightforward, this approach fails to capture the continuous and fine-grained nature of language, limiting the potential benefits of multi-step generation. Continuous diffusion models, on the other hand, operate in a continuous space by mapping tokens to continuous representations, enabling more fine-grained perturbations. However, these models typically apply diffusion at the sentence level, resulting in uniform noise levels across all tokens within a sentence, restricting the model’s ability to leverage contextual information and recover tokens with varying noise levels based on the surrounding context ( Chen et al., 2023 ; Wu et al., 2024 ) . Figure 1: Comparison of the noising paradigms employed by Non-simultaneous Continuous Diffusion and two other diffusion models. The color intensity on the text tokens represents the token-level noising progress (intrinsic time \tau ). Discrete diffusion applies an independent but coarse-grained noising paradigm to each token within a sentence. In contrast, continuous diffusion utilizes a fine-grained noising schedule but applies it uniformly across all tokens. NeoDiff distinguishes itself by assigning an independent, fine-grained intrinsic time \tau to each token, with finer noising schedule in extrinsic time t . To address these limitations, we propose integrating the complementary strengths of discrete and continuous diffusion approaches, enabling fine-grained noise control at the token level. This unified approach aims to provide precise token-level control while maintaining continuous-valued noise distributions, which is absent in existing frameworks. While recent text diffusion models ( Han et al., 2023 ; Gong et al., 2023 ; Wu et al., 2024 ) have made advances, they do not fully address this requirement, necessitating a unified theoretical framework that bridges discrete and continuous diffusion paradigms through a carefully designed forward process. Furthermore, we observe that existing approaches primarily focus on enhancing the forward process, overlooking the inherent varying difficulties in denoising different tokens and the impact of generation context. In analyzing the reverse process, we recognize that tokens with lower noise levels can guide the recovery of more heavily corrupted tokens, thereby enhancing the overall text generation quality. In response to these challenges, we present N on-simultan e ous C o ntinuous Diff usion Models (NeoDiff), which unifies discrete and continuous diffusion models through a bi-temporal framework. The key insight is to generalize the time variable in previous diffusion models into extrinsic time t , representing the diffusion progress of the entire sentence, and intrinsic time \tau , tracking the diffusion progress of each individual token. This generalization enables us to introduce a novel Poisson process as the forward process, seamlessly integrating the flexibility of discrete noise with the fine granularity of continuous noise. An overview of this noising paradigm is illustrated in Figure 1 . To optimize the reverse process, we develop a context-aware time predictor that estimates the intrinsic time \tau using an adaptive modulation function to guide the denoising process. The extrinsic time schedule is further calibrated through Bayesian optimization, providing precise control over the noise distribution. NeoDiff achieves a fine-grained, improved diffusion process in both forward and reverse directions, naturally overcoming the constraints of previous discrete and continuous diffusion models, and exhibiting superior generation quality. We evaluate NeoDiff on a diverse set of NLP tasks, including machine translation, paraphrasing, text simplification, and question generation. NeoDiff consistently outperforms previous non-autoregressive diffusion-based and iteration-based methods, as well as autoregressive diffusion baselines. Specifically, our contributions can be summarized as follows: • We introduce NeoDiff, a unified theoretical framework that combines the advantages of discrete and continuous noise, generalizing and unifying existing text diffusion models. • We propose the Poisson diffusion process as the forward process, enabling fine-grained corruption of text data, a context-aware time predictor that adaptively modulates the reverse process based on semantic context, and an optimized extrinsic time schedule for precise noising control. • We conduct extensive experiments to evaluate the effectiveness of NeoDiff and compare it to existing text diffusion models. Our results highlight the advantages of our unified framework and suggest its potential to advance diffusion-based text generation. 2 Background 2.1 Diffusion Models Diffusion models assume a gradual noise injection process over time for data samples \bm{z}_{0}\in\mathbb{R}^{N\times d} . The forward diffusion process forms a series of latent variables \bm{z}_{1},\bm{z}_{2},\cdots,\bm{z}_{T} satisfying the Markov property, and finally become pure Gaussian noise \bm{z}_{T}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right) : \displaystyle q(\bm{z}_{t}|\bm{z}_{t-1}) \displaystyle=\mathcal{N}\left(\bm{z}_{t};\sqrt{\alpha_{t}}\bm{z}_{t-1},\beta_{t}\mathbf{I}\right), (1) where \alpha_{t}+\beta_{t}=1 , determining the degree of noising at time t and consituting the noise schedule. The reverse process is parameterized as \displaystyle p_{\theta}(\bm{z}_{t-1}|\bm{z}_{t}) \displaystyle=\mathcal{N}\left(\bm{z}_{t-1};\mathbf{\mu}_{\theta}\left(\bm{z}_{t},t\right),\mathbf{\Sigma}_{\theta}\left(\bm{z}_{t},t\right)\right), (2) Here, \mathbf{\mu}_{\theta}\left(\cdot\right) and \mathbf{\Sigma}_{\theta}\left(\cdot\right) are the model’s estimates of the distribution mean and covariance matrix, respectively. The training objective is derived from the variational lower bound of the negative log-likelihood loss, and can be then simplified as an MSE loss ( Ho et al., 2020 ; Li et al., 2022 ; Gao et al., 2024 ) : \mathcal{L}_{\mathrm{VLB}}=\mathbb{E}\left[\left\lVert\bm{z}_{\theta}\left(\bm{z}_{t},t\right)-\bm{z}_{0}\right\rVert^{2}-\log p(\mathbf{z}_{0}\mid\mathbf{z}_{1})\right] 2.2 Discrete Diffusion Models Discrete diffusion models directly model the

Read the complete text