Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun · 2026

Read Continuous Diffusion Scales Competitively with Discrete Diffusion for Language by Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18530.txt.

Introduction

Zhihan Yang and seven co-authors revisit Plaid in a 2026 paper, a likelihood-based continuous diffusion language model, and build RePlaid by aligning it with modern discrete diffusion models. In this unified setting they derive the first scaling law for continuous diffusion models that rivals discrete ones. RePlaid shows only a 20-fold compute gap to autoregressive models and sets a new perplexity bound of 22.1 on OpenWebText among continuous models. Tuning the noise schedule to minimize ELBO variance yields linear cross-entropy over time, spreading denoising difficulty evenly.

Catalog record

CREATORZhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE121999
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun Abstract While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20 × 20\times compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of 22.1 22.1 among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO’s variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain. 1 Introduction Discrete diffusion language models (DLMs) show significant promise, increasingly rivaling the performance and scalability of autoregressive (AR) models across various benchmarks [ 36 , 60 ] . Despite these successes, continuous diffusion remains a compelling alternative due to its inherent potential for controllability and acceleration. The smooth geometry of a continuous latent space enables structured editing [ 35 , 11 ] and allows for the application of efficient ODE-based solvers [ 71 , 32 ] , facilitating distillation of complex sampling trajectories into high-fidelity, single-step generators [ 52 , 58 ] . Unlike discrete diffusion, which often suffers from severe quality degradation under limited sampling budgets, continuous diffusion offers a principled path toward efficient text generation by enabling superior trade-offs between computational cost and sample quality [ 27 , 26 ] . While continuous DLMs present unique advantages, previous work finds a significant training scalability gap compared to discrete DLMs. Previous studies show empirical evidence that continuous diffusion underperforms discrete diffusion and AR models when evaluated at equivalent training FLOPs and parameter counts [ 2 , 17 ] ; the original Plaid study reported a significant 64\times compute overhead compared to autoregressive models to achieve matching likelihood [ 17 ] . This figure is frequently cited as evidence of the poor scalability of continuous DLMs. By revisiting this principled likelihood-based paradigm, we aim to determine whether this performance gap is an inherent limitation of continuous diffusion or a consequence of incompatible scaling analyses. To address this question, we introduce RePlaid , a revised version of Plaid featuring a transformer architecture carefully aligned with modern discrete DLM practices. While the original 64\times gap suggested continuous diffusion was impractical at scale, those findings were obtained under a different setup from the protocols used to establish current discrete diffusion scaling laws. By adopting the exact protocol from 50 —including the same dataset, optimization hyperparameters, and compute budget—we perform the first unified scaling comparison between continuous and discrete DLMs. Crucially, in this controlled environment, the performance gap narrows to 20\times for RePlaid versus AR, placing continuous diffusion on par with discrete diffusion and challenging the narrative of its inherent unscalability. Furthermore, we find that RePlaid achieves consistent improvements in generation quality over other recent continuous models like FLM [ 28 ] and LangFlow [ 9 ] . In contrast to other recent work, RePlaid’s training objective (like Plaid) is explicitly derived from a likelihood bound (ELBO) , providing a theoretically grounded framework for training that heuristic cross-entropy or flow-matching methods lack. We attribute the success of RePlaid to two primary factors: (i) optimizing the noise schedule via ELBO variance naturally recovers a near-linear cross-entropy schedule that evenly distributes denoising difficulty without heuristic reparameterization, and (ii) ELBO-based embedding optimization inherently regularizes the latent geometry to improve likelihood, creating a structured, low-entropy space that prevents potential token dispersion in cross-entropy-based training. Our main contributions are summarized as follows: • RePlaid & Benchmarking: We introduce RePlaid, a modernized likelihood-based DLM achieving a state-of-the-art 22.1 perplexity among existing continuous DLMs on OpenWebText. • Unified Scaling Analysis: We present the first unified scaling law comparison between continuous and discrete DLMs, showing that continuous models scale on par with discrete counterparts when properly aligned. • Theoretical Insights: We show that the ELBO objective: (i) naturally recovers a linear cross-entropy noise schedule, eliminating heuristic time reparameterizations, and (ii) inherently regularizes embedding geometry for likelihood improvement and prevents token dispersion. 2 Background: Plaid—A Likelihood-Based Continuous DLM Plaid [ 17 ] is a Variational Diffusion Model (VDM) [ 27 ] for text. Let [V]:=\{1,\ldots,V\} be the set of vocabulary with size V . A length- L sequence {\mathbf{x}}\in[V]^{L} is identified with a matrix in \{0,1\}^{L\times V} whose l -th row {\mathbf{x}}^{l} is the one-hot vector of the l -th token. Plaid defines the embedded sequence {\mathbf{e}}:={\mathbf{x}}{\mathbf{E}}\in\mathbb{R}^{L\times d_{e}} , where {\mathbf{E}}\in\mathbb{R}^{V\times d_{e}} is a learnable token-embedding matrix with each row constrained to unit Euclidean length; the l -th row of {\mathbf{e}} , {\mathbf{e}}^{l} , is the embedding of the non-zero index of {\mathbf{x}}^{l} . Unless otherwise stated, we use low-dimensional embeddings ( d_{e}=16 ), as in Plaid. Forward and reverse processes. The forward process q is Gaussian noising on embedding {\mathbf{e}} : \displaystyle q({\mathbf{z}}_{t}\mid{\mathbf{x}})=\mathcal{N}(\alpha_{t}{\mathbf{e}},\sigma_{t}^{2}{\mathbf{I}}),\quad t\in[0,1], (1) where \alpha_{t},\sigma_{t}>0 are smooth scalar functions of t . VDMs assume a variance-preserving (VP) process, \alpha_{t}^{2}+\sigma_{t}^{2}=1 , yielding a strictly decreasing signal-to-noise ratio {\mathrm{SNR}}(t):=\alpha_{t}^{2}/\sigma_{t}^{2} . The true reverse posterior q({\mathbf{z}}_{s}\mid{\mathbf{z}}_{t},{\mathbf{x}}) for s<t is then \displaystyle q({\mathbf{z}}_{s}\mid{\mathbf{z}}_{t},{\mathbf{x}})=\mathcal{N}\big((\alpha_{t|s}\sigma^{2}_{s}/\sigma_{t}^{2}){\mathbf{z}}_{t}+(\alpha_{s}\sigma^{2}_{t|s}/\sigma^{2}_{t}){\mathbf{e}},(\sigma^{2}_{t|s}\sigma^{2}_{s}/\sigma^{2}_{t}){\mathbf{I}}\big), (2) where \alpha_{t|s}:=\alpha_{t}/\alpha_{s} and \sigma^{2}_{t|s}:=\sigma_{t}^{2}-\alpha_{t|s}^{2}\sigma_{s}^{2} . See [ 27 , App. A.2] for the proof. Training. Let {\mathbf{x}}_{\theta}({\mathbf{z}}_{t},t):\mathbb{R}^{L\times d_{e}}\times[0,1]\rightarrow(\Delta^{V})^{L}\subset\mathbb{R}^{L\times V} denote a time-conditioned denoising model outputting a categorical distribution over the vocabulary for every position (i.e., each row is a probability vector on the simplex \Delta^{V} ). Before softmax, Plaid augments the final logit head output of {\mathbf{x}}_{\theta} with output prior logits, a closed-form Gaussian log-density that helps focus the denoiser’s prediction to plausible tokens given {\mathbf{z}}_{t} . Plaid parameterizes the reverse process by plugging the denoiser’s prediction into Eq. 2 : \displaystyle p_{\theta}({\mathbf{z}}_{s}\mid{\mathbf{z}}_{t})=q({\mathbf{z}}_{s}\mid{\mathbf{z}}_{t},{\mathbf{x}}={\mathbf{x}}_{\theta}({\mathbf{z}}_{t},t)). (3) With prior p({\mathbf{z}}_{1})=\mathcal{N}({\mathbf{0}},{\mathbf{I}}) , this yields the Negative Evidence Lower Bound (NELBO) containing respectively prior loss , reconstruction loss , and diffusion loss (see Appendix C for the derivation): \displaystyle-\log p_{\theta}({\mathbf{x}})\leq\mathcal{L}_{\mathrm{NELBO}}({\mathbf{x}}):= \displaystyle\operatorname{KL}(q({\mathbf{z}}_{1}\mid{\mathbf{x}})\,\|\,p({\mathbf{z}}_{1}))+\mathbb{E}_{{\mathbf{z}}_{0}\sim q({\mathbf{z}}_{0}\mid{\mathbf{x}})}\big[{\textstyle\sum}_{l=1}^{L}-\log\langle{\mathbf{x}}_{\theta}^{l}({\mathbf{z}}_{0},0),{\mathbf{x}}^{l}\rangle\big] \displaystyle-\tfrac{1}{2}\ \mathbb{E}_{t\sim\mathcal{U}[0,1],{\mathbf{z}}_{t}\sim q({\mathbf{z}}_{t}\mid{\mathbf{x}})}[{\mathrm{SNR}}^{\prime}(t)\|{\mathbf{e}}_{\theta}({\mathbf{z}}_{t},t)-{\mathbf{e}}\|^{2}], (4) where {\mathbf{e}}_{\theta}({\mathbf{z}}_{t},t):={\mathbf{x}}_{\theta}({\mathbf{z}}_{t},t){\mathbf{E}} is the predicted clean embedding, and \langle\cdot,\cdot\rangle denotes the inner product. During training, Plaid adaptively allocates a fraction of each batch to the reconstruction loss (Monte Carlo over {\mathbf{z}}_{0} ) and the remainder to the diffusion loss (Monte Carlo over t and {\mathbf{z}}_{t} ); the prior loss uses the whole batch. See Appendix B for the training algorithm. Learnable noise schedule. Plaid inherits the VDM parameterization [ 27 ] of the noise schedule \gamma(t)=\gamma_{0}+(\gamma_{1}-\gamma_{0})\tilde{\gamma}(t) , with learnable scalar endpoints \gamma_{0}<\gamma_{1} and interior shape \tilde{\gamma}:[0,1]\rightarrow[0,1] modeled by a monotonically increasing neural net. This gives \alpha^{2}_{t}=\operatorname{sigmoid}(-\gamma(t)) , \sigma^{2}_{t}=\operatorname{sigmoid}(\gamma(t)) , and {\mathrm{SNR}}(t)={\mathrm{e}}^{-\gamma(t)} .

Read the complete text