DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs
Longxuan Yu, Yunshu Wu, Yu Fu, Siheng Xiong, Rob Brekelmans, Hui Liu, Yue Dong, Greg Ver Steeg · 2026
Read DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs by Longxuan Yu, Yunshu Wu, Yu Fu, Siheng Xiong, Rob Brekelmans, Hui Liu, Yue Dong, Greg Ver Steeg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01024.txt.
Introduction
Yu and co-authors adapt a masked diffusion language model to denoise in a smooth space in this 2026 paper. Masked diffusion models write text by unveiling tokens step by step. With few steps, they face a trade: short clean text, or long text that repeats itself. Smooth denoising in embedding space can dodge the trade, but training such a model from scratch is unsolved.
The team starts from LLaDA-8B-Instruct and adapts it lightly. One thousand extra training steps swap hard masks for soft Gaussian noise per token. The result, DSL-LLaDA, evolves all positions at once and commits to tokens only at the last step. On zero shot summarization with at most 16 passes, it posts the best ROUGE-1 score on all four benchmarks, they report. It largely skips the early stop and repeat traps of plain unmasking. It also fixes corrupted tokens while leaving clean ones alone.
Catalog record
| CREATOR | Longxuan Yu, Yunshu Wu, Yu Fu, Siheng Xiong, Rob Brekelmans, Hui Liu, Yue Dong, Greg Ver Steeg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs Longxuan Yu, Yunshu Wu, Yu Fu, Siheng Xiong, Rob Brekelmans, Hui Liu, Yue Dong, Greg Ver Steeg Abstract Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text. Continuous denoising can sidestep this tradeoff by evolving all positions jointly in embedding space, but building such a model from scratch at scale remains an open problem. We show that a pretrained masked DLM can instead be lightly adapted to support continuous embedding-space denoising. Starting from LLaDA-8B-Instruct, we continue-pretrain for only 1,000 steps with Discrete Stochastic Localization (DSL), replacing binary masking with continuous per-token Gaussian noise as a soft mask. The adapted model supports continuous inference that evolves all positions jointly in embedding space and defers hard token commitment to the final step. On zero-shot summarization at low step budgets ( ≤ \leq 16 forward passes), DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks and largely avoids the premature-termination / repetition tradeoff of iterative unmasking. The same adaptation also yields selective noisy-state robustness: the model corrects corrupted tokens while preserving clean ones. Control experiments using standard masked diffusion training with the same compute demonstrate neither behavior. 1 Introduction Masked diffusion language models (DLMs) have recently scaled to billions of parameters and begun to approach autoregressive quality on standard benchmarks 22 ; 29 ; 30 , yet their iterative unmasking inference degrades sharply at low step budgets, producing truncated outputs, repetition loops, and incoherent text 14 . The root cause is that each position is decoded independently from its context-conditioned marginal distribution, neglecting inter-token correlations 14 : this factorization introduces little error when each step changes only a few positions, but becomes severe at low step budgets, where each step must commit many positions at once. Figure 1: DSL-LLaDA replaces binary masking with continuous per-token noise, enabling SDE-style continuous inference from a pretrained masked DLM. (A) Original LLaDA uses discrete masked diffusion, where each token is either fully visible or replaced by a [MASK] token. (B) DSL-LLaDA introduces continuous per-token noise at varying SNR. A softmax converter maps each noisy embedding back to the backbone’s input space, producing three phases: unknown (mask-like at low SNR), unreliable (random tokens at mid SNR), and clear (gold token at high SNR), as shown by the per-position probability distributions. These failures motivate methods that improve few-step generation. Improved discrete samplers 26 ; 12 ; 23 and consistency distillation 16 ; 13 reduce the step cost of discrete inference, but they still operate in the discrete token space. Continuous denoising offers a complementary route: it sidesteps the token-space factorization by evolving all positions jointly in a shared embedding space 14 ; 17 , and recent work confirms that continuous flows can match or exceed discrete diffusion in the few-step regime 14 ; 4 . However, building a continuous DLM from scratch at scale remains an open design problem, as embedding geometry, noise scheduling, and training objectives are still actively explored 7 ; 4 ; 14 , and no continuous DLM has been trained beyond moderate scale. Rather than training a continuous DLM from scratch, we show that an existing masked DLM can be lightly adapted to accept continuous inputs (Figure 1 ). Starting from LLaDA-8B-Instruct, we continue-pretrain for only 1,000 steps with Discrete Stochastic Localization (DSL; 28 ), replacing the binary clean-or-mask input with a soft mixture over token embeddings that encodes graded per-position uncertainty. The resulting model, DSL-LLaDA, supports a stochastic differential equation (SDE) based continuous denoising sampler with delayed hard token commitment. Empirically, the adapted model shows three behaviors absent in iterative unmasking. On open-ended generation, the SDE sampler’s delayed token commitment keeps repetition below 9% at all step budgets, while discrete baselines either truncate early or exceed 60% repetition (Section 4.2 ). On zero-shot summarization, continuous inference achieves the best ROUGE-1 on all four benchmarks at \leq 16 forward passes without length-control heuristics (Section 4.2 ). On error correction, the model corrects most corrupted tokens while leaving {>} 98% of clean tokens untouched, whereas XDLM-style random-token training preserves fewer than half (Section 4.1 ). A same-compute binary-masking control experiment acquires none of these behaviors, confirming that continuous-noise exposure, not additional training alone, drives the improvement. Our contributions are: 1. We show that a pretrained 8B masked DLM can be converted to support continuous-space denoising through lightweight DSL adaptation. 2. We demonstrate that continuous SDE inference avoids the low-NFE failure modes of iterative unmasking on generation and summarization tasks. 3. We identify a selective noisy-state robustness induced by continuous-noise training, absent from same-compute discrete baselines. 2 Background 2.1 Masked Diffusion Language Models Notation. Let x_{1:L} denote a token sequence over a vocabulary of size V . The backbone maps each token to an embedding via a learned matrix W_{\text{backbone}}\in\mathbb{R}^{(V+1)\times d_{\text{model}}} , where the extra row corresponds to a dedicated [MASK] token. Training. A masked diffusion language model defines a forward process that independently replaces each token with [MASK] . At diffusion step t , position i is either fully visible ( x_{i}^{t}=x_{i} ) or fully masked ( x_{i}^{t}=\texttt{[MASK]} ). No intermediate state exists. The model is trained to predict the original token at every position using cross-entropy loss. Inference. Generation starts from a fully masked sequence and proceeds by iterative unmasking : at each of T steps the model predicts token probabilities at all masked positions in parallel, samples tokens from these distributions, and commits (unmasks) a subset of positions. Uncommitted positions are re-masked for the next step. This generative process induces two sources of error relevant to this work: (1) each position is decoded independently given the current state, factorizing the reverse transition across positions and neglecting inter-token correlations 14 , and (2) sampling tokens from an imperfect model creates a mismatch with the training data, which cannot be corrected after unmasking. 2.2 Continuous Denoising for Discrete Tokens An alternative to iterative unmasking is to maintain a continuous state for each token position throughout generation 8 ; 28 . Every position holds a soft, noisy representation rather than a committed token, and a denoiser iteratively recovers clean-token estimates until hard tokens are read out at the final step. Since the denoiser conditions on the full noisy sequence, the update is coupled across positions, avoiding the independent-position error described above 4 . However, continuous denoising requires a backbone trained to process soft inputs, a capability that standard masked diffusion models lack. When the continuous state lives in token embedding space, Stochastic Localization 28 offers a natural formulation: as a per-token signal-to-noise ratio (SNR) increases from zero, the model’s belief about which token occupies each position sharpens from uniform (completely uncertain) to concentrated on the correct token. Constraining the token embeddings to the unit sphere makes the optimal denoiser depend only on the noisy state and not on explicit parameters such as the SNR 28 . This SNR-invariance has two practical consequences: the model does not need an explicit timestep input, which is what makes it possible to adapt a pretrained masked DLM without