GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
Xiaohang TangUCL Dept. of Statistical ScienceUCL Centre for AI, Keyue JiangAlibaba GroupUCL Centre for AI & Dept. of EEEChe LiuImperial College LondonQifang ZhaoAlibaba GroupXiaoxiao XuAlibaba GroupSangwoong YoonUNIST, Ilija BogunovicUniversity of Basel · 2026
Read GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models by Xiaohang TangUCL Dept. of Statistical ScienceUCL Centre for AI, Keyue JiangAlibaba GroupUCL Centre for AI & Dept. of EEEChe LiuImperial College LondonQifang ZhaoAlibaba GroupXiaoxiao XuAlibaba GroupSangwoong YoonUNIST, Ilija BogunovicUniversity of Basel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29398.txt.
Introduction
Xiaohang Tang proposes Guided Denoiser Self-Distillation for diffusion language models. Reinforcement learning can improve the denoiser, but its likelihood cannot be computed directly. Standard methods swap in an evidence lower bound, but this adds bias. It distills the denoiser from an optimal self-teacher, matching logits with a normalization-free objective. On planning, math, and coding benchmarks, it beats prior evidence lower bound methods with more stable rewards.
Catalog record
| CREATOR | Xiaohang TangUCL Dept. of Statistical ScienceUCL Centre for AI, Keyue JiangAlibaba GroupUCL Centre for AI & Dept. of EEEChe LiuImperial College LondonQifang ZhaoAlibaba GroupXiaoxiao XuAlibaba GroupSangwoong YoonUNIST, Ilija BogunovicUniversity of Basel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72188 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29398
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models Xiaohang TangUCL Dept. of Statistical ScienceUCL Centre for AI, Keyue JiangAlibaba GroupUCL Centre for AI & Dept. of EEEChe LiuImperial College LondonQifang ZhaoAlibaba GroupXiaoxiao XuAlibaba GroupSangwoong YoonUNIST, Ilija BogunovicUniversity of Basel Abstract Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of methods replaces the likelihood in standard RL with its evidence lower bound (ELBO), estimated from randomly masked sequences. Despite being well aligned with pre-training, these approaches introduce bias through training–inference mismatch by using the ELBO as a likelihood surrogate, which can degrade performance. In this work, we propose Guided Denoiser Self-Distillation (GDSD) to directly distill the denoiser of dLLMs from an advantage-guided self-teacher, derived from the closed-form optimum of reverse-KL regularized RL. GDSD matches the dLLM’s denoiser logits to the teacher’s via a normalization-free objective, which reduces RL to likelihood-free self-distillation and thus bypasses the TIM biases. Recent ELBO-based methods emerge as instances of applying different distillation divergences, but with diagnosable pathologies that GDSD avoids. On planning, math, and coding benchmarks with LLaDA-8B and Dream-7B, GDSD consistently outperforms prior state-of-the-art ELBO-based methods with a more stable training reward dynamics, achieving test-accuracy improvements of up to + 19.6 % +19.6\% . These results suggest that direct denoiser self-distillation, without relying on an ELBO likelihood surrogate, can provide a more stable and effective RL procedure for dLLMs. Code is available at https://github.com/GaryBall/GDSD . 1 Introduction Diffusion Large Language Models (dLLMs) have emerged as efficient alternatives to autoregressive models (ARMs). dLLMs generate multiple tokens in a single decoding step and do not follow a strictly left-to-right generation order, thereby improving generation efficiency and unlocking token dependencies beyond the left-to-right paradigm in ARMs. This promise is also reflected in several recent releases, which have scaled model size while substantially improving inference efficiency. Open-weight reasoning dLLMs have grown from 8B-parameter models [ 31 , 66 ] to 100B parameters in LLaDA 2.0 [ 5 ] , with inference reported to be more than 3\times faster than that of even smaller ARMs [ 4 ] . Closed models such as Mercury similarly report speedups of up to 10\times relative to ARMs [ 22 ] . Despite these efficiency gains, dLLMs still lag behind the state-of-the-art ARMs in generation quality, highlighting the need for effective fine-tuning methods, such as reinforcement learning (RL). The central obstacle to RL with dLLMs is that the policy likelihood is intractable . Two families of methods have emerged as strong-performing solutions. RL based on trajectory likelihood estimates the exact likelihood of the trajectory of a sequence in the reverse process by accumulating the transition probabilities [ 19 , 54 , 8 , 51 ] . While unbiased in principle, these methods incur substantial computational cost per gradient step, being misaligned with the pre-training objectives. RL based on sequence likelihood applies likelihood evidence lower bound (ELBO) as a surrogate, estimated from randomly masked sequences [ 62 , 58 , 66 , 48 , 32 , 52 ] . This approach is computationally efficient, and naturally aligned with the objective of current dLLMs pre-training, which is itself also defined on a randomly masked sequence. Thus, we focus on the sequence likelihood family in this work. RL based on sequence likelihood with ELBO surrogate is inherently off-policy, yet existing methods attempt to correct it with importance sampling ratio based on ELBO. These methods naively plug the ELBO surrogate into policy gradient or PPO-style objectives in the place of true likelihood, introducing a well-known bias called training-inference mismatch (TIM) [ 37 , 24 ] . First, the non-negligible gap between the ELBO estimate and the true likelihood [ 21 ] biases the importance ratio, which has been shown to degrade performance [ 21 ] , and can even cause training collapse [ 64 ] . Additionally, dLLM rollouts are produced by iterative re-masking and block-wise decoders [ 59 , 31 , 2 ] , whose induced sampling distribution differs from the training policy (i.e., ELBO). The importance-sampling formulation requires a tractable training-sampling relationship that ELBO does not provide, which motivates a different formulation. To address the TIM bias, we shift from importance-sampling RL with the ELBO surrogate to direct, off-policy self-distillation on the denoiser. To our knowledge, this is the first principled method recasting RL for dLLMs as self-distillation to avoid TIM bias by design. Our contributions are: • We reduce RL with dLLMs to denoiser self-distillation: under reverse-KL regularized RL, the closed-form optimal policy induces a guided denoising distribution, an advantage guided denoiser as self-teacher, which we aim to distill at each RL step. (Section 3.1 ) • We propose Guided Denoiser Self-Distillation (GDSD) , a squared-logit distillation loss with a normalization-free reformulation that eliminates the partition function via logit centralization. Distillation allows for off-policy update, thus bypassing TIM bias entirely (Sections 3.2 , 3.3 ) • We provide insights on how existing ELBO-based methods are connected to alternative divergence choices under our distillation framework. We demonstrate that our squared-logit instance avoids both the data inefficiency of weighted ELBO instances (wd1 [ 48 ] , DMPO [ 67 ] ) and the TIM bias of policy gradient instances (SPG [ 52 ] , UniGRPO [ 58 ] , ESPO [ 32 ] ). (Section 4 ) • In experiments compared to prior state-of-the-art ELBO-based methods, GDSD shows more stable training reward dynamics. On planning tasks, GDSD with Dream-7B achieves test-accuracy gains of up to +19.6\% . With LLaDA-8B, GDSD also consistently improves performance across planning, math, and coding benchmarks, with gains ranging from +0.6\% to +5\% . (Section 5 ) 2 Preliminaries In this section, we introduce the formulation of current diffusion large language models (dLLMs), namely masked diffusion models, and their reinforcement learning formulation. 2.1 Masked Diffusion Models Masked Diffusion Models (MDMs) [ 40 ] represent a prominent class of non-autoregressive generative models that operate over a discrete state space. Let \mathcal{V} denote the categorical vocabulary and [\text{M}] a dedicated mask token, such that the augmented space is \mathcal{V}^{\prime}=\mathcal{V}\cup\{[\text{M}]\} . For a clean sequence with length N , denoted by x_{0}\in\mathcal{V}^{N} , the forward process q(x_{t}|x_{0}) defines a continuous-time transition for t\in(0,1] , where masked sequence x_{t}\in\mathcal{V^{\prime}}^{N} . Forward transition from state x_{s} to x_{t} ( 0\leq s<t ) is formulated as: q(x_{t}|x_{s})=\mathrm{Cat}(x_{t};Q(s,t)^{\top}x_{s}) where Q(s,t) is the transition kernel. The MDM, denoted by p_{\theta} , is then trained to model the reverse process. In particular, we term the reverse model for clean sequence prediction (i.e. p(x_{0}|x_{t}) ) as denoiser in this work. Let x_{0}^{(n)} denote the n -th token of the clean sequence, open large-scale MDMs, such as LLaDA-8B [ 31 ] and Dream-7B [ 59 ] , directly model the token-level denoising distribution p_{\theta}(x_{0}^{(n)}|x_{t}) , and are pre-trained by maximizing the following negative cross-entropy objective [ 33 , 46 ] : \displaystyle{L}(x_{0}^{(n)};p_{\theta})=\mathbb{E}_{t\sim\mathcal{U}[0,1],\ x_{t}\sim q(x_{t}|x_{0})}\left[w(t)\cdot\mathbf{1}(x_{t}^{(n)}=[\text{M}])\log p_{\theta}(x_{0}^{(n)}|x_{t})\right]. (1) The sequence-level objective is obtained by accumulating the token-level objective, specifically {L}(x_{0};p_{\theta})=\sum_{n=1}^{N}{L}(x_{0}^{(n)};p_{\theta}), which is equivalent to the likelihood evidence lower bound (ELBO). Equation 1 , the token-level contribution to the ELBO [ 32 ] , has also been used to approximate