MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
Haoyu He, Katrin Renz, Yong Cao, Andreas Geiger · 2025
Read MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models by Haoyu He, Katrin Renz, Yong Cao, Andreas Geiger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13148.txt.
Introduction
Haoyu He, Katrin Renz, Yong Cao, and Andreas Geiger fix a training mismatch in masked diffusion language models in a 2025 paper. These models write by unmasking tokens step by step, a promising alternative to left to right generation. In training, though, tokens are hidden at random, so the model never practices the orderly unmasking it must perform in use. That gap costs performance and had been largely overlooked.
The team recast learning the unmasking path as a sequence of decisions and applied reinforcement learning. Their method trains the model under the same progressive schedule used at generation time. It matched the previous best method with far fewer training updates, and beat it clearly on math benchmarks at equal updates. A companion remasking trick, needing no training, let the model revise tokens more flexibly and added further gains.
Catalog record
| CREATOR | Haoyu He, Katrin Renz, Yong Cao, Andreas Geiger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models Haoyu He, Katrin Renz, Yong Cao, Andreas Geiger Abstract Diffusion language models, as a promising alternative to traditional autoregressive (AR) models, enable faster generation and richer conditioning on bidirectional context. However, they suffer from a key discrepancy between training and inference: during inference, MDLMs progressively reveal the structure of the generated sequence by producing fewer and fewer masked tokens, whereas this structure is ignored in training as tokens are masked at random. Although this discrepancy between training and inference can lead to suboptimal performance, it has been largely overlooked by previous works, leaving closing this gap between the two stages an open problem. To address this, we frame the problem of learning effective denoising trajectories as a sequential decision-making problem and use the resulting framework to apply reinforcement learning. We propose a novel Masked Diffusion Policy Optimization (MDPO) to exploit the Markov property diffusion possesses and explicitly train the model under the same progressive refining schedule used at inference. MDPO matches the performance of the previous state-of-the-art (SOTA) method with 60× fewer gradient updates, while achieving average improvements of 9.6% on MATH500 and 54.2% on Countdown over SOTA when trained within the same number of weight updates. Additionally, we improve the remasking strategy of MDLMs as a plug-in inference replacement to overcome the limitation that the model cannot refine tokens flexibly. This simple yet effective training-free strategy, what we refer to as RCR, consistently improves performance and yields additional gains when combined with MDPO. Our findings establish great potential for investigating the discrepancy between pre-training and inference of MDLMs. Code: https://github.com/autonomousvision/mdpo . Project Page: https://cli212.github.io/MDPO/ . 1 Introduction Remarkable advances in language modeling have largely been driven by the autoregressive (AR) paradigm ( Hochreiter & Schmidhuber 1997 ; Vaswani et al. 2017 ) , where tokens are generated in a causal fashion. An alternative line of work, Diffusion Language Models (DLMs) generates by iteratively denoising, enabling parallel prediction, bidirectional conditioning and faster generation speed. Recently, DLMs such as Mercury Coder ( Khanna et al. 2025 ) , Google Gemini Diffusion ( Google DeepMind 2025 ) , and Seed Diffusion ( Song et al. 2025 ) have emerged as promising alternatives that excel at external math and coding benchmarks. Built upon theoretical insights and practical optimizations in the discrete diffusion framework ( Lou et al. 2024 ; Sahoo et al. 2024 ; Ou et al. 2025 ) , Masked Diffusion Language Models (MDLMs) leads the current development of DLMs. Diffusion gradually adds noise to data in the forward process, and the model learns to reverse this corruption in the backward process. In the forward process of MDLMs, tokens are randomly chosen to either remain stationary or transition to a predefined absorbing state, such as a special masking token, motivated by the success of BERT ( Devlin et al. 2019 ) . Then in the backward process, the model learns to invert the noise by predicting all masked tokens conditioned on partially observed (unmasked) context. For generating a sequence of tokens, MDLMs alternates between predicting masked tokens and selectively remasking a fraction of the predictions based on heuristics and uncalibrated token-wise scores.. Representative works as LLaDA-8B ( Nie et al. 2025 ) and Dream-7B ( Ye et al. 2025 ) demonstrate competitive performance to their similarly sized AR counterparts. Despite promising empirical results, we identify two fundamental, largely overlooked problems of MDLMs that hinder effective denoising trajectories. (1) The training–inference divide : at inference time, MDLMs follow a model-dependent, confidence-guided unmasking schedule that progressively reveals structure of the generated sequence. By contrast, training masks tokens randomly at sampled diffusion steps, ignoring this progressive schedule. This mismatch prevents the model from learning effective denoising trajectories and often causes Answer Backslide: correct intermediate solutions that are ‘refined’ into wrong final outputs. (2) The common remasking approaches freeze tokens once predicted, making it impossible to revise early low-confidence predictions later. We address these problems with complementary methods in this work. MDPO casts denoising as a multi-step decision-making process, explicitly training the model under the same progressive, model-dependent unmasking schedule used at inference. By using reinforcement learning with intermediate rewards to optimize these trajectories, MDPO aligns training with actual inference dynamics, closing the training–inference divide. Running Confidence Remasking (RCR) is a training-free decoding strategy that tracks per-position confidence over time, enabling flexible remasking and revision of low-confidence tokens at any step. In summary, our contributions are three-fold: • Answer Backslide: We observe a novel and previously unknown phenomenon in MDLMs: models occasionally produce correct answers at intermediate steps but ‘refine’ them into incorrect results. We refer to this phenomenon as Answer Backslide, which inspires us to supervise models not only on the final results but also intermediate denoising steps. • Masked Diffusion Policy Optimization (MDPO): We propose MDPO to learn effective denoising trajectories without direct supervision. By exploiting the fact that MDLMs yield complete generations at every inference step, MDPO optimizes the model with policy gradients via intermediate-step rewards. Unlike prior RL fine-tuning on MDLMs, MDPO explicitly targets addressing the training-inference divide overlooked by previous works. • Running Confidence Remasking (RCR): Instead of freezing tokens based on their single-step confidence, RCR allows flexible remasking by continuously tracking the running lowest confidence over denoising trajectories. Our experiments demonstrate consistently improved performance of using RCR on both LLaDA pre-trained as well as MDPO fine-tuned models. Empirical results on challenging mathematical and reasoning benchmarks show that both MDPO and the training-free RCR substantially improve generation quality and sample efficiency. Our findings underscore the value of addressing the training-inference divide for masked diffusion language models and open new directions for research in this emerging paradigm. 2 Related Work Figure 1: MDPO generates a group of answers given a query for RL rollouts. Then all completions at intermediate and final steps are verified with a reward model. Based on verified rewards, MDPO estimates the advantage of step t by considering rewards of the other steps in the current rollout and step t from other rollouts in the group. These estimated advantages are used for policy optimization. 2.1 Diffusion Language Models Diffusion probabilistic models ( Sohl-Dickstein et al. 2015 ; Ho et al. 2020 ) have become the de facto standard for generative modeling in continuous domains, including images, videos, 3D shapes, and robotic trajectories. Building on early efforts in discrete diffusion ( Austin et al. 2021 ; Hoogeboom et al. 2021 ) , several works have adapted diffusion models for text generation. Diffusion-LM ( Li et al. 2022 ) and Self-Conditioned Embedding Diffusion ( Strudel et al. 2022 ) have pioneered embedding-based diffusion approaches, achieving comparable performance to AR methods. Liu et al. 2025 introduces a novel framework that separates the generation process into two models: a planner and a denoiser. Meanwhile, masked diffusion has been established as a special case of discrete diffusion processes ( Austin et al. 2021 ; Sahoo et al. 2024 ; Ou et al. 2025 ) , with recent research significantly scaling their capabilities. Notably, DiffuLLaMA ( Gong et al. 2025 ) initializes masked diffusion language models using pretrained LLaMA weights. Block Diffusion ( Arriola et al. 2025 ) models sequences in autoregressive blocks with intra-block