Don’t Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
Zemin Huang, Yuhang Wang, Zhiyang Chen, Guo-Jun Qi · 2025
Read Don’t Settle Too Early: Self-Reflective Remasking for Diffusion Language Models by Zemin Huang, Yuhang Wang, Zhiyang Chen, Guo-Jun Qi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23653.txt.
Introduction
Diffusion language models write by unmasking tokens, but a wrong token, once placed, usually stays forever. Zemin Huang and 3 co-authors give their model, RemeDi, a way to take tokens back. At each step it predicts both token distributions and a confidence score per token. Low-confidence tokens get remasked and can be resampled later with richer context around them.
Training teaches the skill in two stages: supervised fine-tuning on detecting and remasking errors, then reinforcement learning over full generation paths. On several data sets, RemeDi reached the best results among open-source diffusion language models. The authors present remasking as a core mechanism, not a patch.
Catalog record
| CREATOR | Zemin Huang, Yuhang Wang, Zhiyang Chen, Guo-Jun Qi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Don’t Settle Too Early: Self-Reflective Remasking for Diffusion Language Models Zemin Huang, Yuhang Wang, Zhiyang Chen, Guo-Jun Qi Abstract Mask-based Diffusion Language Models (DLMs) struggle to revise incorrect tokens: once a token is generated, it typically remains fixed. The key challenge is to identify potential errors in the inputs. In this paper, we propose Rem asking- e nabled Di ffusion Language Model (RemeDi) , a mask-based DLM that introduces remasking as another fundamental mechanism, enabling more flexible text refinement in diffusion-based text generation. To achieve this, RemeDi jointly predicts token distributions and per-token confidence scores at each step. The confidence scores determine which tokens to be unmasked after the current step, allowing the model to identify tokens with low quality and remask them. These remasked tokens can be resampled with richer context in subsequent steps. We design a remask-aware pipeline to train this ability, including supervised fine-tuning which teaches the model to detect and remask incorrect tokens in addition to predict mask tokens, and reinforcement learning which optimizes full generation trajectories toward higher rewards. Experiments show that RemeDi achieves the state-of-the-art results among open-source DLMs on multiple datasets. https://github.com/maple-research-lab/RemeDi 1 Introduction Diffusion Language Models (DLMs) have recently emerged as a promising alternative to autoregressive language models Nie et al. 2025 ; Ye et al. 2025 ; Lou et al. 2024 ; Arriola et al. 2025 . A DLM defines a forward process that gradually corrupts text into a noise prior, and learns a reverse process to recover clean text Campbell et al. 2022 ; Lou et al. 2024 . Unlike autoregressive models, DLMs do not commit to a fixed left-to-right order, offering greater flexibility in generation and an inherent ability to predict multiple tokens in parallel. A dominant variant is the mask-based DLM Nie et al. 2025 ; Ye et al. 2025 , where the noise is represented by a special mask token. Under this formulation, the model learns to recover masked tokens during training, while assuming that once tokens are unmasked, they are supposed to be correct without having to clean them later. This assumption is problematic: the model may generate wrong tokens, which should be revealed and corrected in later steps when more contexts are available. However, most existing DLMs Nie et al. 2025 ; Ye et al. 2025 keep already unmasked tokens fixed, preventing them from being revised by self-reflecting on errors. To address this, several works have explored methods to revise generated tokens. von Rütte et al. 2025 defines a new noise schedule by interpolating between masking and uniform noise, enabling revision of wrong tokens on small-scale models. Wang et al. 2025a applies predictor-corrector samplers by stochastically remasking a subset of tokens only at inference time, where the remasking is performed randomly without training the model how to find and remask incorrect tokens. For large-scale DLMs, Seed Diffusion Song et al. 2025 allows all tokens to be resampled at every step. However, it lacks a mechanism to ensure that the number of mask tokens decreases monotonically — it is a key feature for diffusion models to ensure decreasing noise levels over steps Guo et al. 2025 , so that the mask tokens will eventually vanish at the final step to complete the generation. In this paper, we propose a self-reflective remasking approach to train DLMs. As illustrated in Fig. 1(a) , it aims to train DLMs with the ability of finding wrong tokens and turning them back to mask ones so that they can be resampled with richer context in later steps. Based on this, we introduce Remasking-enabled Diffusion Language Model ( RemeDi ) , a mask-based DLM that incorporates self-reflective remasking to revise already generated but incorrect tokens. RemeDi jointly predicts token distributions and per-token confidence scores. At each diffusion step, high-confidence tokens are unmasked while low-confidence ones are (re-)masked, regardless of whether they have been previously unmasked. The key challenge is to train the model how to remask incorrect tokens in a self-reflective manner. To this end, we design a remask-aware training pipeline in two stages: 1) Remask SFT , where the model learns to identify and remask incorrect tokens, while predicting masked tokens. We construct an input sequence for Remask SFT by randomly masking its tokens or replacing them with random alternatives to simulate the noise. The noise schedule deciding how many tokens are masked or randomly replaced is designed to follow the criterion that the noise level should monotonically decrease over steps. The model is then trained to remask and revise incorrect tokens over noisy input sequences. 2) Remask RL , where the model is further fine-tuned with outcome-based reinforcement learning. It seeks to optimize the entire generation trajectories toward final outputs with higher rewards by considering how to remask and predict tokens in each step. As shown in Fig. 1(b) , RemeDi achieves the state-of-the-art performance among open-source DLMs, achieving competitive results on various benchmark datasets, including math problems (89.1% on GSM8K Cobbe et al. 2021 , 52.9% on MATH Hendrycks et al. 2021 ), code generation ( 73.2% on HumanEval Chen et al. 2021 , 59.4% on MBPP Austin et al. 2021 ), and general tasks (24.5% on AlpacaEval Dubois et al. 2024 , 85.4% on IFEval Zhou et al. 2023 , and 87.7% on ARC-C Clark et al. 2018 ). (a) Green and Red tokens are already unmasked in the input sequence of the current step. Red tokens are remasked. Blue tokens are unmasked in the outputs. Gray tokens keep masked in the outputs, and we display the token with the highest probability at these positions. (b) Radar plot comparing the performance of RemeDi with other DLMs across various evaluation benchmarks. Figure 1: (a) Illustration of remasking for quality improvement: RemeDi initially predicts the token “left”, but later identifies a semantic mismatch in the phrase “left for the pies”. The model then remasks this token and corrects it to the more appropriate “used”. (b) RemeDi outperforms existing DLMs in various tasks, including math, code and general benchmark. 2 Related Work 2.1 Mask-based Diffusion Language Models Diffusion language models (DLMs) have emerged as promising alternatives to auto-regressive (AR) models for text generation. Among them, mask-based DLMs Nie et al. 2025 ; Ye et al. 2025 ; Zheng et al. 2023 ; Ou et al. 2024 dominate, generating text by progressively denoising mask tokens. Recent studies Arriola et al. 2025 ; Fathi et al. 2025 ; Huang & Tang 2025 ; Sahoo et al. 2025 ; Wang et al. 2025b ; Gat et al. 2025 have increasingly explored the fusion of AR and diffusion models, often through an iterative block-wise decoding strategy: inference proceeds by iteratively appending a block of mask tokens to the input sequence and denoising it, repeating until the EOS token is generated. This paradigm inherits the strengths — flexible generation order and parallel decoding from DLMs, and cache efficiency from AR — yielding faster inference without sacrificing quality. In our work, we adapt LLaDA-8B-Instruct Nie et al. 2025 to variable-length block-by-block generation, serving as the backbone for our remasking mechanism. 2.2 Revising Errors in Diffusion Language Models A key limitation of mask-based DLMs is their inability to revise tokens once unmasked, even if they are incorrect. Existing