Understanding and Accelerating the Training of Masked Diffusion Language Models
Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji, Seungryong Kim, Jong Chul Ye · 2026
Read Understanding and Accelerating the Training of Masked Diffusion Language Models by Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji, Seungryong Kim, Jong Chul Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13026.txt.
Introduction
Hong and seven co-authors ask why masked diffusion language models train far more slowly than autoregressive models, and how to speed them up. They trace the drag to locality bias: predictive information for a token sits mostly nearby. Their remedy, bell-shaped time sampling, lets models trained their way reach the same validation loss about four times faster on the One Billion Word Benchmark. Generative perplexity, zero-shot perplexity and downstream task scores improve faster too.
Catalog record
| CREATOR | Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji, Seungryong Kim, Jong Chul Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118278 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13026
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding and Accelerating the Training of Masked Diffusion Language Models Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji, Seungryong Kim, Jong Chul Ye Abstract Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to ∼ 4 × \sim 4\times faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks. Figure 1 : Locality bias in language makes MDMs slow learners. The red box illustrates the concept of locality bias: a token is influenced more strongly by nearby tokens and more weakly by distant ones. During training, ARMs learn to predict the target token from left-filled sequences, where an appropriate amount of local information is always available; in contrast, MDMs learn to predict the target token from arbitrary masked contexts, which may include cases where local information is either extremely scarce or extremely abundant, as highlighted by the red boxes . Our analysis suggests that these two regimes may be among the critical factors underlying the slow training of MDMs. 1 Introduction Recently, discrete diffusion models ( 39 ; 2 ) have rapidly emerged as a compelling framework for language modeling. They can be broadly divided into uniform discrete diffusion models ( 39 ; 57 ) and masked diffusion models (MDMs) ( 62 ; 39 ; 56 ; 48 ) , with various frameworks ( 67 ; 38 ; 17 ; 31 ; 75 ; 78 ; 53 ; 51 ; 23 ) further developed from these two families. Among this wide range of studies, the most basic MDM framework ( 48 ; 56 ) remains the most widely adopted in real-world applications precisely because of its simplicity. Large-scale models, such as LLaDA ( 47 ) , Seed Diffusion ( 63 ) , and Dream ( 70 ) , ultimately follow the most standard generation scheme, in which model predictions are used to gradually convert a fully masked sentence into tokens. Various inference-time techniques ( 35 ; 18 ) , reinforcement learning methods ( 22 ; 74 ) , and even retrieval-augmented generation ( 71 ; 30 ) are built upon this basic MDM framework. Recent evidence suggests standard MDMs can become competitive with autoregressive models (ARMs). For general-purpose models, MDMs such as LLaDA ( 47 ) and Dream ( 70 ) have surpassed same-sized ARM counterparts such as LLaMA 3 8B ( 14 ) , while for coding models, DiffuCoder ( 13 ) and Dream-Coder ( 68 ) have matched same-sized Qwen2.5-Coder ( 25 ) . Prior work ( 52 ) further suggests that given sufficiently long training, MDMs can eventually match the performance of ARMs. However, the main bottleneck of MDMs is that they learn much more slowly than ARMs. Specifically, MDMs are known to require \sim 16\times training compute to reach the same validation loss of ARMs ( 52 ; 58 ) . This becomes a major obstacle when scaling MDMs to real-world LLMs, since the training cost of LLMs is extremely high; for example, even ARM-based LLaMA 3.1 405B spent \sim 30M H100 GPU hours for training. Differences in training efficiency can translate into enormous differences in both cost and time. In this context, we raise a key question that has been overlooked in prior works: How can we accelerate standard MDM training while maintaining its final performance? To this end, we first analyze why MDM training is slow. Stepping back from MDM, we begin by separating two possible causes of slow any-order learning. The first is the locality bias of language: each token in natural language is influenced much more strongly by nearby tokens than by distant ones ( 10 ) . Since any-order generation includes orders beyond left-to-right (L2R), locality bias can shape its training dynamics differently. The second is the large order-space complexity of the any-order generation problem: unlike a fixed L2R order, models must learn a vastly larger family of generation problems. We show that the former factor, locality bias, has a substantially greater impact on training speed. Furthermore, we analyze why the locality bias slows MDM training, and find that both high-context (almost no masked) and low-context (almost fully masked) samples waste training budget. Although L2R bias, closely related to locality bias, has been pointed out as a challenge in many prior works ( 32 ; 69 ) , there has been little detailed analysis of how it slows down training. Our second contribution is to introduce a simple, effective, and generalizable technique for accelerating MDM training . Motivated by our analysis of why MDMs learn slowly, we propose bell-shaped time sampling , which increases the probability of sampling t from the middle-context region, and show that this significantly speeds up MDM training. Notably, bell-shaped time sampling achieves up to \mathbf{\sim 4\times} faster training in reaching the same NLL on LM1B ( 5 ) . We show that it also leads to faster improvements in generative perplexity (PPL), and zero-shot PPL for models trained on OpenWebText ( 11 ) . Finally, we show that our method remains effective even in the continual pretraining with ARM initialization ( 12 ) setting commonly adopted for large-scale MDMs ( 3 ) , and can further improve downstream task performance. 2 Background 2.1 Preliminaries To analyze why MDMs are slow learners, we compare various classes of language models. We briefly explain those frameworks and objectives. Refer to Appendix A for details and related works. Notation. Let {\mathbf{x}} be a categorical sample with L dimensions and x^{i}\in\{1,\dots,V\} be the i -th dimension of {\mathbf{x}} . We define special mask state as {m}=V+1 . Let {\mathbf{e}} denote the standard basis of \mathbb{R}^{V+1} . Thus, each token is represented by a one-hot vector {\mathbf{e}}_{i} in \mathbb{R}^{V+1} . Let \mathrm{Cat}(\bm{\pi}) denote the categorical distribution over V\!+\!1 classes with \bm{\pi}\in\Delta^{V+1} , the (V+1) -simplex. We denote a noised sequence at time t by {\mathbf{x}}_{t} . We further define a mask indicator {\mathbf{M}}\in\{0,1\}^{L} and the masked sequence {\mathbf{x}}[{\mathbf{M}}] that is obtained by replacing x^{i} with {m} whenever M^{i}=1 , while leaving it unchanged whenever M^{i}=0 . Let {\mathbf{M}}_{n} denote a mask indicator with mask number of n and \mathcal{M}_{n} as set of all possible combinatorial {\mathbf{M}}_{n} . For ARM, let {\mathbf{x}}^{<i}:=(x^{1},\ldots,x^{i-1}) . We write uniform distribution as \mathrm{U}(\cdot) . Masked diffusion models. MDM ( 56 ; 62 ) defines the forward corruption process using the absorbing mask strategy: once a token is masked, it remains masked throughout the remaining