Unifying Masked Diffusion Models with Various Generation Orders and Beyond Chunsan Hong, Sanghyun Lee, Jong Chul Ye Abstract Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation, but generation quality depends critically on the generation order. Prior work either hard-codes an ordering (e.g., blockwise left-to-right) or learns an ordering policy for a pretrained MDM, which incurs extra cost and can yield suboptimal solutions due to the two-stage optimization. Motivated by this, we propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes with various generation orders, enabling the interpretation of MDM, ARM, and block diffusion in a single framework. Furthermore, building on OeMDM, we introduce learnable-order masked diffusion model (LoMDM), which jointly learns the generation ordering and diffusion backbone through a single objective from scratch, enabling the diffusion model to generate text in context-dependent ordering. Empirically, we confirm that LoMDM outperforms various discrete diffusion models across multiple language modeling benchmarks. 1 Introduction Diffusion models have recently been extended to language modeling in the form of masked diffusion models (MDMs) ( 16 ; 26 ) , which are emerging as a potential alternative to autoregressive models (ARMs). MDMs define a forward process that replaces tokens in random positions with [ MASK ] and learn a corresponding reverse process. Training typically minimizes a negative evidence lower bound (NELBO) ( 30 ; 26 ) to fit the reverse dynamics. Figure 1 : Conceptual illustration of learnable-order masked diffusion model (LoMDM) and other language models. Black text denotes already generated tokens, while the colored tokens indicate the generation candidates, with a lower color represents low generation order priority. In training time, LoMDM jointly learns what to generate and where to generate next , and in inference-time, LoMDM selects where to unmask next and predict a token. One of the most actively studied directions in MDMs is the probabilistic modeling design with improved generation orderings beyond purely random orderings. However, classical MDMs are inherently order-agnostic : both the corruption process and the denoising objective are defined over randomly masked subsets, yielding a training signal that does not explicitly favor any particular generation ordering. Alternatively, structured orderings are incorporated through hand-designed schedulers or by redesigning the framework for each specific ordering. A representative example is block diffusion (BD3LM) ( 1 ) , which adopts a blockwise left-to-right ordering with random permutations within each block. GenMD4 ( 30 ) learns a token-dependent scheduler, differentiating the noising and denoising ratio by vocabulary. Both approaches require a modified modeling, which redefines the corruption/reverse processes to accommodate the imposed ordering. Another line of work focuses on post-training the unmasking position sampler for improved orderings. Given an MDM trained with random ordering, various works ( 9 ; 23 ; 11 ) learn a sampler that determines the unmasking order. However, such post-training approaches suffer from two key limitations: they incur additional training costs, and they may converge to suboptimal solutions due to the two-stage optimization between the MDM backbone and the unmasking ordering. These observations raise two fundamental questions: 1) how to model various generation orders in the masked diffusion framework, and 2) how to learn generation order and the diffusion model jointly in a principled way. To address this, we introduce order-expressive masked diffusion model (OeMDM) that provides a lens for understanding diverse generation orderings in MDMs with a generalized NELBO. Recall that in classical MDM formulation ( 16 ; 26 ) , a position-invariant noise schedule makes the forward masking rate uniform across positions, which in turn yields a uniform denoising rate in the reverse process and results in a primarily random ordering in both training and inference. In contrast, our OeMDM makes the generation order explicit by treating the scheduler as a modeling component , enabling the generation order to be considered in both training and inference. Specifically, the key contribution of OeMDM is to show that: • Training an order-aware diffusion model requires sampling masked sequences from an order-induced corruption distribution, rather than using uniform random masking. • The generalized NELBO for OeMDM can be decomposed into a reconstruction loss and a mismatch loss between corruption and unmasking order, yielding a principled mechanism for order-aware learning. • This lens enables systematic analysis of different paradigms (e.g., ARMs, standard MDMs, BD3LM, and GenMD4) under a single formulation. Additionally, our OeMDM framework naturally leads to learnable-order masked diffusion model (LoMDM) , which considers context-aware generation order. In contrast to the prior approaches that either hand-design the generation ordering or learn only restricted schedulers, LoMDM parameterizes a position-dependent scheduler conditioned on the full sequence so that we can learn full-context aware generation order and the diffusion model jointly by minimizing a single NELBO. This unified learning provides two benefits: (i) the learned scheduler is directly usable at generation time to decide where to unmask next, improving sample quality; and (ii) training focuses tokens with higher generation priority, so the scheduler simultaneously shapes a more order-aware training signal for the diffusion model. Empirically, LoMDM achieves lower test perplexity than a range of discrete diffusion baselines, including BD3LM and GenMD4. 2 Background Discrete diffusion. Discrete diffusion models ( 2 ; 10 ) have emerged as a competitive paradigm for discrete data, e.g., text. The main objective of discrete diffusion modeling is to model the discrete data distribution via a continuous-time diffusion process. There are broadly two types of forward corruption processes in discrete diffusion models: (1) uniform corruption ( 16 ; 28 ) , which replaces tokens with uniformly random tokens, and (2) masking corruption ( 26 ; 30 ) , which replaces tokens with a special [ MASK ] token. Among these, masked diffusion models (MDMs) have emerged as a leading class of discrete diffusion models for text generation ( 26 ) . Our work falls into this line of research and aims to improve MDMs in a continuous-time setting. Further details on related work are provided in Appendix A . Notation. Let the vocabulary size be V and define the token space \mathcal{X}:=\{\mathbf{v}\in[0,1]^{V+1}\mid\sum_{j=1}^{V+1}\mathbf{v}_{j}=1,\ \mathbf{v}_{V+1}=0\} , where each word is represented by a one-hot vector. Let the mask token be \mathbf{m}:=\mathbf{e}_{V+1}\notin\mathcal{X} . Let \mathrm{Cat}(\cdot;\bm{\pi}) denote the categorical distribution over V\!+\!1 classes with \bm{\pi}\in\Delta^{V+1} , the (V+1) -simplex. Let a sequence \mathbf{x}=(\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(L)})\in\mathcal{X}^{L} , where \mathbf{x}^{(i)} denotes the i -th token in a sequence. We denote a noised sequence by \mathbf{z}\in\mathcal{Z}^{L} and, at time t , by \mathbf{z}_{t}\in\mathcal{Z}_{t}^{L} , where \mathcal{Z}:=\mathcal{Z}_{t}:=\mathcal{X}\cup\{\mathbf{m}\} (we keep the subscript t to emphasize time). For vectors \mathbf{a},\mathbf{b} , \langle\mathbf{a},\mathbf{b}\rangle denotes the dot product. Masked diffusion language modeling (MDLM). In continuous-time masked diffusion modeling, the most representative work is MDLM ( 26 ) . MDLM defines the forward corruption process using the absorbing mask strategy: once a token is masked, it remains masked throughout the remaining process. For the diffusion process, define the time interval as t\in\mathcal{T}=[0,1] where we corrupt the data from t=0 (least noisy) to t=1 (most noisy). Formally, the forward process at time t is given as follows: q(\mathbf{z}_{t}^{(i)}\mid\mathbf{x})=q(\cdot\mid\mathbf{x}^{(i)})=\mathrm{Cat}\!\left(\cdot;\alpha_{t}\mathbf{x}^{(i)}+(1-\alpha_{t})\mathbf{m}\right), where the forward process gradually adds noise as t grows. In this regard, the noise scheduler should satisfy \alpha_{0}\approx 1 , \alpha_{1}\approx 0 , \alpha_{t}^{\prime}<0 , and is typically set to \alpha_{t}=1-t . Following 26 , discretize the time interval \mathcal{T} with T+1 steps, and define s(\tau)=\tau/(T+1) and t(\tau)=(\tau+1)/(T+1) such that generative distribution is divided into T diffusion reverse steps ( \mathbf{z}_{t(T)}\rightarrow\dots\rightarrow\mathbf{z}_{t(0)} ) and 1 reconstruction step ( \mathbf{z}_{t(0)}\rightarrow\mathbf{x} ). Then, the true reverse posterior can be derived as follows: \displaystyle q(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x})=q(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x}) \displaystyle=\begin{cases}\mathrm{Cat}(\mathbf{z}_{s}^{(i)};\mathbf{z}_{t}^{(i)}),&\text{if }\mathbf{z}_{t}^{(i)}\neq\mathbf{m},\\[6.0pt] \mathrm{Cat}\!\left(\mathbf{z}_{s}^{(i)};\tfrac{(1-\alpha_{s})\mathbf{m}+(\alpha_{s}-\alpha_{t})\mathbf{x}^{(i)}}{1-\alpha_{t}}\right),&\text{if }\mathbf{z}_{t}^{(i)}=\mathbf{m}.\end{cases} where we drop \tau in s(\tau) and t(\tau) for brevity. Hereafter, when the argument of a categorical distribution is clear from context, we omit it and write \mathrm{Cat}(\bm{\pi}) instead of \mathrm{Cat}(\cdot;\bm{\pi}) . To mimic the true reverse posterior, 26 propose a parametrized reverse process as follows: \displaystyle p_{\theta}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})=q\!\left(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x}=\mathbf{x}_{\theta}(\mathbf{z}_{t},t)\right) (1) \displaystyle=\begin{cases}\mathrm{Cat}(\mathbf{z}_{t}^{(i)}),&\text{if }\mathbf{z}_{t}^{(i)}\neq\mathbf{m},\\[6.0pt] \mathrm{Cat}\!\left(\tfrac{(1-\alpha_{s})\mathbf{m}+(\alpha_{s}-\alpha_{t})\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t)}{1-\alpha_{t}}\right),&\text{if }\mathbf{z}_{t}^{(i)}=\mathbf{m}.\end{cases} where sc_{2} such that both \alpha_{\phi} and \alpha_{\psi} satisfy the condition of free-form scheduler class, and NELBO is always finite by Proposition C.3 . We denote NormSig as the normalized Sigmoid defined on the overall sequence, i.e . , [\text{NormSig}(\mathbf{v})]_{i}=\sigma(\mathbf{v}_{i})-\sum_{j=1}^{L}\sigma(\mathbf{v}_{j})/L for vector \mathbf{v}\in\mathbb{R}^{L} . As shown in Figure 3 , each \phi and \psi is composed of 1 transformer layer followed by 1 MLP layer. This simple parametrization helps to learn generation order effectively, since g_{\phi} and g_{\psi} can be directly optimized toward reconstruction loss. Furthermore, normalization helps to forces overall velocity to be regularized, such that g_{\psi} and g_{\phi} just modulates relative generation order priority. See further detail of LoMDM parametrization in Appendix F.1 , 4.3 Training Algorithm In \mathcal{L}_{\mathrm{LoMDM{}}} (Eq. 3 ), we can see that \alpha_{\phi} is included in the expectation term. This means that we should sample \mathbf{z}_{t}\sim q_{\alpha_{\phi}}(\cdot\mid\mathbf{x}) , and it also requires gradient descent to be performed. In this section, we provide how we handle this. Gradient of \phi . The naive gradient estimator for \phi is: \displaystyle\nabla_{\phi}\mathcal{L}_{\text{LoMDM{}}}=\mathbb{E}_{t\sim\text{Uniform}([0,1])}\big[\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}(\mathcal{L}_{\text{main}}+\mathcal{L}_{\text{velocity}}) \displaystyle+\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\text{main}}+\mathcal{L}_{\text{velocity}})]\big]. However, the estimator \mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})] has high-variance in reinforcement learning perspective ( 30 ) . In this regard, we sample \mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}(\cdot|\mathbf{x}) independently and utilize a non-biased low variance estimator ( 14 ) as follows: \displaystyle\mathcal{L}_{\text{rloo}}=\frac{1}{2}\log\tfrac{q_{\alpha_{\phi}}(\mathbf{z}_{t}^{1}|\mathbf{x})}{q_{\alpha_{\phi}}(\mathbf{z}_{t}^{2}|\mathbf{x})}\Bigl(\operatorname{Sgd}(\mathcal{L}_{\mathbf{z}_{t}^{1}})-\operatorname{Sgd}(\mathcal{L}_{\mathbf{z}_{t}^{2}})\Bigr), (8) \displaystyle\nabla_{\phi}\mathcal{L}_{\text{LoMDM{}}}=\mathbb{E}_{t}[\mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\phi}(\tfrac{1}{2}(\mathcal{L}_{\mathbf{z}_{t}^{1}}+\mathcal{L}_{\mathbf{z}_{t}^{2}})+\mathcal{L}_{\text{rloo}})]], where \mathcal{L}_{\mathbf{z}_{t}^{i}} refers to \mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}} for \mathbf{z}_{t}^{i} . Here, i is different from (i) which indicates i -th token, i.e . \mathbf{z}_{t}^{i} itself is full sequence such that \dim(\mathbf{z}_{t}^{i})=(V+1)\times L . Further details are given in Appendix E.1 , and note that \mathcal{L}_{\mathrm{rloo}} is invariant to gradient of \phi and \theta . Training algorithm. Combining all above, we provide Algorithm 1 , which includes sampling \mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}(\cdot|\mathbf{x}) and directly optimize the single NELBO for three parameters. We have intentionally omitted the batch process to intensify readability, yet we sample t number of B//2 , and learn \mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2} for each t , so a total of B samples can be learn within one batch. Our training algorithm requires 1 more forward pass of \theta , and requires 2 more forward/backward passes of \phi,\psi (composed of 1 transformer/MLP layer) than conventional MDLM, so that the number of tokens seen per second is slightly lower than that of MDLM. However, we observed that our LoMDM substantially outperforms MDLM within the same trained hours, and we report these observations in following section. Algorithm 1 Training algorithm of LoMDM 1: while Until converge do 2: Sample \mathbf{x}\sim p(\mathbf{x}) and t\sim\mathrm{Uniform}([0,1]) 3: f\leftarrow\mathrm{Sgd}(\theta_{\text{TF}}(\mathbf{x})) 4: \alpha_{\phi}(\mathbf{x},t),\quad A_{\phi}(\mathbf{x},t)\leftarrow Eq. 4 and 5 using f 5: Sample \mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}(\mathbf{z}_{t}|\mathbf{x}) 6: for i\in\{0,1\} do 7: \mathbf{x}_{\theta}^{i}\leftarrow\mathrm{Softmax}\!\big(\theta_{\mathrm{MLP}}(\theta_{\mathrm{TF}}(\mathbf{z}_{t}^{i}))\big) 8: where \theta_{\mathrm{TF}}(\cdot) is cached 9: f_{\mathbf{z}^{i}}\leftarrow\textrm{Sgd}(\theta_{\text{TF}}(\mathbf{z}_{t}^{i})) 10: \hat{A}_{\psi}(\mathbf{z}_{t}^{i},t)\leftarrow Eq. 7 using f_{\mathbf{z}^{i}} 11: \mathcal{L}_{\mathbf{z}_{t}^{i}}\leftarrow Eq. 3 using A_{\phi},\hat{A}_{\psi},\mathbf{x}_{\theta} and \mathbf{x} 12: end for 13: \mathcal{L}_{\text{rloo}}\leftarrow Eq. 8 14: \mathcal{\widehat{L}}_{\mathrm{LoMDM{}}}\leftarrow\tfrac{1}{2}(\mathcal{L}_{\mathbf{z}_{t}^{1}}+\mathcal{L}_{\mathbf{z}_{t}^{2}})+\mathcal{L}_{\text{rloo}} 15: Perform gradient descent on \mathcal{\widehat{L}}_{\mathrm{LoMDM{}}} for \{\theta,\phi,\psi\} simultaneously 16: end while 5 Experimental results Table 1 : Test perplexities (PPL; \downarrow ) on LM1B and OpenWebText. Best diffusion value is bolded. ¶ Denotes the dataset didn’t incorporate sentence packing. For diffusion models, we report the bound on the likelihood. In GenMD4, ‡ denotes our trained model due to the absence of experiments. All diffusion models were trained with a batch size of 512, whereas reported PPL in OWT of GenMD4 ( 30 ) was that of model trained with a batch size of 1024, so that we marked it as \geq . Otherwise, reported values are imported from 28 . L^{\prime} in BD3LM refers to length of each block. LM1B ¶ LM1B OWT Autoregressive Transformer 22.3 22.8 † 17.5 Diffusion (Uniform-state / Gaussian) D3PM Uniform ( 2 ) - 137.9 - Diffusion-LM ( 15 ) - 118.6 - SEDD Uniform ( 16 ) 40.3 - 29.7 UDLM ( 29 ) 31.3 36.7 27.4 Duo ( 28 ) 29.9 33.7 25.2 Diffusion (Absorbing state) BERT-Mouth ( 33 ) - 142.9 - D3PM Absorb ( 2 ) - 76.9 - DiffusionBert ( 7 ) - 63.8 - SEDD Absorb ( 16 ) 32.7 - 24.1 MDLM ( 26 ) 27.0 31.8 23.2 Autoregressive + Diffusion BD3LM ( 1 ) L^{\prime} =16 - 30.6 22.3 L^{\prime} =8 - 29.8 21.7 L^{\prime} =4 - 28.2 20.7 Diffusion (Absorbing state + Learnable scheduler) GenMD4 ( 30 ) 26.9 ‡ 30.0 ‡ 21.8 ≥ LoMDM (Ours) 25.4 27.2 \mathbf{20.4} Table 2 : Zero-shot perplexities ( \downarrow ) of models trained for 1M steps on OpenWebText. All perplexities for diffusion models are upper bounds. † Taken from 1 . Otherwise, reported values are imported from 28 . Best diffusion values are bolded and diffusion values better than AR are underlined . PTB Wikitext LM1B Lambada AG News Pubmed Arxiv Autoregressive Transformer 82.05 25.75 51.25 51.28 52.09 49.01 41.73 Diffusion (Uniform-state / Gaussian) SEDD Uniform 105.51 41.10 82.62 57.29 82.64 55.89 50.86 Plaid 142.60 50.86 91.12 57.28 - - - UDLM 112.82 39.42 77.59 53.57 80.96 50.98 44.08 Duo 89.35 33.57 73.86 49.78 67.81 44.48 40.39 Diffusion (Absorbing state) SEDD Absorb 100.09 34.28 68.20 49.86 62.09 44.53 38.48 D3PM Absorb 200.82 50.86 138.92 93.47 - - - MDLM 95.26 32.83 67.01 47.52 61.15 41.89 37.37 Autoregressive + Diffusion BD3LM † (L^{\prime}=4) 96.81 31.31 60.88 50.03 61.67 42.52 39.20 Diffusion (Absorbing state + Learnable scheduler) LoMDM (Ours) 80.40 27.82 61.19 36.32 53.53 37.73 32.88 5.1 Main Results Experimental settings. We evaluate LoMDM following the widely adopted experimental settings in continuous-time discrete diffusion language modeling ( 28 ; 1 ) . We train LoMDM on three datasets, including One Billion Words dataset (LM1B) ( 3 ) with/without sentence packing and OpenWebText (OWT) ( 5 ) . We utilize three metrics: test perplexity, zero-shot test perplexity, and generative perplexity. Every discrete diffusion model, including ours are trained for 1M steps with a batch size of 512 unless specified. Note that we actually sample 256 texts for each batch since LoMDM utilizes a two-sample estimator, so the experimental setting is fair. Finally, we set c_{1}=0.7,c_{2}=0.65 for LoMDM for every dataset. Further details are provided in Appendix G.1 , and ablation study on c_{1},c_{2} is given in Appendix F.2 Likelihood evaluation. On LM1B and OWT (Table 1 ), our LoMDM outperforms every other discrete diffusion models with large margin. For LM1B with sentence packing and OWT, LoMDM achieves PPL values that are more than 3 points lower than those of MDLM. Furthermore, we observe that LoMDM achieves lower PPL across all benchmarks than BD3LM even when BD3LM uses a block size of L^{\prime}=4 , which injects an almost autoregressive L2R bias. Finally, for GenMD4, which also learns the scheduler like our method, LoMDM outperforms in every benchmark. Notably, in OWT, even the batch size is set to 1024 for GenMD4 meaning it sees twice as many training samples as LoMDM, yet our LoMDM achieves much lower PPL. Zero-shot likelihood evaluation. We measure the zero-shot generalization of the models trained on OWT by evaluating their PPL on 7 other datasets. Following 26 ; 28 , our zero-shot datasets include the validation splits of Penn Tree Bank (PTB; 17 ), WikiText ( 18 ) , LM1B, Lambada ( 21 ) , AG News ( 36 ) , and Scientific papers from ArXiv and Pubmed ( 4 ) . We observe that LoMDM achieves a lower PPL than MDLM on 7/7, notably, outperforming an autoregressive transformer on 4/7 datasets, and state-of-the-art among all discrete diffusion models on 6/7 datasets. In addition, 30 report that their GenMD4 model trained on OWT exhibited a bias toward certain words, which in turn degraded its zero-shot PPL; for this reason, zero-shot PPL metric was not reported. Generative perplexity. Table 3 : Generative PPL ( \downarrow ) of models trained on OWT, computed by a pre-trained GPT-2 Large on 256 generated samples with length of 1024. LoMDM uses matched train–inference scheduling at test time is marked as bold. † denotes no-scheduler ablations where the scheduler effect is disabled at inference by setting c_{2}{=}0 . #NFE 128 256 512 1024 MDLM 116.71 79.43 55.50 42.56 LoMDM ( c_{1}{=}0.7,\,c_{2}{=}0.65 ) 92.78 73.98 48.29 38.87 Ablation study LoMDM† ( c_{1}{=}0.7,\,c_{2}{=}0 ) 107.95 78.05 59.34 44.42 LoMDM† ( c_{1}{=}1,\,c_{2}{=}0 ) 122.48 83.68 59.88 48.30 Finally, we test the effectiveness of LoMDM for the quality of generated texts. We employed ancestral sampling ( 26 ) with reverse transition p_{\theta,\hat{\alpha}_{\psi}}(\mathbf{z}_{s}|\mathbf{z}_{t}) . As shown in Table 3 , our LoMDM outperforms the MDLM baseline with a large margin, which indicates our velocity \hat{A}_{\psi} actually improves the text generation ability of MDM. Furthermore, LoMDM achieves lower generative PPL across various NFE settings, indicating that it improves text generation quality while retaining the fast generation enabled by parallel decoding. We further conduct an ablation study to isolate the effect of the learned scheduler at inference time. While keeping the training setup fixed at (c_{1},c_{2})=(0.7,0.65) for all models, we disable the scheduler effect during generation by setting c_{2}=0 (Table 3 , †). This consistently degrades the generative PPL compared to the matched train–inference setting, indicating that the learned velocity \hat{A}_{\psi} provides a beneficial generation path rather than merely acting as a training-time regularizer. 5.2 Training Dynamics of LoMDM Figure 4 : Pearson correlation per training step for LoMDM trained on OWT. We report correlations among A_{\phi}^{(i)}(\mathbf{x},t) , \hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t) , and \langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle . When measuring correlation with \langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle , we compute it only over masked positions in \mathbf{z}_{t} , since \mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t) is zero at unmasked positions. Correlation between \theta , \phi , \psi . To understand how the learned scheduler interacts with the diffusion backbone, we track Pearson correlations between the forward velocity A_{\phi}^{(i)}(\mathbf{x},t) , the reverse-time velocity predicted by \psi , \hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t) , and the backbone’s token-level reconstruction confidence toward ground truth \langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle . Figure 4 shows that A_{\phi}^{(i)}(\mathbf{x},t) quickly becomes positively correlated with reconstruction confidence and remains stable throughout training. This suggests that the learned \phi assigns higher denoising emphasis to tokens that the backbone \theta can reconstruct more reliably at the current time t . We also observe a positive correlation between \hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t) and reconstruction confidence, indicating that \psi learns to infer which masked tokens are currently easier to predict from the partially observed context \mathbf{z}_{t} . Finally, the strong correlation between A_{\phi} and \hat{A}_{\psi} supports that the velocity-matching term \mathcal{L}_{\mathrm{velocity}} effectively aligns the forward and reverse velocities. LoMDM is a fast and efficient learner. Figure 5 : Test PPL per wall-clock-time during training on OWT. We truncate the curves at the point where LoMDM matches the 1M-step MDLM performance (PPL = 23.0). At this cutoff, MDLM had reached PPL = 24.9 with \sim 0.30M steps, while our method had reached PPL = 23.0 with \sim 0.18M steps. Figure 5 reports test PPL on OWT with wall-clock time, where the curves are truncated when LoMDM reaches the 1M-step MDLM reference performance (PPL=23.0). Notably, LoMDM attains this target after only 0.18M steps, compared to 1M steps for MDLM, indicating that comparable performance can be achieved after seeing only \sim 18% of tokens. Moreover, across the shared wall-clock budget in Figure 5 , LoMDM consistently achieves lower PPL than MDLM, suggesting that it learns faster and more efficiently in practice. 6 Conclusion We introduced OeMDM, a unified framework of masked diffusion models with various orderings, which treats the noise scheduler as a minimal-constrained and position-dependent object and yields a generalized NELBO. Building on this framework, we proposed LoMDM, which jointly learns a sequence-dependent scheduler and the diffusion backbone through a single NELBO, so that the learned scheduler is directly exploitable at generation time and concentrates learning on more tractable prediction paths. Empirically, LoMDM achieves substantially lower test perplexities than prior discrete diffusion baselines. Moreover, on OWT, LoMDM matches the 1M-step MDLM at only 180K steps, highlighting that LoMDM is a fast and efficient learner. Impact Statements The objective of our work is to advance the discrete diffusion-based language modeling. Potential societal consequences are similar to those of other text generation methods, and we do not anticipate impacts beyond those already well established for generative language models. Appendix A Related works Discrete diffusion models. Diffusion probabilistic models have become a dominant approach for continuous domains such as images, audio, and video ( 8 ; 32 ) . This success has motivated extensions to discrete domains, including text, leading to discrete diffusion models ( 2 ; 10 ) . The forward corruption process is typically designed in one of two ways: (i) uniform corruption, which replaces tokens with random vocabulary elements ( 16 ; 28 ) , or (ii) masking -based corruption, which maps tokens to an absorbing [MASK] state ( 26 ; 30 ) . Notably, 16 propose SEDD, a continuous-time formulation based on a score-entropy objective that accommodates both uniform and masking corruptions. Continuous-time masked diffusion models. Masked diffusion models (MDMs) form a widely used subclass of discrete diffusion for text generation, built around an absorbing mask state ( 16 ; 26 ; 30 ) . Empirically, masking-based formulations often yield stronger likelihood bounds than their uniform-corruption counterparts, and they have become a common baseline for discrete diffusion language modeling. In the continuous-time setting, a representative instantiation is MDLM ( 26 ) , which defines an absorbing forward process where once a position becomes masked, it remains masked thereafter. This design substantially simplifies the forward dynamics and enables a principled continuous-time training objective based on an NELBO derivation. Time-agnostic masked diffusion and its relation to continuous-time formulations. Beyond continuous-time formulations that rely on explicit time-conditioned schedulers, several works study time-agnostic training for masking-based diffusion models and show that the resulting objectives can be implemented with simple cross-entropy losses ( 37 ; 20 ) . The key intuition is that a scheduler primarily determines the fraction of masked tokens over time; hence many scheduler-dependent terms can be rewritten as functions of the current mask ratio, yielding a practically convenient framework that is widely adopted in recent large-scale MDM implementations ( 35 ; 19 ) . Importantly, the distinction between time-agnostic and continuous-time MDMs does not lie in whether the denoiser is time-conditioned: many continuous-time MDMs, including MDLM, BD3LM, and VADD, keep the denoiser itself time-agnostic ( 26 ; 1 ; 34 ) . Rather, the difference is whether the diffusion process is specified via an explicit time-conditioned scheduler. 37 derives a scheduler-free NELBO and proposes the First-Hitting sampler, which enables sampling without an explicit scheduler by repeatedly selecting random positions to update. However, time-agnostic formulations effectively assume that the corruption strength is identical across positions and make time implicit via the number (or ratio) of masked tokens. As a result, constructing order-dependent, position-varying MDMs naturally calls for revisiting the continuous-time formulation with explicit, time-conditioned schedulers. Importance of generation order in masked diffusion models. A central determinant of MDM generation quality is the unmasking order at inference time. Prior studies have repeatedly observed that naive random unmasking can be suboptimal ( 20 ; 31 ; 12 ) . In continuous-time MDMs, BD3LM ( 1 ) injects a block-wise left-to-right bias and substantially improves over MDLM. Similar phenomena appear in large-scale time-agnostic MDMs: when using random ordering, large-scale MDMs ( 35 ; 19 ) often lag behind large-scale ARMs such as LLaMA-3 ( 6 ) . However, adopting structured decoding—e.g., block-wise L2R generation, within each block, revealing the highest-confidence positions first—can close this gap and even surpass ARMs in some settings. Overall, these results highlight that the choice of unmasking order matters in MDMs. Learning unmasking order in masked diffusion models. In the continuous-time MDMs setting, 23 introduce a planner, an unmasking module, and post-train this planner to improve a fixed diffusion backbone. 30 propose GenMD4, which defines vocabulary-dependent learnable schedulers. In contrast to ours, such vocabulary-dependent designs primarily capture global token preferences (e.g., whether certain token types tend to be sampled earlier), rather than adapting the ordering to the specifi sentence context. In the time-agnostic MDMs setting, learning unmasking strategies has been actively explored due to the simplicity of time-agnostic training objectives ( 9 ; 24 ; 11 ) . 24 propose PAPL, which integrates a planner into the time-agnostic MDM framework, and derive a corresponding NELBO. However, optimizing the NELBO with an explicit planner involves sampling along the Markov chain (i.e., generating intermediate states up to t=0 and an intermediate time t^{\prime} ). To avoid this infeasible sampling cost, they replace the planner with the diffusion network’s confidence scores and instead train a surrogate objective that samples text from a uniform distribution. 9 propose post-training an unmasking policy module using GRPO, and prove that the resulting policy can sample closer to the true data distribution. Summary. MDMs are a promising alternative to ARMs, but their generation quality depends strongly on the unmasking order. However, existing approaches typically 1) improve the order only via post-training ( 9 ; 23 ) , 2) rely on surrogate objectives ( 24 ) , or 3) fail to fully capture context-dependent ordering over the entire sequence ( 30 ) . To address these limitations, we return to the canonical continuous-time MDLM formulation and focus on the scheduler as the central mechanism that governs the unmasking process. This perspective enables a principled integration of MDM training with learnable, context-dependent generation orders within a unified theoretical framework. Appendix B Preliminaries and Notation We first introduce basic objects, masked sequences, absorbing mask (monotone-masking) trajectories, and the corresponding path measures that will be used throughout the appendix. B.1 Masked sequences and absorbing trajectories Definition B.1 (Masked sequence set) . For a length- L sequence \mathbf{x}\in\mathcal{X}^{L} , define the masked sequence set as \displaystyle\mathcal{S}(\mathbf{x}):=\Bigl\{\mathbf{z}\in(\mathcal{X}\cup\{\mathbf{m}\})^{L}\ \big|\ \mathbf{z}^{(i)}\in\{\mathbf{x}^{(i)},\mathbf{m}\},\ \forall i\in[L]\Bigr\}. (9) Definition B.2 (Absorbing mask trajectory) . Fix an endpoint \mathbf{x}\in\mathcal{X}^{L} and a discrete time grid t(0)<\cdots0\quad\Longrightarrow\quad q_{\alpha_{\mathcal{F}}}\!\left(\mathbf{z}_{t(0:T)}\mid\mathbf{x}\right)>0, (11) or equivalently p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})\ll q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x}) for every fixed \mathbf{x} . Proof. We divide the proof in two steps: 1) we first show that q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})>0 for all \mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) , and 2) prove that p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0 implies \mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) . 1. q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})>0,\forall\mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) . Consider any \mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) . Since \mathbf{z}_{t(T)}=\mathbf{m}^{L} by definition, q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(T)}|\mathbf{x})=1 by definition. We now show that for any \mathbf{z}_{s(\tau)}(=\mathbf{z}_{t(\tau-1)} ) and \mathbf{z}_{t(\tau)} from path \mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) , the transition q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}|\mathbf{z}_{t(\tau)},\mathbf{x}) is positive. There are three cases of \mathbf{z}_{s(\tau)}^{(i)},\mathbf{z}_{t(\tau)}^{(i)} : 1) \mathbf{z}_{s(\tau)}^{(i)}=\mathbf{m},\mathbf{z}_{t(\tau)}^{(i)}=\mathbf{m} , 2) \mathbf{z}_{s(\tau)}^{(i)}=\mathbf{x}^{(i)},\mathbf{z}_{t(\tau)}^{(i)}=\mathbf{m} , and 3) \mathbf{z}_{s(\tau)}^{(i)}=\mathbf{x}^{(i)},\mathbf{z}_{t(\tau)}^{(i)}=\mathbf{x}^{(i)} . To recap, q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x}) is given by \displaystyle q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x})=q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})=\begin{cases}\mathrm{Cat}(\mathbf{z}_{t}^{(i)}),&\text{if }\mathbf{z}_{t}^{(i)}\neq\mathbf{m},\\ \mathrm{Cat}\left(\tfrac{(1-\alpha_{\mathcal{F}}^{(i)}(u,s))\mathbf{m}+(\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t))\mathbf{x}^{(i)}}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\right),&\text{if }\mathbf{z}_{t}^{(i)}=\mathbf{m},\end{cases} Since \alpha_{\mathcal{F}}\in\mathcal{F}[I] , for any fixed input the map t\mapsto\alpha_{\mathcal{F}}^{(i)}(u,t) is strictly decreasing with boundary values \alpha_{\mathcal{F}}^{(i)}(u,0)=1 and \alpha_{\mathcal{F}}^{(i)}(u,1)=0 . Hence, for any 0\leq s\alpha_{\mathcal{F}}^{(i)}(u^{(i)},t) , and for any t\in(0,1] we have 0<\alpha_{\mathcal{F}}^{(i)}(u^{(i)},t)<1 . Therefore, all of three cases are strictly positive, such that for any \mathbf{z}_{s(\tau)}(=\mathbf{z}_{t(\tau-1)}) and \mathbf{z}_{t(\tau)} from path \mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) , q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}|\mathbf{z}_{t(\tau)},\mathbf{x}) is positive, and so is their product q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x}) . 2. p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0\Rightarrow\mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) . Suppose p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0 . Then, in particular, all factors in the joint are positive. First, since the initial noise distribution is a fully-masked sequence, \mathbf{z}_{t(T)}=\mathbf{m}^{L} . Next, carry-over unmasking implies that if \mathbf{z}_{t}^{(i)}\neq\mathbf{m} , then \mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t)=\mathbf{z}_{t}^{(i)} , so the reverse transition copies that token with probability 1 . Formally, \displaystyle\Big(\mathbf{z}_{t(\tau+1)}^{(i)}\neq\mathbf{m}\;\Rightarrow\;\mathbf{z}_{t(\tau)}^{(i)}=\mathbf{z}_{t(\tau+1)}^{(i)}\Big)\iff\Big(\mathbf{z}_{t(\tau+1)}^{(i)}\neq\mathbf{m}\;\Rightarrow\;\mathbf{z}_{t(\tau)}^{(i)}\neq\mathbf{m}\Big)\iff\Big(\mathbf{z}_{t(\tau)}^{(i)}=\mathbf{m}\;\Rightarrow\;\mathbf{z}_{t(\tau+1)}^{(i)}=\mathbf{m}\Big) satisfies for every trajectory from Markov process starting from \mathbf{z}_{t(T)}=\mathbf{m}^{L} . Therefore, if p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0 , the trajectory \mathbf{z}_{t(0:T)} satisfies absorbing mask property, and \mathbf{z}_{t(\tau)}^{(i)}\in\{\mathbf{m},\mathbf{v}\} for every \tau\in[T] and fixed one-hot vector \mathbf{v}\in\mathcal{V} . Finally, if there remains mask token in \mathbf{z}_{t(0)} , the variational distribution p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}^{(i)}|\mathbf{z}_{t(0)})=\text{Cat}\Big(\mathbf{x}^{(i)}_{\theta}\big(\mathbf{z}_{t(0)},t(0)\big)\Big) converts it into non-mask token by zero masking property of SUBS parametrization, such that resulting \mathbf{x} is composed of \mathcal{V}^{L} . Since carry-over unmasking property of SUBS parametrization holds for every transition, we can conclude that \mathbf{z}_{t(\tau)}^{(i)}\in\{\mathbf{m},\mathbf{x}^{(i)}\} for every \tau\in[T] and final state \mathbf{x}\in\mathcal{V}^{L} . Since \mathbf{z}_{t(\tau)}^{(i)}\in\{\mathbf{x}^{(i)},\mathbf{m}\} for all i,\tau , and the trajectory is absorbing, p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0\Rightarrow\mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) holds. Since q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})>0,\forall\mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) and p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})>0\Rightarrow\mathbf{z}_{t(0:T)}\in\mathcal{S}_{\mathrm{absorb}}(\mathbf{x}) hold, the given statement p_{\theta,\hat{\alpha}_{\mathcal{F}}}\!\left(\mathbf{x},\mathbf{z}_{t(0:T)}\right)>0\quad\Longrightarrow\quad q_{\alpha_{\mathcal{F}}}\!\left(\mathbf{z}_{t(0:T)}\mid\mathbf{x}\right)>0 hold for every fixed \mathbf{x} . ∎ With Lemma C.1 , we now derive NELBO of OeMDM. To recap, NELBO is given as follows: Proposition 3.2 (NELBO of OeMDM in continuous time) . Under SUBS parametrization, the NELBO of OeMDM in continuous time is given as follows: \displaystyle-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x})\leq\mathcal{L}_{\mathrm{OeMDM{}}}(\mathbf{x},\theta,\alpha_{\mathcal{F}},\hat{\alpha}_{\mathcal{F}}) \displaystyle=\int_{0}^{1}\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\Biggl\{\underbrace{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle}_{\mathcal{L}_{\mathrm{main}}}+\underbrace{A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})}_{\mathcal{L}_{\mathrm{velocity}}}\Biggl\}\Biggr]dt, where the structure of \mathcal{L}_{\mathrm{main}} is equal to \mathcal{L}_{\mathrm{mdlm}} and \mathcal{L}_{\mathrm{velocity}}\geq 0 achieves 0 when A=\hat{A} . Following 26 , discretize the time interval \mathcal{T} with T+1 steps, and define s(\tau)=\tau/(T+1) and t(\tau)=(\tau+1)/(T+1) such that generative distribution is divided into T diffusion reverse steps ( \mathbf{z}_{t(T)}\rightarrow\dots\rightarrow\mathbf{z}_{t(0)} ) and 1 reconstruction step ( \mathbf{z}_{t(0)}\rightarrow\mathbf{x} ). The negative evidence lower bound (NELBO) can be obtained as follows: \displaystyle-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}) \displaystyle=-\log\sum_{\mathbf{z}_{t(0:T)}}p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)}) (12) \displaystyle=-\log\sum_{\mathbf{z}_{t(0:T)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})\frac{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})}{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})}\hskip 125.19194pt\because\text{Lemma~\ref{lem:absolute_continuity_OeMDM{}}} (13) \displaystyle=-\log\mathbb{E}_{\mathbf{z}_{t(0:T)}\sim q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}|\mathbf{x})}\bigg[\frac{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})}{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})}\bigg] (14) \displaystyle\leq-\mathbb{E}_{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}|\mathbf{x})}\bigg[\log\frac{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x},\mathbf{z}_{t(0:T)})}{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(0:T)}\mid\mathbf{x})}\bigg]\hskip 113.81102pt\because\text{Jensen's inequality} (15) \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\!\Biggl[\underbrace{-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}\mid\mathbf{z}_{t(0)})}_{\mathcal{L}_{\text{reconstruction}}}+\underbrace{\sum_{\tau=1}^{T}D_{\mathrm{KL}}\!\bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)},\mathbf{x})\,\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)})\bigr)}_{\mathcal{L}_{\text{diffusion}}^{T}} \displaystyle\hskip 113.81102pt+\underbrace{D_{\mathrm{KL}}\!\bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(T)}\mid\mathbf{x})\,\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{t(T)})\bigr)}_{\mathcal{L}_{\text{prior}}}\Biggr]. (16) Hereafter, we omit \tau in t(\tau) and s(\tau) for brevity. We first derive \mathcal{L}_{\text{diffusion}}^{\infty}=\lim_{T\rightarrow\infty}\mathcal{L}_{\text{diffusion}}^{T} . C.1 Breaking Sequence-Level Diffusion Loss into Token-Level Diffusion Loss We breakdown D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right) into token-wsie KL divergence: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right) \displaystyle=\sum_{\mathbf{z}_{s}}\biggl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})}\biggr) \displaystyle=\sum_{\mathbf{z}_{s}}\biggl(\Bigl(\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)\log\frac{\prod_{i=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{\prod_{i=1}^{L}p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\biggr)\quad\because\begin{cases}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})=\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x}),\\ p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})=\prod_{i=1}^{L}p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})\end{cases} \displaystyle=\sum_{\mathbf{z}_{s}}\biggl(\Bigl(\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)\sum_{i=1}^{L}\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\biggr) \displaystyle=\sum_{\mathbf{z}_{s}}\biggl(\sum_{i=1}^{L}\biggl(\Bigl(\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\biggr)\biggr) \displaystyle=\sum_{i=1}^{L}\sum_{\mathbf{z}_{s}^{(i)}}\;\sum_{\mathbf{z}_{s}^{(1)}}\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\sum_{\mathbf{z}_{s}^{(i+1)}}\cdots\sum_{\mathbf{z}_{s}^{(L)}}\biggl(\Bigl(\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\biggr) \displaystyle=\sum_{i=1}^{L}\sum_{\mathbf{z}_{s}^{(i)}}\;\sum_{\mathbf{z}_{s}^{(1)}}\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\sum_{\mathbf{z}_{s}^{(i+1)}}\cdots\sum_{\mathbf{z}_{s}^{(L)}}\biggl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\Bigl(\prod_{j\neq i}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\biggr) \displaystyle=\sum_{i=1}^{L}\sum_{\mathbf{z}_{s}^{(i)}}\biggl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})}\underbrace{\sum_{\mathbf{z}_{s}^{(1)}}\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\sum_{\mathbf{z}_{s}^{(i+1)}}\cdots\sum_{\mathbf{z}_{s}^{(L)}}\Bigl(\prod_{j\neq i}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr)}_{=1\text{ by (*)}}\biggr) \displaystyle=\sum_{i=1}^{L}\sum_{\mathbf{z}_{s}^{(i)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})} \displaystyle=\sum_{i=1}^{L}D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})\right). where (*) holds by marginalization (or sum of product): \displaystyle\sum_{\mathbf{z}_{s}^{(1)}}\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\sum_{\mathbf{z}_{s}^{(i+1)}}\cdots\sum_{\mathbf{z}_{s}^{(L)}}\Bigl(\prod_{j\neq i}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(j)}\mid\mathbf{z}_{t}^{(j)},\mathbf{x})\Bigr) \displaystyle=\sum_{\mathbf{z}_{s}^{(1)}}\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\sum_{\mathbf{z}_{s}^{(i+1)}}\cdots\sum_{\mathbf{z}_{s}^{(L)}}\Bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(1)}\mid\mathbf{z}_{t}^{(1)},\mathbf{x})\cdots q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i-1)}\mid\mathbf{z}_{t}^{(i-1)},\mathbf{x}) \displaystyle\hskip 119.50148pt\cdot q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i+1)}\mid\mathbf{z}_{t}^{(i+1)},\mathbf{x})\cdots q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(L)}\mid\mathbf{z}_{t}^{(L)},\mathbf{x})\Bigr) \displaystyle=\sum_{\mathbf{z}_{s}^{(1)}}\Bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(1)}\mid\mathbf{z}_{t}^{(1)},\mathbf{x})\sum_{\mathbf{z}_{s}^{(2)}}\Bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(2)}\mid\mathbf{z}_{t}^{(2)},\mathbf{x})\cdots \displaystyle\hskip 147.95424pt\cdots\sum_{\mathbf{z}_{s}^{(i-1)}}\Bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i-1)}\mid\mathbf{z}_{t}^{(i-1)},\mathbf{x})\sum_{\mathbf{z}_{s}^{(i+1)}}\Bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i+1)}\mid\mathbf{z}_{t}^{(i+1)},\mathbf{x})\cdots \displaystyle\hskip 216.2411pt\cdots\sum_{\mathbf{z}_{s}^{(L)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(L)}\mid\mathbf{z}_{t}^{(L)},\mathbf{x})\Bigr)\cdots\Bigr)\cdots\Bigr) \displaystyle=\Bigl(\sum_{\mathbf{z}_{s}^{(1)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(1)}\mid\mathbf{z}_{t}^{(1)},\mathbf{x})\Bigr)\cdots\Bigl(\sum_{\mathbf{z}_{s}^{(i-1)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i-1)}\mid\mathbf{z}_{t}^{(i-1)},\mathbf{x})\Bigr) \displaystyle\hskip 62.59596pt\cdot\Bigl(\sum_{\mathbf{z}_{s}^{(i+1)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i+1)}\mid\mathbf{z}_{t}^{(i+1)},\mathbf{x})\Bigr)\cdots\Bigl(\sum_{\mathbf{z}_{s}^{(L)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(L)}\mid\mathbf{z}_{t}^{(L)},\mathbf{x})\Bigr)=1\cdots 1=1, where \sum_{\mathbf{z}_{s}^{(\ell)}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(\ell)}\mid\mathbf{z}_{t}^{(\ell)},\mathbf{x})=1 for every \ell since the definition of q is categorical distribution. C.2 Deriving Token-level KL into Closed-Form Equation From the previous decomposition, the total KL can be written as the sum of token-wise terms: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right)=\sum_{i=1}^{L}D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})\right). (17) For each token i , since \mathbf{z}_{t}^{(i)}\in\{\mathbf{x}^{(i)},\mathbf{m}\} , we can separate the two cases as: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t})\right) \displaystyle=D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)})\right)\!\langle\mathbf{z}_{t}^{(i)},\mathbf{x}^{(i)}\rangle \displaystyle\quad+D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right)\!\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle. (18) For breaking the KL term, we denote elements of the arbitrary input domain \mathcal{I} and \mathcal{\hat{I}} as u and \hat{u} respectively, i.e . , u\in\mathcal{I} and \hat{u}\in\mathcal{\hat{I}} . Case 1: \mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)} . In this case, both posteriors collapse to the same categorical atom: q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)},\mathbf{x})=p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)})=\mathrm{Cat}(\mathbf{z}_{s}^{(i)};\mathbf{x}^{(i)}), thus \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{x}^{(i)})\right)=0. (19) Case 2: \mathbf{z}_{t}^{(i)}=\mathbf{m} . Using the definitions of the forward and reverse posteriors defined earlier, we have: \displaystyle q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x}) \displaystyle=\frac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}, (20) \displaystyle q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x}) \displaystyle=\frac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}, (21) \displaystyle p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m}) \displaystyle=\frac{(\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))\left\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\right\rangle}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}, (22) \displaystyle p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m}) \displaystyle=\frac{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}. (23) We now directly compute the KL divergence: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) (24) \displaystyle=\sum_{\mathbf{z}_{s}^{(i)}\in\{\mathbf{x}^{(i)},\mathbf{m}\}}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})}. (25) Explicitly expanding both terms: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) \displaystyle=q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})} \displaystyle\quad+q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\log\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})}. (26) Now substitute each probability definition. For the first ratio: \displaystyle\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{x}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})} \displaystyle=\frac{\dfrac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}}{\dfrac{(\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))\left\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\right\rangle}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}} \displaystyle=\frac{(\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))\left\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\right\rangle}. (27) For the second ratio: \displaystyle\frac{q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})}{p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}=\mathbf{m}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})} \displaystyle=\frac{\dfrac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}}{\dfrac{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}} \displaystyle=\frac{(1-\alpha_{\mathcal{F}}^{(i)}(u,s))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s))}. (28) Substituting Eq. 27 and Eq. 28 into Eq. 26 , we have: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) \displaystyle={\color[rgb]{1,0,0}\frac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\log\frac{(\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle}} \displaystyle\hskip 199.16928pt+{\color[rgb]{0,0,1}\frac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\log\frac{(1-\alpha_{\mathcal{F}}^{(i)}(u,s))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s))}}. (29) Finally, since this applies only for masked tokens ( \mathbf{z}_{t}^{(i)}=\mathbf{m} ), the overall KL divergence can be expressed as: \displaystyle D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right)=\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\;D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right). (30) C.3 Diffusion Loss into Tractable Loss with Infinite Discretization Steps The diffusion loss term is \displaystyle\mathcal{L}_{\text{diffusion}}^{T} \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\bigg[\sum_{\tau=1}^{T}D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)})\right)\bigg] (31) \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\bigg[T\cdot\sum_{\tau=1}^{T}\frac{1}{T}D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s(\tau)}\mid\mathbf{z}_{t(\tau)})\right)\bigg] (32) \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\bigg[T\mathbb{E}_{t\in\{\frac{2}{T+1},\frac{3}{T+1},\dots,1\}}\Big[D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right)\Big]\bigg] (33) \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\bigg[\mathbb{E}_{t\in\{\frac{2}{T+1},\frac{3}{T+1},\dots,1\}}\Big[T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}\mid\mathbf{z}_{t})\right)\Big]\bigg] (34) \displaystyle=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\bigg[\mathbb{E}_{t\in\{\frac{2}{T+1},\frac{3}{T+1},\dots,1\}}\Big[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\,T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right)\Big]\bigg] (35) \displaystyle=\mathbb{E}_{t\in\{\frac{2}{T+1},\frac{3}{T+1},\dots,1\}}\bigg[\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Big[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\,T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right)\Big]\bigg] (36) We will now transform T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) into tractable loss when T\rightarrow\infty that corresponds to continuous time. Define A_{T} , \hat{A}_{T} as follows: \displaystyle A_{T}(u,t)\coloneqq\frac{\alpha_{\mathcal{F}}^{(i)}(u,t-\Delta)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{\Delta(1-\alpha_{\mathcal{F}}^{(i)}(u,t))},\qquad\hat{A}_{T}(\hat{u},t)\coloneqq\frac{\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t-\Delta)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}{\Delta(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}, where \Delta=\frac{1}{T+1} . Since \alpha_{\mathcal{F}}^{(i)},\hat{\alpha}_{\mathcal{F}}^{(i)} is AC([0,1]) by definition, \displaystyle\alpha_{\mathcal{F}}^{(i)}(u,t-\Delta)-\alpha_{\mathcal{F}}^{(i)}(u,t)=-\int_{t-\Delta}^{t}\partial_{r}\alpha^{(i)}_{\mathcal{F}}(u,r)dr\quad\Rightarrow\quad\lim_{T\rightarrow\infty}\frac{\alpha_{\mathcal{F}}^{(i)}(u,t-\Delta)-\alpha^{(i)}_{\mathcal{F}}(u,t)}{\Delta}=-\partial_{t}\alpha^{(i)}_{\mathcal{F}}(u,t)\quad\text{a.e.} such that \displaystyle\lim_{T\rightarrow\infty}A_{T}^{(i)}(u,t)=A^{(i)}(u,t)=\frac{-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)},\qquad t\in(0,1] (37) and also \lim_{T\rightarrow\infty}\hat{A}_{T}^{(i)}(\hat{u},t)=\hat{A}^{(i)}(\hat{u},t) . First term expansion of Eq. 29 . The first term of Eq. 29 becomes \displaystyle{\color[rgb]{1,0,0}\frac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\log\frac{(\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s)-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle}} (38) \displaystyle=\Delta A_{T}^{(i)}(u,t)\Big(\log A_{T}^{(i)}(u,t)-\log\hat{A}_{T}^{(i)}(\hat{u},t)-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle\Big). (39) Second term expansion of Eq. 29 . Similarly, the second term of Eq. 29 becomes \displaystyle{\color[rgb]{0,0,1}\frac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\log\frac{(1-\alpha_{\mathcal{F}}^{(i)}(u,s))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t))}{(1-\alpha_{\mathcal{F}}^{(i)}(u,t))(1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},s))}}=(1-\Delta A_{T}^{(i)}(u,t))\Big(\log(1-\Delta A_{T}^{(i)}(u,t))-\log(1-\Delta\hat{A}_{T}^{(i)}(\hat{u},t))\Big) (40) where \displaystyle\log(1-\Delta A_{T}^{(i)}(u,t))-\log(1-\Delta\hat{A}_{T}^{(i)}(\hat{u},t)=-\frac{\Delta}{1-\Delta\zeta_{T}}(A_{T}^{(i)}(u,t)-\hat{A}_{T}^{(i)}(\hat{u},t)),\qquad\zeta_{T}\in(A_{T}^{(i)}(u,t),\hat{A}_{T}^{(i)}(\hat{u},t)), (41) holds by the mean value theorem. Combining both parts. With T\rightarrow\infty such that \Delta=\frac{1}{T+1}\rightarrow 0+ , summing Eq. 37 , Eq. 39 , Eq. 40 , and Eq. 41 , the infinitesimal KL for token i becomes \displaystyle\lim_{T\to\infty}T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) (42) \displaystyle=\lim_{T\to\infty}(\Delta\cdot T)\cdot\frac{1}{\Delta}D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right) (43) \displaystyle=-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big). (44) Final loss form. By Eq. 36 and Eq. 44 , \mathcal{L}_{\text{diffusion}}^{\infty}=\lim_{T\rightarrow\infty}\mathcal{L}_{\text{diffusion}}^{T} gives: \displaystyle\mathcal{L}_{\text{diffusion}}^{\infty} \displaystyle=\lim_{T\rightarrow\infty}\mathbb{E}_{t\in\{\frac{2}{T+1},\frac{3}{T+1},\dots,1\}}\bigg[\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Big[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\,T\cdot D_{\mathrm{KL}}\!\left(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}^{(i)}=\mathbf{m},\mathbf{x})\,\big\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{z}_{t}^{(i)}=\mathbf{m})\right)\Big]\bigg] \displaystyle=\lim_{\delta\rightarrow 0+}\int_{\delta}^{1}\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\} (45) \displaystyle=\lim_{\delta\rightarrow 0+}\int_{\delta}^{1}\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\Biggl\{\underbrace{\frac{\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle}_{\text{original MDLM loss}} (46) \displaystyle-\underbrace{\frac{\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\Biggl(\log\frac{-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}-\log\frac{-\partial_{t}\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}\Biggr)-\Biggl(\frac{-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}-\frac{-\partial_{t}\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}\Biggr)}_{\text{is minimized when ${\partial_{t}\alpha_{\mathcal{F}}(u,t)}/{(1-\alpha_{\mathcal{F}}(u,t))}={\partial_{t}\hat{\alpha}_{\mathcal{F}}(\hat{u},t)}/{(1-\hat{\alpha}_{\mathcal{F}}(\hat{u},t))}$}}\Biggr\}\Biggr]dt. (47) where A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\geq 0 holds and minimized to 0 when A=\hat{A} . Since A^{(i)}(t),\hat{A}^{(i)}(t) can be unbounded as t\to 0^{+} and the discrete-time objective samples from t\in\{\frac{2}{T+1},\frac{3}{T+1}\dots,1\} , we write as \lim_{\delta\to 0^{+}}\int_{\delta}^{1}(\cdot)\,dt ; we explain this detail in Appendix C.6 and prove in Appendix C.7 that the integrand is integrable near t=0 , so the limit is well-defined and finite. Remark. Why A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\geq 0 ? Fix an index i and write a:=A^{(i)}>0 and b:=\hat{A}^{(i)}>0 . Consider the function \displaystyle f(a;b):=a(\log a-\log b)-(a-b)=a\log\frac{a}{b}-a+b. (48) Let r:=\frac{a}{b}>0 . Then \displaystyle f(a;b)=b\big(r\log r-r+1\big). (49) Since b>0 , it suffices to show g(r):=r\log r-r+1\geq 0 for all r>0 . We have \displaystyle g^{\prime}(r)=\log r,\qquad g^{\prime\prime}(r)=\frac{1}{r}>0\ \ (r>0), (50) so g is strictly convex on (0,\infty) and its unique minimizer satisfies g^{\prime}(r)=0 , i.e., r=1 . Evaluating at r=1 gives g(1)=0 , hence g(r)\geq 0 for all r>0 , with equality if and only if r=1 . Therefore f(a;b)\geq 0 for all a,b>0 , and the minimum 0 is attained if and only if a=b , i.e., A^{(i)}=\hat{A}^{(i)} . C.4 Prior Loss Recall that the prior loss is given as follows: \displaystyle\mathcal{L}_{\text{prior}}=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}[D_{\mathrm{KL}}\!\bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(T)}\mid\mathbf{x})\,\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{t(T)})]\bigr) Since t(\tau)=(\tau+1)/(T+1) , t(T) becomes 1. Therefore, p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{t(T)}) just becomes prior distribution, that is, \displaystyle p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{t(T)}^{(i)})=p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{1}^{(i)})=\text{Cat}(\mathbf{m}). Furthermore, substituting t=1 into forward process q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}^{(i)}\mid\mathbf{x})=\mathrm{Cat}\!\left(\,\alpha_{\mathcal{F}}^{(i)}(\cdot,t)\mathbf{x}^{(i)}+(1-\alpha_{\mathcal{F}}^{(i)}(\cdot,t))\mathbf{m}\right) gives: \displaystyle q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{1}^{(i)}\mid\mathbf{x})=\text{Cat}(\mathbf{m}), where \alpha^{(i)}_{\mathcal{F}}(\cdot,1)=0 by the definition of free-form scheduler. Since the above equations hold for every i , prior loss becomes zero: \displaystyle\mathcal{L}_{\text{prior}}=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}[D_{\mathrm{KL}}\!\bigl(q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t(T)}\mid\mathbf{x})\,\|\,p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{t(T)})]\bigr)=0. (51) C.5 Reconstruction Loss Recall that the reconstruction loss is given as follows: \displaystyle\mathcal{L}_{\text{reconstruction}}=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}[-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}\mid\mathbf{z}_{t(0)})]. Since the expectation is conditioned on q_{\alpha_{\mathcal{F}}} , the sampling of \mathbf{z}_{t(0)} is conducted as follows: \displaystyle\mathbf{z}_{t(0)}^{(i)}\sim\mathrm{Cat}\!\left(\,\alpha_{\mathcal{F}}^{(i)}(\cdot,t(0))\mathbf{x}^{(i)}+(1-\alpha_{\mathcal{F}}^{(i)}(\cdot,t(0)))\mathbf{m}\right). Since t(0)=1/(T+1) and by definition of free-form scheduler, in the continuous time case with T\rightarrow\infty , \alpha_{\mathcal{F}} becomes 1: \displaystyle\lim_{T\rightarrow\infty}\alpha^{(i)}_{\mathcal{F}}(\cdot,t(0))=\alpha_{\mathcal{F}}^{(i)}(\cdot,0)=1, where \alpha_{\mathcal{F}[\mathcal{I}]}^{(i)}(u,\cdot)\in AC([0,1]) and \alpha_{\mathcal{F}[\mathcal{I}]}^{(i)}(u,0)=1 hold for all u\in\mathcal{I} and i\in[1,\dots,L] by definition. Therefore, \lim_{T\rightarrow\infty}\mathbf{z}_{t(0)}=\mathbf{x} holds: \displaystyle\mathbf{z}_{t(0)}^{(i)}\sim\lim_{T\rightarrow\infty}\mathrm{Cat}\!\left(\,\alpha_{\mathcal{F}}^{(i)}(\cdot,t(0))\mathbf{x}^{(i)}+(1-\alpha_{\mathcal{F}}^{(i)}(\cdot,t(0)))\mathbf{m}\right) \displaystyle\Rightarrow\mathbf{z}_{t(0)}^{(i)}\sim\text{Cat}(\mathbf{x}^{(i)})\Rightarrow\mathbf{z}_{t(0)}^{(i)}=\mathbf{x}^{(i)}. Furthermore, by carry-over unmasking of SUBS parametrization, p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}^{(i)}|\mathbf{z}_{t(0)})=p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}^{(i)}|\mathbf{x})=1 for all i\in[1,\dots,L] . Combining the above results, the reconstruction loss in continuous time becomes zero: \displaystyle\mathcal{L}_{\text{reconstruction}}=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}[-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x}\mid\mathbf{z}_{t(0)})]=0. (52) Note that MDLM ( 26 ) and BD3LM ( 1 ) follow same derivation process with \alpha_{\text{mdlm}}(t(0))=T/(T+1) . C.6 Final NELBO Objective Combining all the results derived above, the final NELBO objective in continuous time can be rewritten as: \displaystyle-\log p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{x})\leq\underbrace{\mathcal{L}_{\text{diffusion}}^{\infty}}_{Eq.~\ref{eq:final_diffusion_loss}}+\underbrace{\mathcal{L}_{\text{prior}}}_{=0\text{ from }Eq.~\ref{eq:final_prior_loss}}+\underbrace{\mathcal{L}_{\text{reconstruction}}}_{=0\text{ from}~Eq.~\ref{eq:final_reconstruction_loss}} (53) \displaystyle=\lim_{\delta\rightarrow 0+}\int_{\delta}^{1}\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}\Biggr]dt. (54) \displaystyle=\int_{0}^{1}\underbrace{\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}}_{\coloneqq\ell(t)}\Biggr]dt, (55) where A^{(i)}(\cdot,t) and \hat{A}^{(i)}(\cdot,t) are defined only for t\in(0,1] ; yet, at t=0 , the forward process is deterministic since \alpha_{\mathcal{F}}^{(i)}(u,0)=1 implies q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{0}\mid\mathbf{x})=\delta_{\mathbf{x}} , so that \langle\mathbf{z}_{0}^{(i)},\mathbf{m}\rangle=0 almost surely for all i\in[L] . Hence it is natural to extend the integrand to [0,1] by setting its value at t=0 to be 0 without defining A^{(i)}(0) or \hat{A}^{(i)}(0) . However, since the coefficient A^{(i)}(\cdot,t)=-\partial_{t}\alpha^{(i)}(\cdot,t)/(1-\alpha^{(i)}(\cdot,t)) can diverge as t\to 0+ when \alpha_{\mathcal{F}}(0)=1 , we will separately prove that \int_{0}^{1}\ell(t)\,dt<\infty in the following section. C.7 Finiteness of NELBO We show under what conditions the integral over [0,1] is finite. First, we can make a reasonable assumption 2 2 2 Since the composition of finite linear transformations and continuous activation functions results in a continuous mapping, a neural network with a finite number of neurons and finite weights will always produce a finite output for any finite input. that the output of the diffusion network \theta cannot take the values +\infty or -\infty . Furthermore, since \mathbf{x}_{\theta}^{(i)}(\mathbf{z},t)\in\Delta^{V+1} is produced by a neural network followed by a softmax head, i.e . \mathbf{x}_{\theta}^{(i)}(\mathbf{z},t)=\mathrm{Softmax}(\theta^{(i)}(\mathbf{z},t)) , for any fixed \theta , we can prove that -\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle is finite: \displaystyle-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle \displaystyle=-\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+\log\sum_{\mathbf{x}^{\prime}\in\mathcal{X}}\exp(\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{\prime}\rangle)\qquad\because\text{log-softmax} \displaystyle\leq-\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+\log V\max_{\mathbf{x}^{\prime}\in\mathcal{X}}\exp(\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{\prime}\rangle) \displaystyle=-\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+\max_{\mathbf{x}^{\prime}\in\mathcal{X}}\langle\theta^{(i)}(\mathbf{z},t),\mathbf{x}^{\prime}\rangle+\log V \displaystyle<+\infty,\qquad\forall i\in[L],\forall\mathbf{x}\in\mathcal{X}^{L},\forall\mathbf{z}\in\mathcal{S}(\mathbf{x}),\forall j\in[L]\},\forall t\in[0,1], where \mathcal{S}(\mathbf{x}) is set of all possible masked sequence induced from \mathbf{x} (Definition B.1 ). Since -\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle is finite, there exists uniform bound \varepsilon_{\theta} for fixed \theta : \displaystyle-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle\leq\varepsilon_{\theta},\quad\forall i\in[L],\forall\mathbf{x}\in\mathcal{X}^{L},\forall\mathbf{z}\in\mathcal{S}(\mathbf{x}),\forall t\in[0,1]. (56) where \mathcal{X}^{L} and \mathcal{S}(\mathbf{x}) are finite. Proposition C.3 (Finiteness of the OeMDM NELBO) . Assume the diffusion model output is finite everywhere (as is typical for neural networks). Fix arbitrary u and \hat{u} , and assume that A^{(i)}(u,t) and \hat{A}^{(i)}(\hat{u},t) are well-defined such that there exists a constant k\geq 1 satisfying: \displaystyle\frac{A^{(i)}(u,t)}{\hat{A}^{(i)}(\hat{u},t)}\in[k^{-1},k],\qquad\forall\,t\in(0,1],\;\forall\,i\in[L]. Then the OeMDM NELBO is finite: \displaystyle\mathcal{L}_{\mathrm{OeMDM{}}} \displaystyle=\int_{0}^{1}\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}\bigl(\log A^{(i)}-\log\hat{A}^{(i)}\bigr)-\bigl(A^{(i)}-\hat{A}^{(i)}\bigr)\biggr\}\biggr]dt<\infty, where we omit u and \hat{u} in A^{(i)} and \hat{A}^{(i)} for brevity. Proof. To prove the finiteness of NELBO, we will upper bound integrand \ell(t) with an equation without the expectation term. Before that, we recall the input domain of \alpha_{\mathcal{F}} and \hat{\alpha}_{\mathcal{F}} . Note that \alpha_{\mathcal{F}} is the scheduler for forward and true posterior, i.e . , q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}|\mathbf{x}) and q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}|\mathbf{x},\mathbf{z}_{t}) respectively. This means that \mathcal{I} only can include \mathcal{X} : when the true sequence \mathbf{x} is given, every \alpha_{\mathcal{F}} is fixed such that the map r\mapsto\alpha_{\mathcal{F}}^{(i)}(u,r) is evaluated. Furthermore, \hat{\alpha}_{\mathcal{F}} is the scheduler for parametrized reverse posterior, i.e . , p_{\theta,\hat{\alpha}_{\mathcal{F}}}(\mathbf{z}_{s}|\mathbf{z}_{t}) . This means that \mathcal{\hat{I}} can include \mathcal{Z}_{t} and \theta : when they are fixed, every \hat{\alpha}_{\mathcal{F}} is fixed such that the map r\mapsto\hat{\alpha}_{\mathcal{F}}^{(i)}(u,r) is evaluated. When deriving \ell(t) , we will omit u term in A and \alpha_{\mathcal{F}} for brevity, but note that A and \alpha cannot be exists alone without u (which can be \mathbf{x} ). This also should be applied to \hat{A} and \hat{\alpha}_{\mathcal{F}} , they cannot stand alone without \hat{u} (which can be \mathbf{z} or \mathbf{\theta} ). But also note that \mathbf{x} and \mathbf{\theta} are actually given for \ell(t) since we are measuring \log p_{\theta,\alpha_{\mathcal{F}}}(\mathbf{x}) ; so that we only consider that \mathbf{z} should be given when including \hat{A} and \hat{\alpha}_{\mathcal{F}} . Within such property, we transform \ell(t) as follows: \displaystyle\ell(t) \displaystyle:=\mathbb{E}_{q_{\alpha_{\mathcal{F}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}\Biggr] \displaystyle=\sum_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}=\mathbf{z}|\mathbf{x})\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}\Biggr] \displaystyle=\sum_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}\prod_{j=1}^{L}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}^{(j)}=\mathbf{z}^{(j)}|\mathbf{x})\hskip 85.35826pt\because\text{Forward process is independent for every indices} \displaystyle\hskip 85.35826pt\cdot\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}^{(i)},\mathbf{m}\rangle\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}\Biggr] \displaystyle=\sum_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}\Biggl[\sum_{i=1}^{L}\Biggr\{\Big((1-\alpha_{\mathcal{F}}^{(i)})\langle\mathbf{z}^{(i)},\mathbf{m}\rangle\Big)\cdot\Big(\prod_{j\neq i}q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}^{(j)}=\mathbf{z}^{(j)}|\mathbf{x})\Big) \displaystyle\hskip 113.81102pt\biggl(-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr)\Biggr\}\Biggr] \displaystyle\leq\sum_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}\Biggl[\underbrace{\sum_{i=1}^{L}(1-\alpha_{\mathcal{F}}^{(i)})\biggl\{-A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)\biggr\}}_{\coloneqq\hat{\ell}(\mathbf{z},t)}\Biggr]. We upper bound \ell(t) with \mathbf{z}^{*}=\argmax_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}\int_{0}^{1}\hat{\ell}(\mathbf{z},t)dt as follows: \displaystyle-\log p_{\theta,\alpha_{\mathcal{F}}}(\mathbf{x})\leq\int_{0}^{1}\ell(t)dt\leq\int_{0}^{1}\sum_{\mathbf{z}\in\mathcal{S}(\mathbf{x})}\hat{\ell}(\mathbf{z},t)dt\leq 2^{L}\int_{0}^{1}\hat{\ell}(\mathbf{z}^{*},t)dt. (57) Now our goal is to show that \int_{0}^{1}\hat{\ell}(\mathbf{z}^{*},t)dt is finite. Divide \hat{\ell}(\mathbf{z}^{*},t) as follows: \displaystyle\hat{\ell}_{1}(\mathbf{z}^{*},t)=-\sum_{i=1}^{L}(1-\alpha_{\mathcal{F}}^{(i)})A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z},t),\mathbf{x}^{(i)}\rangle dt, \displaystyle\hat{\ell}_{2}(\mathbf{z}^{*},t)=\sum_{i=1}^{L}(1-\alpha_{\mathcal{F}}^{(i)})\Big(A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)dt such that \hat{\ell}(\mathbf{z}^{*},t)=\hat{\ell}_{1}(\mathbf{z}^{*},t)+\hat{\ell}_{2}(\mathbf{z}^{*},t) We first show that \int_{0}^{1}\hat{\ell}_{1}(\mathbf{z}^{*},t)dt is finite: \displaystyle\int_{t=0}^{1}\sum_{i=1}^{L}-(1-\alpha_{\mathcal{F}}^{(i)})A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}^{*},t),\mathbf{x}^{(i)}\rangle dt=\sum_{i=1}^{L}\int_{t=0}^{1}-(1-\alpha_{\mathcal{F}}^{(i)})A^{(i)}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}^{*},t),\mathbf{x}^{(i)}\rangle dt \displaystyle=\sum_{i=1}^{L}\int_{\alpha_{\mathcal{F}}^{(i)}=1}^{0}\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}^{*},t),\mathbf{x}^{(i)}\rangle d\alpha_{\mathcal{F}}^{(i)}=\sum_{i=1}^{L}\int_{\alpha_{\mathcal{F}}^{(i)}=0}^{1}-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}^{*},t),\mathbf{x}^{(i)}\rangle d\alpha_{\mathcal{F}}^{(i)}\leq L\epsilon_{\theta} (58) We now show that \int_{0}^{1}\hat{\ell}_{2}(\mathbf{z}^{*},t)\,dt is finite. Fix i\in[L] . For t\in(0,1] , define A^{(i)}(t):=-\frac{\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)},\qquad\hat{A}^{(i)}(t):=-\frac{\partial_{t}\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}{1-\hat{\alpha}_{\mathcal{F}}^{(i)}(\hat{u},t)}, and r^{(i)}(t):=\frac{A^{(i)}(t)}{\hat{A}^{(i)}(t)}>0,\qquad\kappa(r):=r\log r-(r-1)\geq 0. Then for t\in(0,1] , (1-\alpha_{\mathcal{F}}^{(i)}(u,t))\Big(A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)})\Big)=(1-\alpha_{\mathcal{F}}^{(i)}(u,t))\,\hat{A}^{(i)}(t)\,\kappa(r^{(i)}(t)). By the assumption, for fixed u and \hat{u} , there exists k\geq 1 such that r^{(i)}(t)\in[k^{-1},k] for all t\in(0,1] . Since the map r\mapsto\log r-1+\frac{1}{r} is continuous on (0,\infty) and r is bounded on [k^{-1},k] ; let M:=\sup_{r\in[k^{-1},k]}\Bigl(\log r-1+\frac{1}{r}\Bigr)<\infty. Using A^{(i)}(t)=r^{(i)}(t)\hat{A}^{(i)}(t) , we have for all t\in(0,1] , \displaystyle 0 \displaystyle\leq(1-\alpha_{\mathcal{F}}^{(i)}(u,t))\,\hat{A}^{(i)}(t)\,\kappa(r^{(i)}(t)) \displaystyle=(1-\alpha_{\mathcal{F}}^{(i)}(u,t))\,A^{(i)}(t)\,\frac{\kappa(r^{(i)}(t))}{r^{(i)}(t)} \displaystyle=\bigl(-(\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t))\bigr)\Bigl(\log r^{(i)}(t)-1+\frac{1}{r^{(i)}(t)}\Bigr) \displaystyle\leq M\bigl(-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)\bigr), where we used (1-\alpha_{\mathcal{F}}^{(i)}(u,t))A^{(i)}(t)=-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t) . Since \alpha_{\mathcal{F}}^{(i)}(u,\cdot)\in AC([0,1]) with \alpha_{\mathcal{F}}^{(i)}(u,0)=1 and \alpha_{\mathcal{F}}^{(i)}(u,1)=0 , the fundamental theorem of calculus for absolutely continuous functions yields \int_{0}^{1}-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)\,dt=\alpha_{\mathcal{F}}^{(i)}(u,0)-\alpha_{\mathcal{F}}^{(i)}(u,1)=1. Therefore, \displaystyle\int_{0}^{1}(1-\alpha_{\mathcal{F}}^{(i)}(u,t))\,\hat{A}^{(i)}(t)\,\kappa(r^{(i)}(t))\,dt\leq M\int_{0}^{1}-\partial_{t}\alpha_{\mathcal{F}}^{(i)}(u,t)\,dt=M<\infty. (59) Finally, by Eq. 57 –Eq. 59 , the continuous-time NELBO of OeMDM is finite on t\in[0,1] provided that there exists k\geq 1 such that r^{(i)}(t)\in[k^{-1},k] for all t\in(0,1] and all i\in[L] . ∎ This condition should not be viewed as an artificial assumption introduced solely for the proof; rather, it provides practical intuition on how schedulers should be designed so that the NELBO remains a valid (finite) training objective for masked diffusion models. For example, under the linear scheduler of MDLM, both \alpha and \hat{\alpha} take the form 1-t , so r^{(i)}(t)\equiv 1 and the finiteness of the NELBO follows immediately. More broadly, when constructing input-dependent schedulers, this condition highlights a potential pitfall: if \alpha and \hat{\alpha} are parameterized by overly different function classes so that r^{(i)}(t) cannot be uniformly controlled, then finiteness of the NELBO is no longer guaranteed. In this sense, the bounded-ratio condition serves as a mild regularity guideline for scheduler parameterizations that are expressive yet remain compatible with stable NELBO-based training. Remark. Why NELBO of LoMDM is finite? Recall that the true posterior and reverse velocity of LoMDM are given as follows: \displaystyle A_{\phi}^{(i)}(\mathbf{x},t)=\tfrac{c_{1}+c_{2}\cdot[\text{NormSig}(g_{\phi}(f(\mathbf{x}))]_{i}}{t},\qquad\hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t)=\tfrac{c_{1}+c_{2}\cdot[\text{NormSig}(g_{\psi}(f(\mathbf{z}_{t})))]_{i}}{t}, where c_{1}>c_{2} . Therefore, the inequality \displaystyle\frac{c_{1}-c_{2}}{c_{1}+c_{2}}\leq\frac{A_{\phi}^{(i)}(\mathbf{x},t)}{\hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t)}\leq\frac{c_{1}+c_{2}}{c_{1}-c_{2}} always holds for any \mathbf{x}\in\mathcal{X}^{L} and \mathbf{z}_{t}\in\mathcal{S}(\mathbf{x}) , such that NELBO of LoMDM is always finite by Proposition C.3 . Appendix D Other Proofs D.1 OeMDM Can Express Autoregerssive Models In this section, we provide the complete proof of Proposition 3.3 omitted in the main paper. We first define the ARM scheduler that satisfies the statements in Proposition 3.3 , and further provide the definition of the trajectory sets and one lemma required for the proof. Definition D.1 (ARM scheduler \alpha_{\mathrm{arm},\varepsilon} ) . Fix L\in\mathbb{N} and define time windows t_{i}^{\mathrm{start}}:=1-\frac{i}{L},\qquad t_{i}^{\mathrm{end}}:=1-\frac{i-1}{L},\qquad\Delta:=t_{i}^{\mathrm{end}}-t_{i}^{\mathrm{start}}=\frac{1}{L},\quad i\in[L]. (60) Let S:\mathbb{R}\to[0,1] be the C^{1} smoothstep S(u):=\begin{cases}0,&u\leq 0,\\ 3u^{2}-2u^{3},&0i , we have t\geq t_{j}^{\mathrm{end}} , hence S\!\big(\frac{t-t_{j}^{\mathrm{start}}}{\Delta}\big)=1 and \alpha_{\mathrm{arm},\varepsilon}^{(j)}(t)=\varepsilon(1-t) , so q(\mathbf{z}_{t}^{(j)}\neq\mathbf{m}\mid\mathbf{x})=\alpha_{\mathrm{arm},\varepsilon}^{(j)}(t)\leq\varepsilon . Therefore, conditioned on \mathbf{z}_{t}^{(i)}=\mathbf{m} , the event \mathbf{z}_{t}=\mathbf{y}_{i}=[\mathbf{x}^{(1:i-1)}:\mathbf{m}^{L-i+1}] holds with probability 1-O(\varepsilon) (union bound over j\neq i ). Using the time-agnostic assumption \mathbf{x}_{\theta}(\mathbf{z}_{t},t)=\mathbf{x}_{\theta}(\mathbf{z}_{t}) , we obtain g_{i}(t)=-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle+O(\varepsilon),\qquad t\in[t_{i}^{\mathrm{start}},t_{i}^{\mathrm{end}}], (77) where the O(\varepsilon) term is uniform in t (assuming the integrand is finite, i.e . , \langle\mathbf{x}_{\theta}^{(i)}(\cdot),\mathbf{x}^{(i)}\rangle>0 on the relevant states). Outside the i -th window, we have S^{\prime}(\cdot)=0 , hence \partial_{t}\alpha_{\mathrm{arm},\varepsilon}^{(i)}(t)=-\varepsilon and so \int_{[0,1]\setminus[t_{i}^{\mathrm{start}},t_{i}^{\mathrm{end}}]}\bigl(-\partial_{t}\alpha_{\mathrm{arm},\varepsilon}^{(i)}(t)\bigr)\,g_{i}(t)\,dt=O(\varepsilon). Combining with Eq. 77 in Eq. 76 yields \displaystyle\int_{0}^{1}\bigl(-\partial_{t}\alpha_{\mathrm{arm},\varepsilon}^{(i)}(t)\bigr)\,g_{i}(t)\,dt \displaystyle=\Bigl(-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle\Bigr)\int_{0}^{1}\bigl(-\partial_{t}\alpha_{\mathrm{arm},\varepsilon}^{(i)}(t)\bigr)\,dt+O(\varepsilon) \displaystyle=-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle\cdot\bigl(\alpha_{\mathrm{arm},\varepsilon}^{(i)}(0)-\alpha_{\mathrm{arm},\varepsilon}^{(i)}(1)\bigr)+O(\varepsilon) \displaystyle=-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle+O(\varepsilon), since \alpha_{\mathrm{arm},\varepsilon}^{(i)}(0)=1 and \alpha_{\mathrm{arm},\varepsilon}^{(i)}(1)=0 . Summing over i=1,\ldots,L proves \mathcal{L}_{\text{OeMDM{}}}(\mathbf{x},\theta,\alpha_{\mathrm{arm},\varepsilon},\alpha_{\mathrm{arm},\varepsilon})=\sum_{i=1}^{L}-\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle+O(\varepsilon)=-\log\prod_{i=1}^{L}\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{i}),\mathbf{x}^{(i)}\rangle+O(\varepsilon), which completes the proof. ∎ Furthermore, we can extend the above theoretical results for the auto-regressive modeling of any fixed order: Corollary D.4 . Let \pi be an arbitrary but fixed permutation of [L]:=\{1,\ldots,L\} , and denote the induced generation order by (\pi(1),\ldots,\pi(L)) . For fixed sequence \mathbf{x} , let \mathbf{y}_{\pi,i} be the masked sequence with L-i+1 masks following \pi permutation ordering: \displaystyle\mathbf{y}_{\pi,i}=(\mathbf{x}^{(\pi(1))},\dots,\mathbf{x}^{(\pi(i-1))},\underbrace{\mathbf{m},\dots,\mathbf{m}}_{L-i+1}) Define auto-regressive modeling of p_{\theta,\pi} with time-agnostic model \theta and its corresponding negative log-likelihood as follows: \displaystyle p_{\theta,\pi}(\mathbf{x})=\prod_{i=1}^{L}\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{\pi,i}),\mathbf{x}^{(i)}\rangle,\qquad\mathcal{L}_{\pi}=-\log\prod_{i=1}^{L}\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{y}_{\pi,i}),\mathbf{x}^{(i)}\rangle Then, there exists \alpha_{\pi,\epsilon}\in\mathcal{F}[\emptyset] that satisfies \displaystyle p_{\theta,\alpha_{\pi,\epsilon}}(\mathbf{x})=p_{\theta,\pi}(\mathbf{x})+O(\epsilon),\qquad\mathcal{L}_{\text{OeMDM{}}}(\theta,\alpha_{\pi,\epsilon},\alpha_{\pi,\epsilon})=\mathcal{L}_{\pi}+O(\epsilon). The proof can be easily done by replacing indices 1,\dots,L with \pi(1),\dots,\pi(L) in the proof of Proposition 3.3 . D.2 OeMDM Can Express Block Diffusion Models In this section, we show that OeMDM can also express block diffusion generation schemes. 1 propose block discrete denoising diffusion models (BD3LMs), which interpolate between autoregressive and discrete diffusion language models by (i) factorizing a sequence distribution autoregressively over blocks, and (ii) modeling each block-conditional via a discrete denoising diffusion process restricted to that block. Brief explanation of BD3LMs. Let the length- L sequence be partitioned into B disjoint blocks \{\mathcal{B}_{b}\}_{b=1}^{B} and write \mathbf{x}^{b}:=\mathbf{x}^{\mathcal{B}_{b}} and \mathbf{x}^{\alpha_{\mathrm{Gen}}(t)_{\text{[``cat'']}} 3 3 3 Here, \alpha_{\mathrm{Gen}}(t)\in[0,1]^{V+1} is indexed by vocabulary tokens, and we simply write \alpha_{\mathrm{Gen}}(t)_{\text{[``dog'']}} as the entry corresponding to the one-hot index of the token “dog”. , scheduler noise “cat” more than “dog”. To investigate further, we provide the forward process, true posterior, parametrized reverse process, and continuous-time NELBO of GenMD4 with our notations. We denote by \mathbf{1}\in\mathbb{R}^{V+1} the all-ones vector, and use \odot for element-wise product. For element-wise division we simply write it as a fraction. Then, \displaystyle q_{\alpha_{\text{Gen}}}(\mathbf{z}_{t}^{(i)}\mid\mathbf{x}) \displaystyle=\mathrm{Cat}\!\left(\,\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle\mathbf{x}^{(i)}+\langle\mathbf{1}-\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle\,\mathbf{m}\right), \displaystyle q_{\alpha_{\text{Gen}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x}) \displaystyle=\begin{cases}\mathrm{Cat}(\mathbf{z}_{t}^{(i)}),&\text{if }\mathbf{z}_{t}^{(i)}\neq\mathbf{m},\\ \mathrm{Cat}\!\left(\left\langle\tfrac{\mathbf{1}-\alpha_{\text{Gen}}(s)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\,\mathbf{x}^{(i)}\right\rangle\mathbf{m}+\left(\tfrac{\alpha_{\text{Gen}}(s)-\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}\right)\odot\mathbf{x}^{(i)}\right),&\text{if }\mathbf{z}_{t}^{(i)}=\mathbf{m},\end{cases} \displaystyle p_{\theta,\alpha_{\text{Gen}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t}) \displaystyle=\begin{cases}\mathrm{Cat}(\mathbf{z}_{t}^{(i)}),&\text{if }\mathbf{z}_{t}^{(i)}\neq\mathbf{m},\\ \mathrm{Cat}\!\left(\left\langle\tfrac{\mathbf{1}-\alpha_{\text{Gen}}(s)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\,\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle\mathbf{m}+\left(\tfrac{\alpha_{\text{Gen}}(s)-\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}\right)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right),&\text{if }\mathbf{z}_{t}^{(i)}=\mathbf{m}.\end{cases} \displaystyle\mathcal{L}_{\text{Gen}} \displaystyle=\int_{0}^{1}\mathbb{E}_{q_{\alpha_{\text{Gen}}}}\Biggl[\sum_{i=1}^{L}\langle\mathbf{z}_{t}^{(i)},\mathbf{m}\rangle\Biggl\{\bigg\langle\frac{\partial_{t}\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\Big(\log\langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle\Big)\mathbf{x}^{(i)}+\mathbf{x}^{(i)}-\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t)\bigg\rangle\Biggl\}\Biggr]dt (85) where \alpha_{\text{Gen}}:\mathcal{T}\rightarrow[0,1]^{V+1} is a vocabulary-wise scheduler, so the noising velocity is determined only by single token. More specifically, in the forward process, the term \langle 1-\alpha_{\mathrm{Gen}}(t),\mathbf{x}^{(i)}\rangle represents the probability that the i -th token is replaced by a mask. When \mathbf{x}^{(i)} is a one-hot vector corresponding to the j -th vocabulary token, this masking probability simplifies to 1-\alpha_{\mathrm{Gen}}(t)_{j} . Note that the equations might appear different from 30 since we explicitly use categorical distribution notation, but they are algebraically equivalent. Interpretation. GenMD4 and our OeMDM both consider learnable schedulers, but they differ fundamentally in “what it dependent to” and in the expressive freedom of the scheduler. In GenMD4, the scheduler is a vocabulary-wise vector \alpha_{\text{Gen}}(t)\in[0,1]^{V+1} , so the noising velocity is determined only by the token identity (i.e., which vocabulary entry the clean token belongs to), and is shared across all contexts and positions. In contrast, our framework treats the scheduler as a free-form object \alpha_{\mathcal{F}}^{(i)}(u,t) with minimal constraints, allowing it to depend on rich context u (e.g., the entire sequence \mathbf{x} , the current state \mathbf{z}_{t} , and the position i ). As a result, LoMDM can represent a broad family of generation orderings within a single framework (e.g., ARMs, MDMs, BD3LMs) by appropriate choices of \alpha_{\mathcal{F}} , while GenMD4 corresponds to a specific restricted parametrization. Moreover, we show below that GenMD4 does fall into the OeMDM framework in the continuous-time (infinitesimal) limit. Expressing the GenMD4 forward process and true posterior within OeMDM. For a fixed position i , let \alpha_{\mathcal{F}}^{(i)}(u,t)\;:=\;\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle\;\in\;[0,1], where u contains \mathbf{x}^{(i)} and \alpha_{\mathrm{Gen}}(t) . Since \mathbf{x}^{(i)} is one-hot, we have \langle\mathbf{1}-\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle=1-\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle=1-\alpha_{\mathcal{F}}^{(i)}(u,t). Thus the GenMD4 forward process is exactly the OeMDM forward process with scheduler defined as above: \displaystyle q_{\alpha_{\text{Gen}}}(\mathbf{z}_{t}^{(i)}\mid\mathbf{x})=\mathrm{Cat}\!\left(\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle\mathbf{x}^{(i)}+\langle\mathbf{1}-\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle\mathbf{m}\right) \displaystyle=\mathrm{Cat}\!\left(\alpha_{\mathcal{F}}^{(i)}(u,t)\mathbf{x}^{(i)}+(1-\alpha_{\mathcal{F}}^{(i)}(u,t))\mathbf{m}\right)=q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{t}^{(i)}\mid\mathbf{x}). Moreover, the GenMD4 true posterior also reduces to the OeMDM true posterior with the same scalar scheduler \alpha_{\mathcal{F}}^{(i)} . Indeed, for the mask case \mathbf{z}_{t}^{(i)}=\mathbf{m} , one-hotness implies \left\langle\frac{\mathbf{1}-\alpha_{\text{Gen}}(s)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\,\mathbf{x}^{(i)}\right\rangle=\frac{1-\langle\alpha_{\text{Gen}}(s),\mathbf{x}^{(i)}\rangle}{1-\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle}=\frac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}, and \left(\frac{\alpha_{\text{Gen}}(s)-\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}\right)\odot\mathbf{x}^{(i)}=\left\langle\frac{\alpha_{\text{Gen}}(s)-\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\,\mathbf{x}^{(i)}\right\rangle\mathbf{x}^{(i)}=\frac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\,\mathbf{x}^{(i)}. Plugging these identities into q_{\alpha_{\text{Gen}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x}) shows that the GenMD4 true posterior coincides with q_{\alpha_{\mathcal{F}}}(\mathbf{z}_{s}^{(i)}\mid\mathbf{z}_{t},\mathbf{x})=\mathrm{Cat}\!\left(\frac{1-\alpha_{\mathcal{F}}^{(i)}(u,s)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\,\mathbf{m}+\frac{\alpha_{\mathcal{F}}^{(i)}(u,s)-\alpha_{\mathcal{F}}^{(i)}(u,t)}{1-\alpha_{\mathcal{F}}^{(i)}(u,t)}\,\mathbf{x}^{(i)}\right),\qquad\text{when }\mathbf{z}_{t}^{(i)}=\mathbf{m}, and the non-mask case \mathbf{z}_{t}^{(i)}\neq\mathbf{m} is identical in both formulations. Therefore, GenMD4’s forward process and true posterior are fully captured by OeMDM via the context-dependent scalar scheduler \alpha_{\mathcal{F}}^{(i)}(u,t)=\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle . What remains is to compare the model-parametrized reverse process. Expressing GenMD4 reverse process within the OeMDM framework via reparametrization. In the discrete-time reverse kernel of GenMD4, the token component is reweighted as \big(\tfrac{\alpha_{\text{Gen}}(s)-\alpha_{\text{Gen}}(t)}{1-\alpha_{\text{Gen}}(t)}\big)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t) , which depends on the pair (s,t) through element-wise weights. For a general vector-valued scheduler \alpha_{\text{Gen}}(\cdot) this token-wise reweighting cannot be written in the plain scalar-mixture form \mathrm{Cat}\!\big(\beta\,\mathbf{m}+(1-\beta)\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big) of the discrete OeMDM reverse process without allowing an additional (s,t) -dependent transformation inside velocity. Hence, exact discrete-time equality for arbitrary s\mathbf{0}. A first-order expansion yields \frac{\alpha_{\text{Gen}}(t-\mathrm{d}t)-\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}=A_{\text{Gen}}(t)\,\mathrm{d}t+o(\mathrm{d}t),\qquad\frac{\mathbf{1}-\alpha_{\text{Gen}}(t-\mathrm{d}t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}=\mathbf{1}-A_{\text{Gen}}(t)\,\mathrm{d}t+o(\mathrm{d}t). Plugging these into the GenMD4 reverse kernel (mask case) gives \displaystyle p_{\theta,\alpha_{\text{Gen}}}\!\left(\mathbf{z}^{(i)}_{t-\mathrm{d}t}\mid\mathbf{z}_{t}\right)=\mathrm{Cat}\!\left(\big(1-\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\rangle\,\mathrm{d}t\big)\mathbf{m}+\big(A_{\text{Gen}}(t)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big)\mathrm{d}t\right)+o(\mathrm{d}t), where we used \langle\mathbf{1},\mathbf{x}^{(i)}_{\theta}\rangle=1 and \mathbf{z}^{(i)}_{t}=\mathbf{m} . Define the reparametrized model output \tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t):=\frac{A_{\text{Gen}}(t)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)}{\left\langle A_{\text{Gen}}(t),\,\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle}\in\Delta^{V+1},\qquad\text{(note that }x^{(i)}_{\theta,\mathbf{m}}=0\Rightarrow\tilde{x}^{(i)}_{\theta,\mathbf{m}}=0\text{)}. Then A_{\text{Gen}}(t)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)=\left\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle\,\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t), and moreover the scalar hazard can be written using only (A_{\text{Gen}},\tilde{\mathbf{x}}_{\theta}) as \displaystyle\left\langle A_{\text{Gen}}^{\odot-1}(t),\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle \displaystyle=\left\langle A_{\text{Gen}}^{\odot-1}(t),\frac{A_{\text{Gen}}(t)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)}{\left\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle}\right\rangle \displaystyle=\frac{\langle\mathbf{1},\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\rangle}{\left\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle}=\frac{1}{\left\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle}, where \odot^{-1} denotes the element-wise inverse operator, applied to each coordinate of A_{\mathrm{Gen}}(t) . Hence, \left\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle=\left\langle A_{\text{Gen}}^{\odot-1}(t),\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle^{-1}. Substituting into the infinitesimal kernel yields the scalar-mixture form \displaystyle p_{\theta,\alpha_{\text{Gen}}}\!\left(\mathbf{z}^{(i)}_{t-\mathrm{d}t}\mid\mathbf{z}_{t}\right)=\mathrm{Cat}\!\left(\big(1-\hat{A}^{(i)}(\hat{u},t)\,\mathrm{d}t\big)\mathbf{m}+\hat{A}^{(i)}(\hat{u},t)\,\mathrm{d}t\;\mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t)\right)+o(\mathrm{d}t),\qquad\mathbf{z}^{(i)}_{t}=\mathbf{m}, by choosing \mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t):=\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t),\qquad\hat{A}^{(i)}(\hat{u},t):=\left\langle A_{\text{Gen}}^{\odot-1}(t),\mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t)\right\rangle^{-1}, where \hat{u} contains (\mathbf{z}_{t},t) and the scheduler (equivalently A_{\text{Gen}} ). Therefore, GenMD4 is exactly expressed by the continuous-time (velocity-form) OeMDM reverse parameterization. NELBO equivalence. We now show that the continuous-time NELBO of GenMD4 coincides with the OeMDM NELBO under the above identifications. First, since the forward process (and hence the true posterior) is identical under \alpha_{\mathcal{F}}^{(i)}(u,t)=\langle\alpha_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle , the trajectory distributions agree, so the expectations are taken over the same law: \mathbb{E}_{q_{\alpha_{\mathcal{F}}}}[\cdot]=\mathbb{E}_{q_{\alpha_{\text{Gen}}}}[\cdot]. Next, consider the OeMDM integrand inside \mathcal{L}_{\mathrm{OeMDM{}}} on masked positions. With the infinitesimal reparametrization established above, we substitute \mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t)=\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t) and \hat{A}^{(i)}(\hat{u},t)=\langle A_{\text{Gen}}^{\odot-1}(t),\mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t)\rangle^{-1}=\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\rangle. Moreover, for the forward (true) scalar velocity we take the GenMD4-induced hazard A^{(i)}=\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle (which is consistent with the above choice of \alpha_{\mathcal{F}}^{(i)} ). Using one-hotness of \mathbf{x}^{(i)} , we have \Big\langle\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\Big\rangle=\frac{\big\langle A_{\text{Gen}}(t)\odot\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle}{\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle}=\frac{\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle\,\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle}{\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle}. Substituting this identity together with A^{(i)}=\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle and \hat{A}^{(i)}=\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\rangle into the OeMDM integrand -A^{(i)}\log\langle\mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}},\mathbf{x}^{(i)}\rangle+A^{(i)}(\log A^{(i)}-\log\hat{A}^{(i)})-(A^{(i)}-\hat{A}^{(i)}) : \displaystyle-A^{(i)}\log\Big\langle\mathbf{x}^{(i)}_{\theta,\mathrm{OeMDM{}}}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\Big\rangle+A^{(i)}\big(\log A^{(i)}-\log\hat{A}^{(i)}\big)-(A^{(i)}-\hat{A}^{(i)}) \displaystyle=-A^{(i)}\log\Big\langle\tilde{\mathbf{x}}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\Big\rangle+A^{(i)}\big(\log A^{(i)}-\log\hat{A}^{(i)}\big)-(A^{(i)}-\hat{A}^{(i)}) \displaystyle=-A^{(i)}\log\!\left(\frac{\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle\,\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle}{\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle}\right)+A^{(i)}\big(\log A^{(i)}-\log\hat{A}^{(i)}\big)-(A^{(i)}-\hat{A}^{(i)}) \displaystyle=-A^{(i)}\Big(\log\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle+\log\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle-\log\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle\Big) \displaystyle+A^{(i)}\log A^{(i)}-A^{(i)}\log\hat{A}^{(i)}-A^{(i)}+\hat{A}^{(i)} \displaystyle=-A^{(i)}\log\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle-A^{(i)}\log\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle+A^{(i)}\log\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle \displaystyle+A^{(i)}\log A^{(i)}-A^{(i)}\log\hat{A}^{(i)}-A^{(i)}+\hat{A}^{(i)}. Now substitute A^{(i)}=\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle and \hat{A}^{(i)}=\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle . Then the terms -A^{(i)}\log\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\rangle+A^{(i)}\log A^{(i)} cancel, and the terms A^{(i)}\log\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\rangle-A^{(i)}\log\hat{A}^{(i)} also cancel, yielding -A^{(i)}\log\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle-A^{(i)}+\hat{A}^{(i)}=-\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle\log\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle-\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle+\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle. Finally, using \frac{\partial_{t}\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)}=-A_{\text{Gen}}(t) , the OeMDM integrand becomes \displaystyle-\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle\log\big\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\big\rangle-\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}\big\rangle+\big\langle A_{\text{Gen}}(t),\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\big\rangle \displaystyle=\left\langle\frac{\partial_{t}\alpha_{\text{Gen}}(t)}{\mathbf{1}-\alpha_{\text{Gen}}(t)},\,(\log\langle\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle)\mathbf{x}^{(i)}+\mathbf{x}^{(i)}-\mathbf{x}^{(i)}_{\theta}(\mathbf{z}_{t},t)\right\rangle which matches the GenMD4 integrand in Eq. 85 . Appendix E Training Stabilization Techniques E.1 REINFORCE Leave-One-Out In Section 4.3 , we have stated that \mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})] in \nabla_{\phi}\mathcal{L}_{\mathrm{LoMDM{}}}=\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})]+\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})] is a high-variance estimator. Therefore, we use a low-variance estimator of such optimization problems proposed by 14 . Before deriving a low-variance estimator, we treat \mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})] as follows: \displaystyle\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\log q_{\alpha_{\phi}}\cdot(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})]=\nabla_{\phi}\mathbb{E}_{q_{\alpha_{\phi}}}[\text{Sgd}(\mathcal{L}_{\mathrm{main}}+\mathcal{L}_{\mathrm{velocity}})]=\nabla_{\phi}\mathbb{E}_{\mathbf{z}_{t}\sim q_{\alpha_{\phi}}}[\text{Sgd}(\mathcal{L}_{\mathbf{z}_{t}})], where we denote stop-gradient as Sgd and \mathcal{L}_{\text{main}}+\mathcal{L}_{\text{velocity}} for \mathbf{z}_{t} as \mathcal{L}_{\mathbf{z}_{t}} as in main paper. Then, the low-variance estimator ( 14 ) is given as follows: \displaystyle\nabla_{\phi}\mathbb{E}_{\mathbf{z}_{t}\sim q_{\alpha_{\phi}}}\bigl[\operatorname{Sgd}\bigl(\mathcal{L}_{\mathbf{z}_{t}}\bigr)\bigr] \displaystyle\approx\frac{1}{k}\sum_{i=1}^{k}\nabla_{\phi}\log q_{\alpha_{\phi}}(\mathbf{z}_{t}^{i}|\mathbf{x})\left(\mathcal{L}_{\mathbf{z}_{t}^{i}}-\frac{1}{k-1}\sum_{j\neq i}\mathcal{L}_{\mathbf{z}_{t}^{j}}\right) (86) \displaystyle=\frac{1}{k-1}\sum_{i=1}^{k}\nabla_{\phi}\log q_{\alpha_{\phi}}(\mathbf{z}_{t}^{i}|\mathbf{x})\left(\mathcal{L}_{\mathbf{z}_{t}^{i}}-\frac{1}{k}\sum_{j=1}^{k}\mathcal{L}_{\mathbf{z}_{t}^{j}}\right). (87) Here we set k=2 so that, \displaystyle\nabla_{\phi}\mathbb{E}_{\mathbf{z}_{t}\sim q_{\alpha_{\phi}}}\bigl[\operatorname{Sgd}\bigl(\mathcal{L}_{\mathbf{z}_{t}}\bigr)\bigr]\approx\frac{1}{2}\Bigl(\nabla_{\phi}\log q_{\alpha_{\phi}}(\mathbf{z}_{t}^{1}|\mathbf{x})-\nabla_{\phi}\log q_{\alpha_{\phi}}(\mathbf{z}_{t}^{2}|\mathbf{x})\Bigr)\Bigl(\mathcal{L}_{\mathbf{z}^{1}_{t}}-\mathcal{L}_{\mathbf{z}^{2}_{t}}\Bigr). (88) Final gradient estimation Let \displaystyle\mathcal{\widehat{L}}_{\text{LoMDM{}}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})\coloneqq\tfrac{1}{2}(\mathcal{L}_{\mathbf{z}_{t}^{1}}+\mathcal{L}_{\mathbf{z}_{t}^{2}})+\mathcal{L}_{\text{rloo}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}),\qquad\text{where}\quad\mathcal{L}_{\text{rloo}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})\coloneqq\frac{1}{2}\log\tfrac{q_{\alpha_{\phi}}(\mathbf{z}_{t}^{1}|\mathbf{x})}{q_{\alpha_{\phi}}(\mathbf{z}_{t}^{2}|\mathbf{x})}\Bigl(\operatorname{Sgd}(\mathcal{L}_{\mathbf{z}_{t}^{1}})-\operatorname{Sgd}(\mathcal{L}_{\mathbf{z}_{t}^{2}})\Bigr), such that \mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\phi}\mathcal{L}_{\mathrm{rloo}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})] is the two sample gradient estimator of reinforce term. Furthermore, since \mathcal{L}_{\text{rloo}} is invariant to the gradient of \psi , \theta , final gradient is given as folows: \displaystyle\nabla_{\phi}\mathcal{L}_{\text{LoMDM{}}}=\mathbb{E}_{t}[\mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\phi}\mathcal{\widehat{L}}_{\text{LoMDM{}}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})]], \displaystyle\nabla_{\psi}\mathcal{L}_{\text{LoMDM{}}}=\mathbb{E}_{t}[\mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\psi}\mathcal{\widehat{L}}_{\text{LoMDM{}}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})]], \displaystyle\nabla_{\theta}\mathcal{L}_{\text{LoMDM{}}}=\mathbb{E}_{t}[\mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\theta}\mathcal{\widehat{L}}_{\text{LoMDM{}}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})]], where the two-sample estimator \mathbb{E}_{t}[\mathbb{E}_{\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2}\sim q_{\alpha_{\phi}}}[\nabla_{\phi}\mathcal{\widehat{L}}_{\text{LoMDM{}}}(\mathbf{z}_{t}^{1},\mathbf{z}_{t}^{2})]] gives lower variance than the naive estimator \nabla_{\phi}\mathcal{L}_{\mathrm{LoMDM{}}}=\mathbb{E}_{\mathbf{z}_{t}\sim q_{\alpha_{\phi}}}[(\nabla_{\phi}\log q_{\alpha_{\phi}})\cdot\mathcal{L}_{\mathbf{z}_{t}}]+\mathbb{E}_{q_{\alpha_{\phi}}}[\nabla_{\phi}\mathcal{L}_{\mathbf{z}_{t}}] . Appendix F Details about LoMDM parametrization F.1 Rationale behind the LoMDM parametrization In this section, we explain why we parametrize LoMDM as in Eq. 4 –Eq. 7 . Unlike discrete diffusion, which has only recently begun to be explored, the continuous domain (e.g., image diffusion) has a longer history of studying learnable noise schedulers. However, the image and text domains differ substantially, making it nontrivial to directly transfer such techniques. We therefore first briefly review how learnable schedulers are typically introduced in continuous diffusion. Learnable schedulers in continuous diffusion: variational diffusion models. A representative example is the variational diffusion model (VDM), which introduces a single learnable scheduler shared across the entire image ( 13 ) . Since the scheduler must be monotone decreasing in time, VDM learns it via an additional monotonic neural network that is separate from the diffusion model. In contrast, such monotonic-network-based parametrizations are difficult to apply in our case, modeling a different generation order in a text space. First, since the diffusion model itself cannot be constrained to be monotone in t , a monotonic neural network (NN) must be decoupled from the diffusion model. In VDM, the monotonic NN learns a single scheduler shared across all pixels; because this mapping is low-dimensional and input-agnostic, it is easy to train and can be implemented with a small network. In contrast, in text generation, inducing nontrivial generation orders requires a scheduler that depends on the given context and outputs token-wise scheduling values. This would necessitate a large, standalone monotonic NN separate from the diffusion model, which is computationally and statistically inefficient. Second, the quantity that is learned in our formulation is not \alpha(\cdot,t) itself, but rather A(\cdot,t)\;=\;\partial_{t}\alpha(\cdot,t)\oslash\bigl(1-\alpha(\cdot,t)\bigr). In particular, the OeMDM NELBO is expressed as a weighted sum of token-wise reconstruction losses with weights given by A^{(i)}(\cdot,t) . Under such a nontrivial functional relationship between \alpha and A , naively parameterizing \alpha via a monotonic NN as in VDM provides no direct mechanism to ensure that tokens with lower reconstruction loss receive smaller effective training weights. Consequently, VDM-style monotonic-network parametrizations are ill-suited for discrete diffusion with token-wise scheduling, motivating other parametrizations for texts. Learnable schedulers in continuous diffusion: multivariate learned adaptive noise. 27 further advance this line of work by proposing multivariate learned adaptive noise (MULAN). In a text-conditioned image diffusion model, MULAN takes the conditioning text as input and produces pixel-wise schedulers, allowing different pixels to follow different noise schedules. However, in light of the limitations of monotonic NNs discussed above, MULAN adopts a different design: rather than learning a full scheduler via a standalone monotonic network, it parameterizes the scheduler as a quintic polynomial controlled by three parameters. For image dimension d=L\times H , MULAN define neural networks \mathbf{a}_{\vartheta}(\mathbf{x}):\mathcal{V}^{L}\to\mathbb{R}^{d} , \mathbf{b}_{\vartheta}(\mathbf{x}):\mathcal{V}^{L}\to\mathbb{R}^{d} , and \mathbf{c}_{\vartheta}(\mathbf{x}):\mathcal{V}^{L}\to\mathbb{R}^{d} with parameters \vartheta . Let f_{\vartheta}:\mathcal{V}^{L}\times[0,1]\to\mathbb{R}^{d} be defined as f_{\vartheta}(\mathbf{x},t)=\frac{\mathbf{a}_{\vartheta}(\mathbf{x})^{2}}{5}\,t^{5}\;+\;\frac{\mathbf{a}_{\vartheta}(\mathbf{x})\,\mathbf{b}_{\vartheta}(\mathbf{x})}{2}\,t^{4}\;+\;\frac{\mathbf{b}_{\vartheta}(\mathbf{x})^{2}+2\,\mathbf{a}_{\vartheta}(\mathbf{x})\,\mathbf{c}_{\vartheta}(\mathbf{x})}{3}\,t^{3}\;+\;\mathbf{b}_{\vartheta}(\mathbf{x})\,\mathbf{c}_{\vartheta}(\mathbf{x})\,t^{2}\;+\;\mathbf{c}_{\vartheta}(\mathbf{x})^{2}\,t, (89) where the multiplication and division operations are elementwise. The noise schedule \alpha_{\vartheta}(\mathbf{x},t) is given by \alpha_{\vartheta}(\mathbf{x},t)=\text{Sigmoid}\bigg(-\Big(\gamma_{\min}+\bigl(\gamma_{\max}-\gamma_{\min}\bigr)\,\frac{f_{\vartheta}(\mathbf{x},t)}{f_{\vartheta}(\mathbf{x},t=1)}\Big)\bigg), (90) where it always guarantees that \alpha is monotone decreasing. However, it remains difficult to apply this approach directly to text. As in VDM, the quantity that is directly learned is the velocity A , rather than \alpha itself. Concretely, for tokens that are poorly reconstructed, training should reduce the corresponding weights A^{(i)} . Yet, since \alpha_{\vartheta} is already a complex function, and A becomes even more intricate as it is induced through \alpha_{\vartheta} , aligning A^{(i)} with reconstruction quality becomes highly challenging. We empirically attempted to train with this parametrization; yet, when we measured the Pearson correlation between A_{\phi}^{(i)}(\mathbf{x},t) , A_{\psi}^{(i)}(\mathbf{z}_{t},t) , and \langle\mathbf{x}_{\theta}^{(i)}(\mathbf{z}_{t},t),\mathbf{x}^{(i)}\rangle throughout training, we observed that none of these correlations increased meaningfully, contrast to that of our LoMDM shown in in Figure 4 . Parametrizion of LoMDM. From the continuous-diffusion case above, we draw one key lesson: the velocity A should admit a simple form that enables direct optimization. While continuous diffusion often allows highly flexible parameterizations of \alpha , in text space we instead require a suitably regularized and simple functional form. Motivated by the simplest scheduler used in MDLM, \alpha_{\mathrm{mdlm}}(t)=1-t , we have proposed the LoMDM parametrization. To recap, \displaystyle\alpha_{\phi}^{(i)}(\mathbf{x},t):=1-t^{\,c_{1}+c_{2}\cdot\bigl[\mathrm{NormSig}(g_{\phi}(f(\mathbf{x})))\bigr]_{i}},\qquad A_{\phi}^{(i)}(\mathbf{x},t)=\frac{c_{1}+c_{2}\cdot\bigl[\mathrm{NormSig}(g_{\phi}(f(\mathbf{x})))\bigr]_{i}}{t}, (91) \displaystyle\hat{\alpha}_{\psi}^{(i)}(\mathbf{z}_{t},t):=1-t^{\,c_{1}+c_{2}\cdot\bigl[\mathrm{NormSig}(g_{\psi}(f(\mathbf{z}_{t})))\bigr]_{i}},\qquad\hat{A}_{\psi}^{(i)}(\mathbf{z}_{t},t)=\frac{c_{1}+c_{2}\cdot\bigl[\mathrm{NormSig}(g_{\psi}(f(\mathbf{z}_{t})))\bigr]_{i}}{t}. (92) where we can modulate the overall denoising velocity by c_{1} and the variance of generation order priority by c_{2} . We also experimented with softmax; however, it became overly biased toward a single position, preventing training from stabilizing. Another thing we should consider is that learning a large, standalone scheduler network for text space would be infeasible. Fortunately, 9 have shown that we can post-train unmasking ordering with one transformer layer attached to the diffusion backbone model. They empirically show that the post-trained unmasking policy model boosts the performance of MDM even without learning MDM itself; i.e . , MDM is frozen while training. Inspired by this work, we set f(\cdot)=\mathrm{Sgd}\!\big(\theta_{\mathrm{TF}}(\cdot)\big) as the feature extractor in the above equations where \theta_{\mathrm{TF}} is the backbone of diffusion model. F.2 Ablation study on c_{1} and c_{2} Table 4 : Hyperparameter ablation on (c_{1},c_{2}) for LoMDM on LM1B with sentence packing: we train separate LoMDM models for each (c_{1},c_{2}) for 1M steps and report the resulting test PPL ( \downarrow ). When c_{2}=0 , the scheduler is non-learnable and reduces to the polynomial form \alpha(t)=1-t^{c_{1}} . In particular, (c_{1},c_{2})=(1,0) recovers MDLM. (c_{1},c_{2}) (0.7,0.0) (0.7,0.65) (1.0,0.0) (1.0,0.95) (1.3,0.0) (1.3,1.25) Test PPL \downarrow 31.7 27.2 31.8 28.7 31.4 28.9 We here provide and discuss an ablation study on our LoMDM that was omitted in the main paper, where we train the model under various choices of c_{1} and c_{2} . To recap, \alpha_{\phi} and \alpha_{\psi} are given by: \displaystyle\alpha_{\phi}^{(i)}(\mathbf{x},t):=1-t^{c_{1}+c_{2}\cdot[\text{NormSig}(g_{\phi}(f(\mathbf{x})))]_{i}},\qquad\hat{\alpha}_{\psi}^{(i)}(\mathbf{z}_{t},t):=1-t^{c_{1}+c_{2}\cdot[\text{NormSig}(g_{\psi}(f(\mathbf{z})))]_{i}}. In particular, when (c_{1},c_{2})=(1,0) , our scheduler reduces exactly to that of MDLM. More generally, if c_{2}=0 and c_{1}>0 is arbitrary, then the identical polynomial scheduler is applied at every position, i.e . , \alpha^{(i)}(t)=1-t^{c_{1}} for all i\in[L] . Before presenting the experiments, we provide a more detailed interpretation of c_{1} and c_{2} . A larger c_{1} implies that \alpha decays more slowly (since t<1 ), meaning that a substantial amount of masking occurs only when t is close to 1 . Conversely, a smaller c_{1} implies a faster decay of \alpha , so masking is concentrated near t\approx 0 . When c_{2}=0 , although the decay rate varies with c_{1} , the NELBO becomes effectively independent of the scheduler after an appropriate change of variables with respect to \alpha ; hence all such cases yield the same objective, which is also mentioned by 26 . We now turn to the case c_{2}\neq 0 . The parameter c_{2} controls the position-dependent strength determined by the scheduler models g_{\phi} and g_{\psi} . In other words, c_{2} governs the variance of generation-order priorities across positions: as c_{2} increases, some indices can be masked much earlier (i.e., with a much faster decay) while others can be masked much later (i.e., with a much slower decay). Through the ablation over (c_{1},c_{2}) , we aim to answer two questions. First, we ask whether the performance gains of our LoMDM are driven by the choice of c_{1} —that is, by using a polynomial time scheduler unlike MDLM—or whether the gains truly come from learning position-dependent schedules via g_{\phi} and g_{\psi} . Second, we investigate how to set (c_{1},c_{2}) so that our LoMDM achieves its best performance. The results are reported in Table 4 . First, for every tested value of c_{1} , setting c_{2}\neq 0 yields substantially lower PPL than c_{2}=0 , confirming that the context-aware scheduler in our LoMDM is indeed beneficial for learning. Furthermore, when c_{2}=0 , the PPL remains nearly constant at \sim 31.8 across different choices of c_{1} . As discussed above, this is expected because c_{2}=0 makes the scheduler non-learnable, and a change of variables with respect to \alpha renders the NELBO invariant to the specific choice of c_{1} ; thus, varying c_{1} mainly changes which masked states are visited, but not the overall amount of training signal allocated to the diffusion model. Second, among c_{2}\neq 0 settings, performance improves as c_{1} decreases. Unlike the c_{2}=0 case, c_{2}\neq 0 yields a context- and position-dependent scheduler, allowing the objective to reweight learning across tokens according to learned generation priorities. Since heavily masked states provide limited contextual information, many masked positions in such states can yield weak or noisy gradients. Our method alleviates this by concentrating the training signal on the subset of tokens that remain learnable under the current context via learned, position-wise priorities. With smaller c_{1} , training visits more highly corrupted states, and the learned scheduler can allocate more of the learning budget within those states to the most informative (learnable) tokens, offering a plausible explanation for the consistent gains at smaller c_{1} when c_{2}\neq 0 . Appendix G Experimental Details G.1 Experimental Settings We detokenize the One Billion Words dataset following 16 ; 26 , whose official code can be found. We tokenize LM1B using the bert-base-uncased tokenizer, consistent with 7 . We then concatenate and pack the sequences to a fixed length of 128 ( 25 ) . For OpenWebText (OWT), we use the GPT-2 tokenizer and similarly concatenate and pack sequences to length 1,024; during packing, we insert an eos token between consecutive documents. As OWT does not provide an official validation split, we reserve the last 100k documents for validation. We parameterize the diffusion backbone of our LoMDM with the modified diffusion transformer architecture ( 22 ) from 16 ; 26 . We use 12 layers, a hidden dimension of 768, 12 attention heads, and for LoMDM, one more same size of transformer layer for each \phi,\psi . For diffusion backbone, we use the AdamW optimizer with a batch size of 512, constant learning rate warmup from 0 to a learning rate of 3e-4 for 2,500 steps following prior works ( 26 ; 16 ; 28 ) . For schedulers \phi and \psi , we use the AdamW optimizer with constant learning rate warmup from 0 to a learning rate of 1e-5 for 2,500 steps. We use a dropout rate of 0.1. As specified in the main paper, we use c_{1}=0.7,c_{2}=0.65 for the parametrization of \phi and \psi . For both LM1B with/without sentence packing, we utilized 2 H200 GPUs for training. For OWT, we utilized 8 H100 GPUs for training. G.2 Qualitative Examples We here provide qualitative examples. Text sample generated by LoMDM trained on OpenWebText with 1024 sampling steps. <|endoftext|>, Indian states have dampened the productivity of some core industries. Most have been acquired by Western companies and a few new industries, supported by outsourcing opportunities. Nevertheless structural changes and Indian forces in recent years brought economists thinking that some company reforms that were made possible with better work conditions --- better wages and unpaid IBM engineers --- made industry jobs much easier to work. Rathan’s opinion in Global European, judged by economist Ernst Karelis hardly ignores workers, the technologies that have bought them information, and the basic human dignity of the world’s largest companies. Nor does he argue that the economic and social growth of world industrial and economic powers are mutually beneficial in the near future. Mostly, however, it does not deny that India can further improve its productivity, as is indeed the case today. India is a leading economic power and one of the world’s leading ``empower’’ countries (see OECD). Moreover, it is a large and developed country with its own market, in the shape of the European Economic Area, an area of aggregating services and goods in the labour market. Only 4.4 percent of the tariff-free transport network will be shipped from India. But with the fast-liberalisation of India, which is sure to attract workers from small company sizes, it will be possible to make wage cuts targeting GST. India’s share in exports is quite large. Actually, imports generate the largest share, because imports are brought by low-wage workers like Indian workers. That is thanks to India’s production of many large steel plants. Most of its neighbours, and Germany, would have enjoyed the same kind of work at their industrial plants. Of course, erasing the global income gap requires massive capital controls. Sure, it could tailor its economic policies to its trade policy, including trade with Japan. But closing the larger gap means one much bigger problem. Firms can’t rise above borders and capital controls generally exist outside the country. Trying to foresee how India might achieve structural changes may discount rising income levels of workers in countries like Japan. Since unions are a larger part of the workforce, additional labour measures would require India to take far less in one move. Nor is it worth mentioning that only Sweden, Brazil, Germany, and Argentina are three foreign countries which don’t have any trade with India but manage Delhi’s existing exports. Nor, is it possible that a worker, independent of bringing his or her social status in once and all, will benefit from capital gains from India’s policies. In the countries where the ratio of workers in population has increased by some sort in recent years, the standard labor standard of workers has fallen in India, as for even industrial countries as advanced as the United States. For decades, the improvement in average-adjusted life expectancy in developed nations was greater than at its 30 years prior, partly as a result of growth in productivity. The recent recession has prevented a full recovery, partly because an increasing sustained supply of workers has only increased, and some even fewer workers died from disability. One of the most notable examples coming in the Second World War, in which, the Soviet Union created a new type of economy that raised the full-time wages of some 200 million, lowered the rise in life expectancy for workers who retired over 15-month tenure, and vastly reduced its labour force by hundreds of workers for economic reasons. In India, export the majority of heavy goods and its labour force has expanded incrementally. Since 1950, industrial productivity has shot rise to 7.9 percent, just a little higher for an Indian citizen. The average-adjusted productivity (which had been rising since 1942) continued for several decades. It is the middle-age expectation that India will be able to match the rising productivity to 11 percent in five decades, and so should have a strong advantage. To meet the needs of its civilian workers, world industrial growth has called on India to increase its number of highly skilled workers. Consider that the steel sector, like any other part of Japan, is a modern industry that, like Japanese, has remarkable rates of growth. Politicians from all 160 member states would respond well to a takeover of industry and population against a province with half the exports of the industrialized world. In recent years, however, a wave of entrepreneurs have begun to expand drastically, spread to the manufacturing of leather (in China), buring bags, and other new technologies done in developed countries. While a new (and more compact) global order must, it could best be avoided. In points of view, globalization is part of a process that stemmed from the fall of empires; that no modern country or even some of the modern states can handle the consequences of the modern industry.’s inability to protect the masses if necessary or rail on other kinds of horrors on a new day. The country blurring our attention is India. It is the politically advanced country that needs to improve its income, its productivity and its vested interests who intend to plunder it. Ameliorating Indian<|endoftext|> Text sample generated by LoMDM trained on OpenWebText with 1024 sampling steps. In an interview with the Houston Chronicle, Hummel Brock said it was an ideal club for Ryan’s season. And coming into his decision to stay in Texas he indicated that that was the correct decision. ``And I said, `I’m looking forward to enjoying myself here,’’’ he said, according to the Chronicle. ``And I said, `I’m crazy’ haven’t played here after less than four weeks. I’m more than happy to sit down, grow this team and teach myself and keep learning so that we can depend on like that. ``Now that I’m healthy and I can return to the home game which when I was a player, it’ll be my starter home again.’’ The Texans fought against competition, with players Alvin Nakamura, DB, and Eli Kennedy entering the open and ended up being the Kos’ offseason target. Bacette has been selected after a few stints. Ryan took the roster in Week 4 which gave Ryan a chance to establish some momentum and some could have expected the beast would move to Houston next offseason.<|endoftext|>With the crew got fired off on the air set uttering on Tuesday, Fox announced that it was making progress on its upcoming ``The X-Men’’ film sequel. The upcoming film stars J.J. Smith and Mark Shumalo, and the producers the studio is owned by Fox Vacation Pictures was imposing a real-life partnership with co-writing producers Simon Kinberg and Kosethel. The two executives agreed that there were previously even fewer contributors, facing procedural issues the company of production may have overshadowed over Fox’s top priorities --- such as five-page paperwork and video cameras. Rather, Fox said they were there to hit ``the Network’’ and FX production costs and began building a set of ``bases and buildings,’’ which wound up curbing their runs and the weekend totals charge for years to come. PHOTOS: The Official Story For The First X-Men Movie Wayne Thomas and Chanel Wigehouse were the executive producers and the script for the film revolves around being done by a first-time movie producer for Kinberg and by Patti Jones for the showrunners. ``The producers found a success and gave us what we needed,’’ said original executive producer Breanna Carrappen. ``At the same time, we were pressed to get things done, no one felt it was possible with the extraordinary looks and the production quality.’’ Menial debut to two X-Men titles Smith worked on the animated ``Doctor Strange,’’ with Chris Evans in charge with the role of Peter Parker and Peter X Then produced the fully animated ``The X-Men’’ with James Cameron attached as director. ``I shaped and wrote the script,’’ Kosethel has said. ``I was Dell/the manager of the production design.’’ Jones’ own stint was a rocky one at first. She wrote of NBC Channel ``The X-Men’’ film she was first directed by Nicholas Meyer and successfully slipped on her way to showrunner for ``Fox’’ until late early 2012. Her replacement did not until 2013. Kosethel had the experience and charm of collaborating on several X-Men titles with screen producers, like ``Watchmen’’, S.E.K.D.L.C.E.L.D (``The Terrific X-Men’’ and ``Best of Viewers’’), which liked the director’s choices to match their diverse demands. X-Men ``Not every person was when they wanted to make it except the top directors of the time period,’’ explained Thomas. ``It would never go that way, every time you do something completely different, it would go through these different subtleties.’’ Kosethel also starkly conflicted the studio as to John being on a staff project and as to what the matter of the material was. ``I developed `Vning’ these characters and there was no other movie that would relate to them, might have an impact here,’’ said Thomas. ``I had to assemble the right pieces that they all agreed with and that there were all things that were right for it.’’ McFarlane might be back, full time, and he will instill the journey with the returning cast by taking his leave. ``It has always been important for me to produce who have career highs,/or statuses, who have important roles to play. I have to understand that script and studio are two of the arenas to drive the business<|endoftext|>