Balancing Understanding and Generation in Discrete Diffusion Models Yue Liu, Yuzhong Zhao, Zheyong Xie, Qixiang Ye, Jianbin Jiao, Yao Hu, Shaosheng Cao, Yunfan Liu Abstract In discrete generative modeling, two dominant paradigms demonstrate divergent capabilities: Masked Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot generalization, whereas Uniform-noise Diffusion Language Models (UDLM) achieve strong few-step generation quality, yet neither attains balanced performance across both dimensions. To address this, we propose XDLM, which bridges the two paradigms via a stationary noise kernel. XDLM offers two key contributions: (1) it provides a principled theoretical unification of MDLM and UDLM, recovering each paradigm as a special case; and (2) an alleviated memory bottleneck enabled by an algebraic simplification of the posterior probabilities. Experiments demonstrate that XDLM advances the Pareto frontier between understanding capability and generation quality. Quantitatively, XDLM surpasses UDLM by 5.4 points on zero-shot text benchmarks and outperforms MDLM in few-step image generation (FID 54.1 vs. 80.8). When scaled to tune an 8B-parameter large language model, XDLM achieves 15.0 MBPP in just 32 steps, effectively doubling the baseline performance. Finally, analysis of training dynamics reveals XDLM’s superior potential for long-term scaling. Code is available at this http URL . 1 Introduction Figure 1 : Left : XDLM combines the noise kernel of UDLM ( \mathbf{u} ) and MDLM ( \mathbf{m} ) to achieve a favorable trade-off between the two methods. [NORMAL] denotes normal tokens, while [MASK] represents the mask token. Right : The trade-off between understanding capability (zero-shot perplexity; lower is better) and generation capability (generation perplexity in 32 sampling steps; lower is better). The proposed XDLM with a mixing ratio of k=0.1 achieves the optimal balance, labeled as the ‘Sweet Spot’. Diffusion models have achieved remarkable success in continuous domains, particularly in image and audio generation ( 11 ; 7 ; 24 ; 13 ) . Inspired by this potential, Discrete Denoising Diffusion Probabilistic Model (D3PM) has emerged as a promising paradigm shift to the discrete state space ( 1 ) . Notably, these Discrete Diffusion Models (DDMs) are now demonstrating strong performance in language modeling, a field long dominated by auto-regressive (AR) architectures. Within this landscape, DDM studies have diverged into two distinct branches: Masked Diffusion Language Models (MDLMs) ( 26 ) and Uniform-noise Diffusion Language Models (UDLMs) ( 28 ) . While MDLMs achieve superior performance in likelihood modeling and zero-shot generalization, they often struggle to generate coherent, contextually consistent outputs with limited inference steps. Conversely, UDLMs excel at low-step generation but frequently lag behind MDLMs when the number of inference steps increases. Despite their individual merits, neither achieves balanced performance across these dimensions. This imbalance carries significant practical implications. In domains like image synthesis, the required inference steps are typically far fewer than the total number of patches. In these few-step regimes, UDLM (and its hidden Gaussian counterparts ( 28 ; 27 ) ) consistently outperforms the masked paradigm. Concretely, our experiments demonstrate that UDLM outperforms MDLM by 17.6% when generating ImageNet-1K images in an 8-step regime. Consequently, balancing the robust few-step generation of UDLMs with the superior semantic understanding of MDLMs remains a critical open challenge. To this end, we propose miXed Diffusion Language Modeling (XDLM), a balanced theoretical formulation that establishes a rigorous bridge between uniform and masked noise distributions. Unlike GIDD ( 31 ) , which relies on a computationally expensive time-inhomogeneous process where noise characteristics shift at every timestep, XDLM enforces a stationary noise kernel where incremental noise structurally matches the marginal noise, as shown in Fig. 1 (left). This consistency allows our training objective to be efficiently factorized into static constants and dynamic schedules, avoiding the complex re-computation of noise distributions required by GIDD. In addition, we prove that MDLM and UDLM are limiting cases of our formulation. Furthermore, XDLM enables a memory-efficient implementation by algebraically simplifying posterior calculations, allowing XDLM to scale to large vocabulary sizes without prohibitive computational costs. Experimental results demonstrate that XDLM consistently achieves superior performance across diverse modalities, validating the efficacy of our balanced discrete diffusion approach. As illustrated in Fig. 1 (right), XDLM advances the Pareto frontier between understanding capability and generative quality. Notably, as the interpolation parameters shift toward their extremes, XDLM matches the performance of UDLM and MDLM respectively, confirming the model’s theoretical foundation. Concretely, in zero-shot language benchmarks, XDLM surpasses UDLM by 5.4 points in averaged metrics and trails MDLM by only 0.45. In conditional image generation, XDLM excels in both efficiency and quality: it significantly outperforms MDLM in few-step generation (reducing FID from 80.8 to 54.1 at 4 steps) and ranks first at 16 steps, surpassing UDLM by 0.4 points. Furthermore, large-scale continual pretraining on 8B-parameter LLMs ( 19 ) yields an MBPP score of 15.0 in only 32 sampling steps, improving upon the vanilla LLaDA baseline by over 120% (15.0 vs. 6.8). Finally, analysis of training dynamics reveals a distinct performance crossover: while masked baselines converge quickly but plateau early, XDLM sustains steady improvement throughout training, demonstrating superior long-term scaling. 2 Preliminary Discrete Diffusion Models (DDMs) are latent generative models defined by two Markov processes: a forward process and a reverse process. Considering a single categorical variable \mathbf{x}_{0} , represented as a one-hot vector in \{0,1\}^{N} , where N=|\mathcal{V}| denotes the vocabulary size. Let \mathbf{z}_{t} represent the latent state at time t\in[0,1] , with \mathbf{z}_{0}\coloneq\mathbf{x}_{0} . The forward process q progressively corrupts the data via transition kernels q(\mathbf{z}_{t}\mid\mathbf{z}_{s}) for s 500k steps), both models significantly outperform the MDLM and GIDD baselines throughout the process. Figure 4 : Training dynamics for (a) text and (b) image generation tasks. Colored regions indicate the dominant model during each phase, while transitions between colors mark points of performance crossover. Impact of Mixing Ratio . The mixing ratio k acts as a pivotal control parameter, interpolating between two distinct paradigms: UDLM ( k\to 1 ) and MDLM ( k\to 0 ), characterized by their divergent capabilities in generation and understanding. As shown in Fig. 1 (right), these boundary cases establish a linear Reference Trade-off Line (dashed black line), representing a zero-sum scenario where gains in generation quality (lower Generation PPL) come at the cost of understanding capability (higher Zero-Shot PPL). However, the empirical behavior of XDLM reveals that the model does not merely traverse this linear path. Crucially, varying k shifts the model’s operating point along a continuous, smooth trajectory. This coherent evolution, as opposed to stochastic fluctuations or random walks, validates the architectural design, confirming that XDLM successfully synthesizes the mechanical properties of both UDLM and MDLM into a unified framework. Moreover, this trajectory bows inward to form a superior Optimal Trade-off Frontier (solid red line), demonstrating a synergistic effect where the mixed objective yields performance gains that exceed the sum of the parts. By navigating this frontier, we identify an empirical ‘sweet spot’ at k=0.1 . As shown by the red marker, this configuration effectively breaks the Pareto frontier of the baselines, achieving robust understanding capabilities without the severe degradation in generation performance typical of pure masked models. Computational Efficiency of XDLM . We analyze the computational efficiency across inference, training, and sampling. While MDLM remains the fastest baseline due to its simple absorbing kernel, XDLM emerges as the most efficient model among those involving uniform noise. Benefiting from our scalar reformulated strategy, XDLM achieves a forward throughput of 396,398 tokens/s, nearly double GIDD’s 199,516 tokens/s, and a training throughput of 137,372 tokens/s. In generation tasks, XDLM maintains a sampling speed of 7,108 tokens/s, significantly surpassing UDLM (2,882 tokens/s). This efficiency extends to memory consumption, where XDLM requires only 31.4 GB compared to UDLM’s 59.7 GB and GIDD’s 40.9 GB. These results confirm that XDLM effectively mitigates large-vocabulary bottlenecks without compromising the expressivity of complex noise kernels. For more detailed results, please refer to Appendix K . Inference Dynamics. To investigate the internal generation mechanism of XDLM, we closely inspect a single generation trajectory over a budget of T=32 steps. The result shows that XDLM effectively leverages a hybrid noise process: it constructs semantic structure via absorbing noise ( [MASK] \to Token) while refining content through uniform noise (Token \to Token). Crucially, XDLM distinguishes itself with a re-masking strategy (transforming tokens back to [MASK] ). Unlike standard iterative models that exhibit high sensitivity to initialization noise, this mechanism allows XDLM to actively reject low-probability tokens derived from uniform noise, effectively escaping local optima. For a detailed analysis, please refer to Appendices D and E . 5 Related work Masked Diffusion Language Modeling . Building on the general and flexible discrete diffusion framework introduced by D3PM ( 1 ) , absorbing-state–based corruptions (often using a special [MASK] token) have emerged as a popular paradigm for discrete diffusion in language, frequently referred to as Masked Diffusion Language Models (MDLMs). MDLM ( 26 ) and MD4 ( 29 ) extend this framework and derive a continuous-time variational objective for masked diffusion as a simple weighted integral of cross-entropy losses, yielding a streamlined and general training recipe for discrete language models. From a continuous-time perspective, CTMC ( 2 ) formalizes the forward process via continuous-time Markov chains, while SEDD ( 16 ) learns density ratios directly, providing a discrete analogue of score matching. These advances have facilitated scaling masked diffusion LMs to larger model sizes and datasets, leading to large diffusion language models such as LLaDA ( 19 ) and Dream ( 34 ) , as well as extensions to multimodal settings ( 35 ; 33 ; 36 ) . Uniform-noise Diffusion Language Modeling . The transition kernel of uniform-noise diffusion is a mixture of the identity and a uniform distribution over the vocabulary. This idea was introduced for discrete spaces in Argmax/Multinomial diffusion ( 12 ) and generalized by D3PM ( 1 ) . Plaid ( 10 ) takes a notable step toward narrowing the likelihood gap between autoregressive models and diffusion-based language models. UDLM ( 28 ) shows that, with appropriate guidance mechanisms, uniform-noise diffusion can yield stronger controllability. Duo ( 27 ) highlights a duality between Gaussian diffusion language models (e.g., Plaid) and uniform-noise diffusion, and proposes distilling a uniform-noise model from an auxiliary latent Gaussian model. Mixed Language Modeling . D3PM ( 1 ) observes that BERT can be viewed as a one-step instance of mixed corruption with transition matrix. However, BERT is trained with a cross-entropy objective on the masked positions only, whereas diffusion-based approaches optimize objectives integrated over a noise schedule (or time), leading to different training dynamics and inductive biases. Recent research has further bridged and generalized these paradigms. ( 8 ) unify autoregressive and diffusion models by assigning distinct noise schedules to individual token positions. Similarly, ReMDM ( 32 ) relaxes the strict absorbing constraint of the forward process, allowing [MASK] tokens to revert to their original states, thereby enabling plug-and-play editing capabilities. Our most related work is GIDD ( 31 ) , which also explores interpolating between masked and uniform-noise diffusion. However, GIDD generally relies on constructing transition matrices that blend these mechanisms dynamically. In contrast, our XDLM framework enforces a stationary noise kernel. This design choice explicitly decouples the noise characteristics from the temporal dynamics, allowing us to derive a highly efficient scalar formulation that simplifies both sampling and training. 6 Conclusion In this paper, we presented the XDLM, a unified approach that theoretically bridges the gap between Masked and Uniform-noise diffusion. By redefining the forward process as a weighted row-stochastic transition, we proved that XDLM recovers existing paradigms (MDLM and UDLM) as special cases. To ensure practicality, we derived a memory-efficient implementation that reduces computational complexity, enabling training on large vocabularies. Empirically, XDLM breaks the Pareto frontier between understanding and generation. We identified a mixing ratio of k=0.1 as the optimal “sweet spot”, where the model combines the robust zero-shot likelihoods of masking models with the superior sample diversity and few-step generation quality of uniform noise models. These advantages extend across domains, achieving state-of-the-art on ImageNet-1K and demonstrating significant scalability in the continual pretraining of 8B-parameter LLMs, where XDLM doubled performance on code generation benchmarks. Limitations Despite these contributions, several avenues for future research remain. First, we have not yet trained XDLM from scratch at a large scale; such pre-training would likely allow for a more comprehensive exploration of the model’s emergent properties. Second, we did not fully investigate the “performance crossover” phenomenon, wherein UDLM and XDLM appear to outperform MDLM in generation tasks involving large sampling steps (approaching autoregressive decoding). Third, domain-specific sampling strategies for XDLM in language modeling and image generation have not yet been optimized. Furthermore, while we confirmed that XDLM balances understanding and generation, the interaction and balance between textual and visual modalities within a single unified model remain uninvestigated. Finally, the development of post-training schemas and inference acceleration techniques for XDLM remains a subject for future work. Impact Statement The primary objective of this research is to advance the technical state-of-the-art in machine learning. Consequently, it entails societal considerations parallel to those found in the general field of language and foundation models. Appendix A The Forward Process with Stationary Kernels Lemma 1 (Construction of Mixing Kernel) . Let \boldsymbol{\pi}\in\mathcal{R}^{N} be a stationary target distribution defined as a mixture of the uniform distribution \mathbf{u} and point-masses on special tokens i\in\mathcal{S} . If the noise kernel \mathbf{K} satisfies the property of instantaneous mixing (i.e., convergence to \boldsymbol{\pi} in a single step), it admits the decomposition: \mathbf{K}=\frac{k}{N}\mathbf{J}+\sum_{i\in\mathcal{S}}\mu_{i}\mathbf{M}_{i}, (20) where \mathbf{J} is the all-ones matrix, \mathbf{M}_{i} is the absorbing matrix for state i , and the weights satisfy k+\sum\mu_{i}=1 . Proof. The condition of instantaneous mixing implies that the transition probability to a destination state j is solely determined by the target marginal \pi_{j} , independent of the antecedent state s . In terms of the stochastic matrix, this imposes row-equivalence: \mathbf{K}_{sj}=\pi_{j} for all s . This is algebraically equivalent to the rank-1 factorization \mathbf{K}=\mathbf{1}\boldsymbol{\pi}^{\top} . Substituting the mixture definition \boldsymbol{\pi}=k\mathbf{u}+\sum_{i\in\mathcal{S}}\mu_{i}\mathbf{e}_{i} into this factorization yields: \displaystyle\mathbf{K} \displaystyle=\mathbf{1}\left(k\mathbf{u}+\sum_{i\in\mathcal{S}}\mu_{i}\mathbf{e}_{i}\right)^{\top} (21) \displaystyle=k(\mathbf{1}\mathbf{u}^{\top})+\sum_{i\in\mathcal{S}}\mu_{i}(\mathbf{1}\mathbf{e}_{i}^{\top}). (22) Identifying the resulting outer products with the canonical transition matrices \mathbf{1}\mathbf{u}^{\top}=N^{-1}\mathbf{J} (uniform transition) and \mathbf{1}\mathbf{e}_{i}^{\top}=\mathbf{M}_{i} (absorbing transition) concludes the proof. ∎ Appendix B Efficient Sampling and Training via Scalar Formulation Before proofing lemma , we introduce this definition: Definition B.1 . \displaystyle\mathbf{g}(s,t,\mathbf{x},\mathbf{z_{t}}) (23) \displaystyle=\alpha_{t}\mathbf{z_{t}}\odot\mathbf{x}+\alpha_{t|s}\beta_{s}\mathbf{z_{t}}\odot K^{\top}\mathbf{x}+\beta_{t|s}\alpha_{s}K\mathbf{z_{t}}\odot\mathbf{x}+\beta_{t|s}\beta_{s}K\mathbf{z_{t}}\odot K^{\top}\mathbf{x} (24) \displaystyle=\alpha_{t|s}f_{s}(\mathbf{x},\mathbf{z_{t}})\mathbf{z_{t}}+\beta_{t|s}r(\mathbf{z_{t}})\left(\alpha_{s}\mathbf{x}+\beta_{s}\left(\frac{k}{N}\mathbf{\mathbf{1}}+\mu\mathbf{m}\right)\right) (25) Thus, we have \displaystyle\mathbf{e}^{\top}\mathbf{g}(s,t,\mathbf{x},\mathbf{z_{t}}) (26) \displaystyle=\alpha_{t|s}f_{s}(\mathbf{x},\mathbf{z_{t}})\mathbf{e}^{\top}\mathbf{z_{t}}+\beta_{t|s}r(\mathbf{z_{t}})\left(\alpha_{s}\mathbf{e}^{\top}\mathbf{x}+\beta_{s}\left(\frac{k}{N}\mathbf{e}^{\top}\mathbf{\mathbf{1}}+\mu\mathbf{e}^{\top}\mathbf{m}\right)\right) (27) \displaystyle=f_{s}(\mathbf{x},\mathbf{z_{t}})\alpha_{t|s}\mathbf{e}^{\top}\mathbf{z_{t}}+\beta_{t|s}r(\mathbf{z_{t}})\left(\alpha_{s}\mathbf{e}^{\top}\mathbf{x}+\beta_{s}r(\mathbf{e})\right) (28) \displaystyle=f_{s}(\mathbf{x},\mathbf{e})\alpha_{t|s}\mathbf{e}^{\top}\mathbf{z_{t}}+\beta_{t|s}r(\mathbf{z_{t}})f_{s}(\mathbf{x},\mathbf{e}) (29) \displaystyle=f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}}) (30) Lemma 2 (Scalar Posterior) . The posterior probability of transitioning to state \mathbf{e} can be formulated as: \displaystyle q(\mathbf{z}_{s}=\mathbf{e}\mid\mathbf{z}_{t},\mathbf{x})=\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z}_{t})}{f_{t}(\mathbf{x},\mathbf{z}_{t})}. (31) The model’s reverse transition p_{\theta} follows the same form by substituting \mathbf{x} with the predicted distribution \tilde{\mathbf{x}}_{0} . Proof. \displaystyle q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\mathbf{x}) (32) \displaystyle=\mathbf{e}^{\top}\frac{(Q_{t|s}\mathbf{z_{t}})\odot(Q_{s}^{\top}\mathbf{x})}{\mathbf{z_{t}}^{\top}Q_{t}^{\top}\mathbf{x}} (33) \displaystyle=\mathbf{e}^{\top}\frac{(\alpha_{t|s}\mathbf{z_{t}}+\beta_{t|s}K\mathbf{z_{t}})\odot(\alpha_{s}\mathbf{x}+\beta_{s}K^{\top}\mathbf{x})}{\mathbf{z_{t}}^{\top}(\alpha_{t}\mathbf{x}+\beta_{t}K^{\top}\mathbf{x})} (34) \displaystyle=\mathbf{e}^{\top}\frac{\alpha_{t}\mathbf{z_{t}}\odot\mathbf{x}+\alpha_{t|s}\beta_{s}\mathbf{z_{t}}\odot K^{\top}\mathbf{x}+\beta_{t|s}\alpha_{s}K\mathbf{z_{t}}\odot\mathbf{x}+\beta_{t|s}\beta_{s}K\mathbf{z_{t}}\odot K^{\top}\mathbf{x}}{\alpha_{t}\mathbf{z_{t}}^{\top}\mathbf{x}+\beta_{t}\mathbf{z_{t}}^{\top}K^{\top}\mathbf{x}} (35) \displaystyle=\mathbf{e}^{\top}\frac{\mathbf{g}(s,t,\mathbf{x},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})} (36) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})} (37) ∎ Lemma 3 (Scalar KL Divergence) . The term D_{\text{KL}}(q(\mathbf{z}_{s}\mid\mathbf{z}_{t},\mathbf{x})\parallel p_{\theta}(\mathbf{z}_{s}\mid\mathbf{z}_{t})) can be written as: \displaystyle D_{\text{KL}}=\frac{\beta_{t|s}\alpha_{s}r(\mathbf{z}_{t})}{f_{t}(\mathbf{x},\mathbf{z}_{t})}h_{t}(\mathbf{x},\mathbf{z}_{t},\tilde{\mathbf{x}}_{0}), (38) where h_{t} is an auxiliary function collecting the logarithmic difference terms: \displaystyle h_{t}(\mathbf{x},\mathbf{z}_{t},\tilde{\mathbf{x}}_{0}) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{z}_{t})}{r(\mathbf{z}_{t})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\frac{f_{s}(\mathbf{x},\mathbf{z}_{t})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z}_{t})}{f_{t}(\mathbf{x},\mathbf{z}_{t})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z}_{t})} \displaystyle\quad-\frac{1}{\alpha_{s}}\log\frac{f_{t}(\mathbf{x},\mathbf{z}_{t})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z}_{t})}+\log\frac{f_{s}(\mathbf{x},\mathbf{x})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{x})} \displaystyle\quad+\frac{k\beta_{s}}{N\alpha_{s}}\sum_{\mathbf{e}\in\mathcal{V}}\log\frac{f_{s}(\mathbf{x},\mathbf{e})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})}. (39) Proof. \displaystyle D_{KL} \displaystyle=\sum_{\mathbf{e}}q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\mathbf{x})\log\frac{q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\mathbf{x})}{p_{\theta}(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}})} (40) \displaystyle=\sum_{\mathbf{e}}q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\mathbf{x})\log\frac{q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\mathbf{x})}{q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\tilde{\mathbf{x}}_{0})} (41) \displaystyle=\sum_{\mathbf{e}}\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\left(\log\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}-\log\frac{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}\right) (42) \displaystyle=\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}f_{s}(\mathbf{x},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})\log\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (43) \displaystyle=\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}f_{s}(\mathbf{x},\mathbf{e})\left(\alpha_{t|s}\delta_{\mathbf{e},\mathbf{z_{t}}}+\beta_{t|s}r(\mathbf{z_{t}})\right)\log\frac{f_{s}(\mathbf{x},\mathbf{e})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (44) \displaystyle=\frac{\alpha_{t|s}f_{s}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (45) \displaystyle+\frac{\beta_{t|s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}f_{s}(\mathbf{x},\mathbf{e}) (46) \displaystyle+\frac{\beta_{t|s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}f_{s}(\mathbf{x},\mathbf{e})\log\frac{f_{s}(\mathbf{x},\mathbf{e})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (47) \displaystyle=\frac{\alpha_{t|s}f_{s}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{\beta_{t|s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (48) \displaystyle+\frac{\beta_{t|s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}\left(\alpha_{s}\delta_{\mathbf{x},\mathbf{e}}+\beta_{s}r(\mathbf{e})\right)\log\frac{f_{s}(\mathbf{x},\mathbf{e})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (49) \displaystyle=\frac{\alpha_{t|s}f_{s}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{\beta_{t|s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (50) \displaystyle+\frac{\beta_{t|s}\alpha_{s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{s}(\mathbf{x},\mathbf{x})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{x})} (51) \displaystyle+\frac{k\beta_{t|s}\beta_{s}r(\mathbf{z_{t}})}{Nf_{t}(\mathbf{x},\mathbf{z_{t}})}\sum_{\mathbf{e}}\log\frac{f_{s}(\mathbf{x},\mathbf{e})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})}+\frac{\mu\beta_{t|s}\beta_{s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{s}(\mathbf{x},\mathbf{m})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{m})} (52) \displaystyle=\frac{\beta_{t|s}\alpha_{s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}h_{t}(\mathbf{x},\mathbf{z_{t}}) (53) We omit the term \log{f_{s}(\mathbf{x},\mathbf{m})}/{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{m})} as it equals zero for all \mathbf{x} and \tilde{\mathbf{x}}_{0} . ∎ Lemma 4 (Limiting Case) . As s\to t , the function h_{t} converges to: \displaystyle h_{t}(\mathbf{x},\mathbf{z}_{t},\tilde{\mathbf{x}}_{0}) \displaystyle\approx\frac{p_{\mathbf{x},\mathbf{z}_{t}}-p_{\tilde{\mathbf{x}}_{0},\mathbf{z}_{t}}}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z}_{t})}-\frac{1}{\alpha_{t}}\log\frac{f_{t}(\mathbf{x},\mathbf{z}_{t})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z}_{t})}+ \displaystyle\log\frac{f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})} \displaystyle+\frac{k\beta_{t}}{N\alpha_{t}}\sum_{\mathbf{e}\in\mathcal{V}}\log\frac{f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})}. (54) Proof. Notice that when s\to t , \beta_{t|s}\alpha_{s}\to 0 , which is a 0/0 div. by applying L’Hôpital’s rule. (the up and down both differentiate by \alpha_{t|s} ), we have \displaystyle\lim_{s\to t}\frac{1}{\beta_{t|s}\alpha_{s}}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (55) \displaystyle=\frac{-1}{\alpha_{s}}\frac{\partial}{\partial\alpha_{t|s}}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (56) \displaystyle=\frac{-1}{\alpha_{s}}\left(\frac{\partial\log f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{\partial\alpha_{t|s}}-\frac{\partial\log f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{\partial\alpha_{t|s}}\right) (57) \displaystyle=\frac{-1}{\alpha_{s}}\left(\frac{\partial\log f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{\partial\alpha_{t|s}}-\frac{\partial\log f_{t}(\mathbf{x},\mathbf{z_{t}})}{\partial\alpha_{t|s}}\right) (58) \displaystyle=\frac{-1}{\alpha_{s}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}\frac{\partial(\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+(1-\alpha_{t})r(\mathbf{z_{t}}))}{\partial\alpha_{t|s}}-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\frac{\partial f_{t}(\mathbf{x},\mathbf{z_{t}})}{\partial\alpha_{t|s}}\right) (59) \displaystyle=\frac{-1}{\alpha_{s}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}\left(\alpha_{s}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}-\alpha_{s}r(\mathbf{z_{t}})\right)-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\frac{\partial f_{t}(\mathbf{x},\mathbf{z_{t}})}{\partial\alpha_{t|s}}\right) (60) \displaystyle=-\frac{p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}-r(\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\frac{p_{\mathbf{x},\mathbf{z_{t}}}-r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})} (61) \displaystyle=\frac{(\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}r(\mathbf{z_{t}}))(p_{\mathbf{x},\mathbf{z_{t}}}-r(\mathbf{z_{t}}))-(\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}+\beta_{t}r(\mathbf{z_{t}}))(p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}-r(\mathbf{z_{t}}))}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (62) \displaystyle=\frac{-\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}r(\mathbf{z_{t}})+\beta_{t}p_{\mathbf{x},\mathbf{z_{t}}}r(\mathbf{z_{t}})+\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}r(\mathbf{z_{t}})-\beta_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (63) \displaystyle=\frac{r(\mathbf{z_{t}})(p_{\mathbf{x},\mathbf{z_{t}}}-p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (64) so \displaystyle\lim_{s\to t}\left(\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})}{r(\mathbf{z_{t}})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}\right)=\frac{p_{\mathbf{x},\mathbf{z_{t}}}-p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (65) ∎ Appendix C Relationship to MDLM and UDLM Lemma 5 . XDLM can be reduced to MDLM by setting k=0 , with the posterior probability: \displaystyle p_{\theta}(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}}) \displaystyle=\delta_{\mathbf{z_{t}},\mathbf{m}}\delta_{\mathbf{e},\mathbf{m}}\frac{\beta_{s}}{\beta_{t}}+\delta_{\mathbf{z_{t}},\mathbf{m}}\delta_{\mathbf{e}\neq\mathbf{m}}\frac{\beta_{t|s}\alpha_{s}p_{\theta,\mathbf{e}}}{\beta_{t}}+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\delta_{\mathbf{e},\mathbf{z_{t}}} (66) \displaystyle=\begin{cases}\frac{\beta_{s}}{\beta_{t}}\delta_{\mathbf{m},\mathbf{z_{t}}},&\mathbf{e}=\mathbf{m}\\ \frac{(\alpha_{s}-\alpha_{t})}{1-\alpha_{t}}p_{\theta,\mathbf{e}},&\mathbf{z_{t}}=\mathbf{m},\mathbf{e}\neq\mathbf{m}\\ \delta_{\mathbf{e},\mathbf{z_{t}}},&\mathbf{z_{t}}\neq\mathbf{m},\mathbf{e}\neq\mathbf{m}\\ \end{cases} (67) and kl divergence \displaystyle KL=-\frac{\beta_{t|s}\alpha_{s}}{\beta_{t}}\log p_{\theta,\mathbf{x}} (68) Proof. since k=0 , \mu=1 , we have \displaystyle r(\mathbf{e}) \displaystyle=\delta_{\mathbf{e},\mathbf{m}} (69) \displaystyle p_{\tilde{\mathbf{x}}_{0},\mathbf{e}} \displaystyle=\delta_{\mathbf{e}\neq\mathbf{m}}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}} (70) hence \displaystyle q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\tilde{\mathbf{x}}_{0}) \displaystyle=\frac{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (71) \displaystyle=\frac{(\alpha_{s}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\beta_{s}r(\mathbf{e}))(\alpha_{t|s}p_{\mathbf{e},\mathbf{z_{t}}}+\beta_{t|s}r(\mathbf{z_{t}}))}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}r(\mathbf{z_{t}})} (72) \displaystyle=\frac{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}\delta_{\mathbf{e},\mathbf{z_{t}}}+\alpha_{s}\beta_{t|s}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}\delta_{\mathbf{z_{t}},\mathbf{m}}+\alpha_{t|s}\beta_{s}\delta_{\mathbf{e},\mathbf{z_{t}}}\delta_{\mathbf{e},\mathbf{m}}+\beta_{s}\beta_{t|s}\delta_{\mathbf{e},\mathbf{m}}\delta_{\mathbf{z_{t}},\mathbf{m}}}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}} (73) \displaystyle=\delta_{\mathbf{z_{t}},\mathbf{m}}\frac{\alpha_{s}\beta_{t|s}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\alpha_{t|s}\beta_{s}\delta_{\mathbf{e},\mathbf{z_{t}}}+\beta_{s}\beta_{t|s}\delta_{\mathbf{e},\mathbf{z_{t}}}}{\beta_{t}} (74) \displaystyle+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\frac{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}\delta_{\mathbf{e},\mathbf{z_{t}}}}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}} (75) \displaystyle=\delta_{\mathbf{z_{t}},\mathbf{m}}\left(\frac{\alpha_{s}\beta_{t|s}}{\beta_{t}}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\frac{\beta_{s}}{\beta_{t}}\delta_{\mathbf{e},\mathbf{z_{t}}}\right)+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\delta_{\mathbf{e},\mathbf{z_{t}}}\frac{p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}}{p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}} (76) \displaystyle=\delta_{\mathbf{z_{t}},\mathbf{m}}\frac{\alpha_{s}\beta_{t|s}}{\beta_{t}}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\frac{\beta_{s}}{\beta_{t}}\delta_{\mathbf{z_{t}},\mathbf{m}}\delta_{\mathbf{e},\mathbf{z_{t}}}+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\delta_{\mathbf{e},\mathbf{z_{t}}} (77) for kl, we have \displaystyle h_{t}(\mathbf{x},\mathbf{z_{t}}) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})}{r(\mathbf{z_{t}})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (78) \displaystyle-\frac{1}{\alpha_{s}}\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\log\frac{f_{s}(\mathbf{x},\mathbf{x})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{x})}+\frac{k\beta_{s}}{N\alpha_{s}}\sum_{\mathbf{e}}\log\frac{f_{s}(\mathbf{x},\mathbf{e})}{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (79) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})}{r(\mathbf{z_{t}})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\frac{(\alpha_{s}p_{\mathbf{x},\mathbf{z_{t}}}+\beta_{s}\delta_{\mathbf{z_{t}},\mathbf{m}})(\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}})}{(\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}})(\alpha_{s}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{s}\delta_{\mathbf{z_{t}},\mathbf{m}})} (80) \displaystyle-\frac{1}{\alpha_{s}}\log\frac{\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}-\log p_{\tilde{\mathbf{x}}_{0},\mathbf{x}} (81) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})}{r(\mathbf{z_{t}})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\left(\delta_{\mathbf{z_{t}},\mathbf{m}}\frac{\beta_{s}\delta_{\mathbf{z_{t}},\mathbf{m}}\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}{\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}\beta_{s}\delta_{\mathbf{z_{t}},\mathbf{m}}}+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\frac{\alpha_{s}p_{\mathbf{x},\mathbf{z_{t}}}\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}{\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}\alpha_{s}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}\right) (82) \displaystyle-\frac{1}{\alpha_{s}}\log\left(\delta_{\mathbf{z_{t}},\mathbf{m}}\frac{\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}{\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}+\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\frac{\alpha_{t}p_{\mathbf{x},\mathbf{z_{t}}}}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}\right)-\log p_{\tilde{\mathbf{x}}_{0},\mathbf{x}} (83) \displaystyle=\frac{f_{s}(\mathbf{x},\mathbf{z_{t}})}{r(\mathbf{z_{t}})}\frac{\alpha_{t|s}}{\beta_{t|s}\alpha_{s}}\log\left(1\right)-\frac{1}{\alpha_{s}}\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\log\frac{p_{\mathbf{x},\mathbf{z_{t}}}}{p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}-\log p_{\tilde{\mathbf{x}}_{0},\mathbf{x}} (84) so \displaystyle D_{KL} \displaystyle=\frac{\beta_{t|s}\alpha_{s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}h_{t}(\mathbf{x},\mathbf{z_{t}}) (85) \displaystyle=-\frac{\beta_{t|s}\delta_{\mathbf{z_{t}},\mathbf{m}}}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\delta_{\mathbf{z_{t}}\neq\mathbf{m}}\log\frac{p_{\mathbf{x},\mathbf{z_{t}}}}{p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}-\frac{\beta_{t|s}\alpha_{s}\delta_{\mathbf{z_{t}},\mathbf{m}}}{\alpha_{t}\delta_{\mathbf{x},\mathbf{z_{t}}}+\beta_{t}\delta_{\mathbf{z_{t}},\mathbf{m}}}\log p_{\tilde{\mathbf{x}}_{0},\mathbf{x}} (86) \displaystyle=-\delta_{\mathbf{z_{t}},\mathbf{m}}\frac{\beta_{t|s}\alpha_{s}}{\beta_{t}}\log p_{\tilde{\mathbf{x}}_{0},\mathbf{x}} (87) ∎ Lemma 6 . XDLM can be reduced to UDLM by setting k=1 , with the posterior probability: \displaystyle p_{\theta}(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}}) \displaystyle=\frac{\delta_{\mathbf{e},\mathbf{z_{t}}}(N\alpha_{t}p_{\theta,\mathbf{e}}+\beta_{s}\alpha_{t|s})+(\alpha_{s}-\alpha_{t})p_{\theta,\mathbf{e}}+\beta_{s}\beta_{t|s}/N}{N\alpha_{t}p_{\theta,\mathbf{z_{t}}}+\beta_{t}} (88) and kl divergance \displaystyle KL \displaystyle=\frac{-\beta_{t|s}\alpha_{s}}{N\alpha_{t}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}-\sum_{\mathbf{e}\neq\mathbf{z_{t}}}\frac{f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})}\right) (89) Proof. since k=1 , \mu=0 , we have \displaystyle\forall\mathbf{e},\,r(\mathbf{e})=r=\frac{1}{N} (90) hence \displaystyle q(\mathbf{z_{s}}=\mathbf{e}|\mathbf{z_{t}},\tilde{\mathbf{x}}_{0}) \displaystyle=\frac{f_{s}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t|s}(\mathbf{e},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})} (91) \displaystyle=\frac{(\alpha_{s}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\beta_{s}/N)(\alpha_{t|s}p_{\mathbf{e},\mathbf{z_{t}}}+\beta_{t|s}/N)}{\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}/N} (92) \displaystyle=\frac{N\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}\delta_{\mathbf{e},\mathbf{z_{t}}}+\alpha_{s}\beta_{t|s}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\alpha_{t|s}\beta_{s}\delta_{\mathbf{e},\mathbf{z_{t}}}+\beta_{s}\beta_{t|s}/N}{N\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}} (93) \displaystyle=\frac{\delta_{\mathbf{e},\mathbf{z_{t}}}(N\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\alpha_{t|s}\beta_{s})+(\alpha_{s}-\alpha_{t})p_{\tilde{\mathbf{x}}_{0},\mathbf{e}}+\beta_{s}\beta_{t|s}/N}{N\alpha_{t}p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}+\beta_{t}} (94) given s\to t , for kl, we have \displaystyle h_{t}(\mathbf{x},\mathbf{z_{t}}) (95) \displaystyle=\frac{p_{\mathbf{x},\mathbf{z_{t}}}-p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{\alpha_{t}}\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\log\frac{f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})}+\frac{k\beta_{t}}{N\alpha_{t}}\sum_{\mathbf{e}}\log\frac{f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})} (96) \displaystyle=f_{t}(\mathbf{x},\mathbf{z_{t}})\frac{p_{\mathbf{x},\mathbf{z_{t}}}-p_{\tilde{\mathbf{x}}_{0},\mathbf{z_{t}}}}{f_{t}(\mathbf{x},\mathbf{z_{t}})f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{\alpha_{t}}\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\log\frac{f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})} (97) \displaystyle+\frac{\beta_{t}}{N\alpha_{t}}\sum_{\mathbf{e}}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})}-\frac{\beta_{t}}{N\alpha_{t}}\sum_{\mathbf{e}}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})} (98) \displaystyle=\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{\alpha_{t}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\right)-\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\log\frac{f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})} (99) \displaystyle+\frac{\beta_{t}}{N\alpha_{t}}\sum_{\mathbf{e}}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})} (100) \displaystyle=\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{\alpha_{t}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\right)-\log\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}+\log\frac{f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})} (101) \displaystyle+\frac{1}{\alpha_{t}}\sum_{\mathbf{e}}f_{t}(\mathbf{x},\mathbf{e})\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})}-\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{x})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{x})f_{t}(\mathbf{x},\mathbf{z_{t}})} (102) \displaystyle=\frac{f_{t}(\mathbf{x},\mathbf{z_{t}})}{\alpha_{t}}\left(\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\right)+\frac{1}{\alpha_{t}}\sum_{\mathbf{e}}f_{t}(\mathbf{x},\mathbf{e})\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})} (103) so \displaystyle D_{KL} \displaystyle=\frac{\beta_{t|s}\alpha_{s}r(\mathbf{z_{t}})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}h_{t}(\mathbf{x},\mathbf{z_{t}}) (104) \displaystyle=\frac{-\beta_{t|s}\alpha_{s}}{N\alpha_{t}}\left(\frac{1}{f_{t}(\mathbf{x},\mathbf{z_{t}})}-\frac{1}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})}-\sum_{\mathbf{e}}\frac{f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\mathbf{x},\mathbf{z_{t}})}\log\frac{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{z_{t}})f_{t}(\mathbf{x},\mathbf{e})}{f_{t}(\tilde{\mathbf{x}}_{0},\mathbf{e})f_{t}(\mathbf{x},\mathbf{z_{t}})}\right) (105) ∎ Appendix D LM1B Sampling Case We investigate the relationship between the inference computational budget (defined by the number of sampling steps T ) and the perceptual quality of the generated text. Tab. 5 presents randomly selected, non-curated samples from the XDLM model trained on the LM1B. Each sample represents the final output of a distinct diffusion process constrained to a specific total step count T\in\{4,8,16,32,64,128\} . At the lower bound of the sampling budget ( T=4 ), the model fails to converge to the data manifold. the output is characterized by disjointed lexical retrieval (e.g., “ the advertising -tag prong ”) where individual tokens are valid but lack syntactic binding. Increasing the budget to T=8 yields the emergence of superficial syntactic structures (e.g., “ curt schilling is a man of newness ”), yet the semantic content remains illogical, indicating that the diffusion process resolves local grammatical dependencies prior to higher-order semantic meaning. As the sampling steps increase to the intermediate regime ( T=16 and T=32 ), we observe a transition from mere grammatical correctness to narrative plausibility. At T=16 , the text exhibits valid sentence structures but suffers from tautological repetition and thematic drift (e.g., “ middle east in the middle east ”). By T=32 , the model generates cohesive clauses with clear subject-verb-object relationships, although the specific entities often remain synthetic or hallucinatory (e.g., “ mr poundhead ”). This suggests that while 32 steps are sufficient for the model to learn the structural rules of the language, the denoising process has not yet fully aligned the output with the specific factual distributions of the training corpus. A distinct phase transition in sample quality is observed at T=64 , which appears to represent the saturation point for high-fidelity generation. At this stage, the model produces semantically robust text indistinguishable from natural news data, successfully handling complex entity lists (e.g., “ Congo, Benin, Ivory Coast ”) and domain-specific terminology (e.g., “ National Transportation Safety Board ”). Extending the process to T=128 yields marginal improvements in long-range consistency and the precision of quantitative figures, but the perceptual gain over T=64 is significantly smaller than the leap observed from T=32 . This trajectory indicates that while XDLM requires a minimum threshold of approximately 64 steps to resolve fine-grained semantic details, further increases in computational cost yield diminishing returns in generation quality. Table 5 : Uncurated samples generated by XDLM trained on LM1B. Each sample represents the output of a distinct diffusion process constrained to a specific total step count T . The sequence length is fixed at 128 tokens. Steps ( T ) Generated Text Sample T=4 [CLS]n. [CLS] the advertising -tag prong those former candidates is sept. [CLS] new and aspiring contenders boasted a present, confusing not candidates not only for 15 - u running bonds. [CLS] passengers on board the puerto rico jet jet out to the ash oftwa, hawaii, wednesday that afternoon. [CLS] as a result of the fusion going from in horsepower and to okter and the ram and to gm, s rentalearing / thirds was the infall. [CLS] times square may undergo restorations [CLS] and " old allah " is an homage to " the a, the " god " in snowy islamic faith. [CLS] unfortunately christina sees, [CLS] T=8 [CLS] be obese to go over. [CLS] curt schilling is a man of newness. [CLS] mr middle had, having not started the business for 15 years, wondered for one only anyone on board had once wanted from kraft. [CLS] the new oft - appointed owner proposed wednesday that the administration should take a " punitive tax supplement from treasury " and insisted on indiana and michigan adhering to gm’s contract terms. [CLS] ms. cummings : one at times decides to undergo restoration. [CLS] and for now pakistani parliament is now resigned to a democratic, destabilization vote in parting zardari, which sees itself [CLS] T=16 [CLS] " fails to clear advertising restrictions. [CLS] the network, which is the only known new middle east in the middle east, is expected for about 10 to 15 years. [CLS] i believe the people on board had called very pleased, how i was able to explain it in me. [CLS] the studio 86 is nearly a lot of contemporary televised - auto show - marquees - cars. the pre - gm bio tasted like two - thirds of iconic americans : one - third of the time that ford used and around 4 years the original was back to a building it made years of. [CLS] in contrast, zardari, who sees washington [CLS] T=32 [CLS] me, to clear the books. [CLS] that sense of neutrality is likely to force some postponements of the dispute, which was resolved wednesday. [CLS] 15 told the telegraph that he always believed the board had convinced him it was time to take the train. [CLS] mr poundhead wants the city council to create a series of new visitors’areas. [CLS] marfor, according to le monde, " is the afghans to steal or take and foot out " - - money the taliban has often used to buy from pakistani taliban, not pakistan’s. [CLS] though some of its friends in the west zardari said he sees k [CLS] T=64 [CLS] me, was the crew of the xv230, which is carrying congo, benin, ivory coast, cape verde, manchester russians and coastguards at luanda. [CLS] the national transportation safety board had called the electrical life support vests to a safe spot above hawaii. [CLS] the daily herald said analysts expect earnings of $ 394 million, or 43 cents a share, due to better - than - expected growth in the mini market. [CLS] general cuddy writes that the controls could be used to prevent revenge attacks, but not to halt the protests by those suspected of disturbing network in the island, throwing up the theory. [CLS] [CLS] T=128 [CLS] me to represent the cuban people they represent. [CLS] the leadership is still taking control of the company, but the problem has not been resolved. [CLS] to tony fernanda santos, of eey investments board, it thought up life’s 328 million - and one - pound tax loss of $ 86 million and projected earnings of $ 37 million next year to 2011. [CLS] next up is " best of belies, " the first mini - series. [CLS] cusuoga, the nationalist method often used to describe revenge attacks, is not experienced in most protests against a presence of the tigers in the island’s sinhala territories. [CLS] [CLS] To investigate the internal generation mechanism of XDLM, we visualize a single generation trajectory with a total budget of T=32 steps and a fixed sequence length of 128 tokens. Tab. 6 details the evolution of the sequence at steps t\in\{0,1,8,16,24,32\} . We utilize a color-coded schema to distinguish the specific transition types inherent to the hybrid noise process. Green denotes transitions from the absorbing state ( [MASK] ) to a token (typical of Masked DLMs), Blue denotes token-to-token refinement (typical of Uniform DLMs), and Red denotes the re-masking operation (token back to [MASK] ). This red transition is particular to XDLM, where the forward process is a mixture of absorbing and uniform noise, allowing the model to stochastically backtrack by rejecting previously generated tokens. The process initializes at t=0 with a mixture of masks and random tokens derived from uniform noise, such as “ coffin ,” “ slippery ,” and “ trumpets .” A critical dynamic is observed immediately at the transition to t=1 . The model actively rejects the majority of these random initializations. We observe extensive Red markings where tokens like “ coffin ” and “ trumpets ” are reverted to [MASK] . This demonstrates that XDLM is not strictly bound by its random initialization from the uniform noise. Unlike standard iterative refinement models that might struggle to escape local minima induced by poor random seeds, XDLM possesses the capacity to identify low-probability tokens and “clean the slate” via re-masking. Simultaneously, valid anchor tokens begin to emerge ( Green ), such as “ shillings ” and “ blocked ,” establishing an initial semantic direction. As the generation progresses through the intermediate steps ( t=1\to t=16 ), the model engages in a dynamic interplay of generation and error correction. The presence of all three transition types indicates a non-monotonic search process. The model generates new candidates ( Green ) and refines existing ones ( Blue ), but crucially, it continues to utilize re-masking ( Red ) to prune inconsistent branches. For instance, tokens generated tentatively in early steps are masked again when they fail to align with the evolving global context. By the final phase ( t=24\to t=32 ), the dynamics shift characteristically. The Red re-masking transitions disappear entirely, indicating structural convergence. The process focuses exclusively on filling remaining gaps ( Green ) and performing fine-grained lexical substitutions ( Blue ). Notable refinements occur here, such as the token “ life infancy ” at step 24 being refined to “ was time ” at step 32, and “ from pound ” refining to “ from pound-land .” This trajectory confirms that XDLM successfully anneals from a high-temperature exploration state—characterized by active deletion and re-generation—to a low-temperature refinement state, ensuring the final output is both globally coherent and locally precise. Table 6 : Step-wise evolution of a generated sequence ( T=32 ). Text colors indicate the transition dynamics inherent to the hybrid noise process: Green represents new tokens generated from masks; Blue represents lexical refinement; and Red highlights the re-masking operation where previously generated tokens are rejected and reverted to [MASK] . Steps ( T ) Generated Text Sample T=0 [MASK] [MASK] faltered [MASK] [MASK] coffin [MASK] [MASK] [MASK] [MASK] och [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] , [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] ##agawa [MASK] [MASK] [MASK] [MASK] slippery [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] harrington [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] trumpets surpassing [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] refuse [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] ##was veteran jan [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] begs [MASK] [MASK] T=1 [MASK] [MASK] faltered [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] ##fo [MASK] is [MASK] [MASK] shillings [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] 1879 [MASK] [MASK] [MASK] [MASK] [MASK] ##anda [MASK] blocked [MASK] ##agawa [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] engined [MASK] [MASK] [MASK] a [MASK] [MASK] ##itive [MASK] [MASK] [MASK] [MASK] [MASK] mar [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] distributing [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] tertiary [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] strode [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] whatever [MASK] ##rda [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] T=8 [MASK] [MASK] [MASK] to clear [MASK] [MASK] [MASK] [CLS] [MASK] [MASK] of [MASK] is [MASK] congo [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] ##rians santa , [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] 15 [MASK] ##anda [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] board had [MASK] ##ffi [MASK] [MASK] [MASK] ##illo [MASK] the [MASK] [MASK] [CLS] [MASK] hawaii [MASK] [MASK] the [MASK] 86 to [MASK] a [MASK] of process [MASK] [MASK] [MASK] [MASK] [CLS] mar [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] , [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] brazil [MASK] [MASK] [MASK] [MASK] [MASK] the [MASK] [MASK] [MASK] used [MASK] [MASK] [MASK] pakistani [MASK] [MASK] [MASK] [MASK] ’ s [MASK] [MASK] ##west [MASK] [MASK] [MASK] [MASK] in [MASK] [MASK] za ##rda [MASK] [MASK] [MASK] sees [MASK] [MASK] T=16 [MASK] [MASK] [MASK] to clear [MASK] [MASK] [MASK] [CLS] [MASK] [MASK] of [MASK] is [MASK] congo [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] the [MASK] , which [MASK] resolved obscene . [CLS] 15 [MASK] ##anda [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] board had convinced ##ffi [MASK] life [MASK] [MASK] take the [MASK] [MASK] [CLS] [MASK] hawaii [MASK] [MASK] the city 86 to [MASK] a [MASK] of new [MASK] [MASK] areas [MASK] [CLS] mar [MASK] vice [MASK] [MASK] le [MASK] , [MASK] [MASK] [MASK] afghan [MASK] to steal [MASK] [MASK] [MASK] [MASK] out [MASK] - [MASK] [MASK] the taliban has often used to [MASK] from pakistani [MASK] [MASK] not pakistan ’ s [MASK] [MASK] though [MASK] of its [MASK] in [MASK] [MASK] za ##rda ##ri said [MASK] sees [MASK] [MASK] T=24 [MASK] [MASK] [MASK] to clear [MASK] [MASK] . [CLS] that [MASK] of neutrality is [MASK] congo force some post ##pone ##ments of the [MASK] , which [MASK] resolved [MASK] . [CLS] 15 [MASK] the telegraph that [MASK] [MASK] believed the board had convinced him [MASK] life infancy [MASK] take the train [MASK] [CLS] from pound [MASK] , the city plans to [MASK] a [MASK] of new [MASK] [MASK] areas [MASK] [CLS] mar [MASK] , according [MASK] le monde , " is the afghan ##s to steal or [MASK] and foot out " - - [MASK] the taliban has often used to [MASK] from pakistani taliban [MASK] not pakistan ’ s [MASK] [MASK] though [MASK] of its friends in the [MASK] za ##rda ##ri said he sees [MASK] [MASK] T=32 [CLS] me , to clear the flag . [CLS] that kind of neutrality is likely to force some post ##pone ##ments of the dispute , which was resolved wednesday . [CLS] 15 told the telegraph that he always believed the board had convinced him it was time to take the train . [CLS] from pound ##land , the city plans to build a series of new retail residential areas . [CLS] mar ##for , according to le monde , " is the afghan ##s to steal or take and foot out " - - money the taliban has often used to buy from pakistani taliban , not pakistan ’ s . [CLS] though some of its friends in the west za ##rda ##ri said he sees k [CLS] Appendix E ImageNet Sampling Case We conduct a qualitative comparison of XDLM against the baseline models MDLM, GIDD, and UDLM on ImageNet-1K. The results are analyzed across two settings: unguided generation (Fig. 5 ) and generation with slight classifier-free guidance (Fig. 6 ). The visual evidence underscores XDLM’s unique ability to combine the strengths of absorbing and uniform noise strategies while mitigating their individual weaknesses. The unguided setting serves as a stress test for structural priors. Here, the limitations of MDLM and GIDD are most apparent. These models struggle to form coherent geometries, resulting in chaotic texture blobs (e.g., the flamingo and ladybug columns) rather than distinct objects. Conversely, UDLM, which relies solely on uniform noise, successfully captures global shapes but fails to generate high-frequency details, resulting in blurry, out-of-focus images. XDLM distinguishes itself by balancing these two extremes. By integrating the structural rigidity of MDLM with the refinement flexibility of UDLM, XDLM achieves structural coherence without sacrificing sharpness. For instance, in the rocket launch example, XDLM is the only model that renders a clear vertical fuselage and distinct smoke plumes without guidance. As illustrated in Fig. 6 , introducing a guidance scale of 2.0 improves general sample quality, yet the characteristic behaviors of the models persist. MDLM continues to suffer from lack of detail refinement, often producing over-saturated textures, with poor semantic structure. UDLM remains overly smooth; while the images are clean, they lack the crispness required for photorealism. XDLM leverages the guidance most effectively to enhance detail. This is evident in the flock of flamingos. XDLM renders distinct, individual birds with sharp feathers, whereas MDLM merges them into a mass and UDLM blurs the boundaries. Similarly, in the food categories (strawberries and pizza), XDLM generates realistic surface textures and lighting reflections that are absent in the UDLM samples and distorted in the MDLM samples. This confirms that XDLM’s hybrid noise mechanism provides a superior foundation for high-fidelity generation. Figure 5 : Qualitative comparison of class-conditional generation on ImageNet-1K without CFG. In the absence of guidance, baseline models struggle significantly: MDLM and GIDD produce chaotic artifacts with poor structural coherence, while UDLM yields recognizable but over-smoothed images. XDLM, by effectively balancing the characteristics of MDLM and UDLM, generates the most coherent and semantically correct samples (e.g., the distinct rocket structure and pizza toppings) even without guidance. Figure 6 : Qualitative comparison of class-conditional generation on ImageNet-1K with a CFG scale of 2.0 . With the addition of guidance, all models improve, but quality disparities remain. MDLM and GIDD still exhibit texture distortion and "burn" artifacts. UDLM produces clean but blurry outputs, lacking fine-grained texture. XDLM demonstrates superior fidelity, combining sharp high-frequency details (seen in the strawberry and volcanoes) with accurate global structure. We further analyze the sampling dynamics of XDLM ( k=0.1 ) over an 8-step generation process on ImageNet-1K with a CFG of 2.0 , comparing it against MDLM, GIDD, and UDLM. The visual progression highlights distinct behaviors in how each model handles noise and feature refinement. A critical advantage of XDLM is its strong refinement capability, facilitated by the integration of uniform noise throughout the sampling process. As observed in rows 3 and 4, both XDLM and UDLM begin generating key semantic features, specifically the dog’s nose and eyes, at the very beginning of the generation cycle. In contrast, MDLM and GIDD adopt a more conservative approach. They exhibit little to no refinement of these specific features. Consequently, by the final step [8/8], MDLM and GIDD still fail to resolve the proper structure of the nose and eyes, leaving them distorted or missing. XDLM, however, establishes these features early and refines them continuously, resulting in a anatomically correct and aesthetically pleasing final output. The models also differ significantly in their ability to generate fine textures, such as the dog’s fur. Because MDLM lacks the ability to correct or refine previously generated tokens, the resulting fur texture appears rough and incoherent. While GIDD produces more delicate fur than MDLM, its performance is hindered by its reliance on time-variant noise, where the beneficial uniform noise mechanism only functions effectively during the very last step, limiting the potential for gradual improvement. Conversely, XDLM refines the fur texture consistently across all steps, ultimately producing the most realistic and high-fidelity texture among the group. Last but not least, the comparison demonstrates that relying exclusively on uniform noise, as seen in UDLM, is insufficient for optimal image quality. While UDLM shares XDLM’s ability to locate features early, the final generated image is significantly more blurred. This lack of sharpness results in a loss of fine-grained details compared to XDLM, which successfully balances structural coherence with sharp, detailed textures. Figure 7 : Visual comparison of sampling dynamics over 8 steps on ImageNet-1K. The figure contrasts the generation trajectories of MDLM, GIDD, XDLM ( k=0.1 ), and UDLM with CFG scale =2.0 . XDLM demonstrates superior structural coherence and detail refinement compared to baseline models, successfully transitioning from noise to a high-fidelity image. Appendix F Detailed Configurations of Experiments Language Modeling Following established methodologies ( 27 ; 28 ; 26 ) , we evaluate our model on standard benchmarks using the LM1B ( 3 ) and OpenWebText (OWT) ( 9 ) datasets. For LM1B, we detokenize the dataset following ( 16 ; 26 ; 27 ) and evaluate with ( 1 ; 27 ) sequence packing. Sequences are processed with a context length of 128 using the bert-base-uncased tokenizer ( 6 ) . For OWT, we use the GPT-2 tokenizer ( 23 ) with a context length of 1,024. Sequences are packed to a maximum length of 1,024 with an EOS token delimiter. As OWT lacks a standard validation split, we reserve the last 100k documents for validation ( 26 ) . Consistent with prior work ( 26 ; 28 ; 27 ) , we parameterize both XDLM and baselines using the modified Diffusion Transformer (DiT) architecture ( 21 ) adapted from ( 16 ) . All models employ 12 layers, 12 attention heads, a hidden dimension of 768, and a timestep embedding dimension of 128. The dropout rate is set to 0.1, and input/output word embeddings are untied. To ensure fair comparison, we retain the AdaLN branch but fix the input to zero instead of using log-linear scheduled time embedding. We train using AdamW ( \beta_{1}=0.9,\beta_{2}=0.999 , weight decay 0 ) with a global batch size of 512. The learning rate is warmed up to 3\times 10^{-4} over the first 2,500 steps and held constant thereafter. We apply an Exponential Moving Average (EMA) rate of 0.9999 ( 26 ; 27 ) . Image Generation We evaluate on discretized CIFAR-10 ( 14 ) and ImageNet-1K ( 5 ) . On ImageNet-1K dataset, We use the VQ-VAE tokenizer from LlamaGen ( 30 ) , , which compresses images into sequences of length 256. The backbone architecture mirrors our language modeling transformer, augmented with class-conditional embeddings to enable class-conditional generation. Training hyperparameters follow the language modeling configuration, with the training duration set to 0.5M steps. For CIFAR-10, we treat raw RGB pixel values as discrete tokens (vocabulary size 256) and flatten the images into sequences of length 32\times 32\times 3 , following ( 28 ) . We employ a U-Net backbone ( 25 ; 11 ) with a discretized truncated logistic output distribution ( 1 ; 26 ) . Class conditioning is implemented by adding label embeddings to the timestep embeddings ( 7 ) . The model is trained for 300k steps using AdamW ( \beta_{1}=0.9,\beta_{2}=0.999 , weight decay 0 ) with a batch size of 512. We use a learning rate warmup of 5,000 steps peaking at 2\times 10^{-4} and maintain an Exponential Moving Average (EMA) rate of 0.9999. Large Language Model Tuning We scale XDLM to large language modeling by performing continual pretraining on LLaDA LLaDA ( 19 ) , a model originally trained via MDLM. We utilize a 10-billion-token subset of the FineWeb-Edu dataset ( 22 ) . The training configuration employs a sequence length of 4,096 and a global batch size of 512. The learning rate is warmed up to 2\times 10^{-5} over 100 steps and held constant for the remaining 500 steps. Unless otherwise specified, we set the mixing ratio to k=0.1 . All experiments were conducted on a node equipped with 8\times Nvidia H800 GPUs. Appendix G Details of Zeroshot Capability of XDLM Trained on OWT To comprehensively assess the generalization capabilities of XDLM, we evaluate zero-shot perplexity on seven external datasets following training on OWT. In this section, we provide the detailed experimental setup, an analysis of the hyperparameter sensitivity regarding the mixing ratio k , and a breakdown of the training dynamics. Aligning with prior work ( 26 ; 27 ) , our evaluation suite includes the validation splits of AG News ( 37 ) , LAMBADA ( 20 ) , LM1B ( 3 ) , Penn Treebank ( 17 ) , Scientific Papers from ArXiv and PubMed ( 4 ) , and WikiText ( 18 ) . The validation PPL is estimated via the negative Evidence Lower Bound (ELBO) with Monte Carlo sampling. We employ an evaluation batch size of 128 on a single GPU, and all text is pre-packed into a sequence length of 1024. To ensure a fair comparison, we adopt the identical sampling configuration used in ( 26 ; 27 ) . We investigate the impact of the mixing ratio k on model performance. Tab. 7 and Tab. 8 present the validation PPL on OWT and the zero-shot PPL on external benchmarks, respectively. As shown in Tab. 7 , XDLM exhibits performance highly correlated with k . With smaller values (e.g., k=1\text{e-}3 and k=0.1 ), XDLM achieves a PPL of 23.495 and 24.097, respectively, which is comparable to MDLM (23.321) and GIDD (23.136). Conversely, as k increases towards 0.9, the performance degrades to 25.731, approaching the UDLM baseline (25.937). This trend is further supported by the zero-shot benchmark results in Tab. 8 . XDLM ( k=0.1 ) achieves an average PPL of 54.110, performing similarly to MDLM (53.650) and significantly outperforming UDLM (59.574). This demonstrates that XDLM successfully retains the strong modeling capabilities of the masked diffusion paradigm. Notably, the superior performance of XDLM is not merely an endpoint phenomenon but a consistent property observed throughout the training process. As illustrated in Fig. 8 , the PPL trajectory of XDLM closely aligns with that of MDLM, steadily decreasing from 64.85 to 54.11. In contrast, the UDLM curve remains notably higher across all training steps. Tab. 9 details the average zero-shot PPL at 100k step intervals. The data confirms that XDLM ( k=0.1 ) maintains a learning dynamic nearly identical to MDLM from the early stages (100k steps) to convergence (1M steps), highlighting the inherent stability and efficiency of the proposed method. Table 7 : Validation PPL on OWT after 1M training steps. Performance is highly correlated with the mixing ratio k : lower values ( k\leq 0.1 ) maintain parity with the strong MDLM baseline (23.321), while higher values ( k\to 0.9 ) degrade towards the UDLM baseline (25.937). Model MDLM GIDD XDLM(k=1e-3) XDLM(k=0.1) XDLM(k=0.5) XDLM(k=0.9) UDLM PPL 23.321 23.136 23.495 24.097 24.818 25.731 25.937 Table 8 : Zero-shot PPL evaluation on seven external benchmarks. XDLM ( k=0.1 ) demonstrates robust generalization, achieving an average PPL (54.110) comparable to pure mask-based methods (MDLM/GIDD) and significantly outperforming UDLM (59.574). Dataset AG News LAMBADA LM1B-GPT2 PTB ArXiv PubMed WikiText Average MDLM 61.374 47.967 65.629 89.049 37.457 41.981 32.093 53.650 GIDD 60.607 47.811 65.898 86.911 39.019 42.634 30.809 53.384 XDLM(k=1e-3) 62.247 45.982 65.944 89.575 38.294 42.206 31.809 53.722 XDLM(k=0.1) 62.768 45.608 68.229 90.796 37.232 41.391 32.748 54.110 XDLM(k=0.5) 67.352 47.966 71.252 91.683 38.460 43.373 33.570 56.236 XDLM(k=0.9) 71.791 49.094 74.116 97.266 39.658 44.701 35.175 58.829 UDLM 69.402 51.272 75.572 95.986 42.671 47.181 34.933 59.574 Figure 8 : Average Zero-Shot PPL trajectory throughout training. Consistent with the final results, XDLM ( k=0.1 ) closely aligns with the MDLM baseline from early stages (64.85) to convergence (54.11), while the UDLM curve remains notably higher across all training steps. Table 9 : Detailed evolution of Zero-Shot PPL throughout training. Evaluated at 100k-step intervals, the data confirms that XDLM ( k=0.1 ) maintains a learning dynamic nearly identical to MDLM, highlighting the method’s stability and efficiency from early training to convergence. Train Steps 100k 200k 300k 400k 500k 600k 700k 800k 900k 1M MDLM 64.209 59.315 57.411 56.129 55.434 54.943 54.466 54.201 53.839 53.650 GIDD 61.097 57.900 56.326 55.389 54.848 54.294 53.960 53.807 53.258 53.384 XDLM(k=1e-3) 63.748 59.097 57.319 56.374 55.576 54.937 54.577 54.336 53.977 53.722 XDLM(k=0.1) 64.847 60.187 58.253 57.201 56.288 55.797 55.271 54.872 54.500 54.110 XDLM(k=0.5) 68.074 62.524 60.489 59.353 58.396 57.881 57.437 56.976 56.530 56.236 XDLM(k=0.9) 70.264 65.384 63.147 61.758 60.977 60.263 59.723 59.438 58.915 58.829 UDLM 71.220 65.789 63.692 62.320 61.258 60.937 60.577 60.192 59.711 59.574 Appendix H Detailed Language Generation Results In this section, we present the comprehensive numerical results for the language generation experiments initially discussed in Sec. 4.2 . To systematically analyze the trade-off between the absorbing (MDLM-like) and uniform (UDLM-like) diffusion processes, we evaluate XDLM across varying mixing ratios k\in\{10^{-3},0.1,0.5,0.9\} . For generation, we adopt the ancestral sampling setting ( 26 ; 28 ; 27 ) , employing parallel token sampling from the estimated posterior via Gumbel noise. The resulting performance is evaluated based on sample quality, quantified by GPT-2 Large perplexity (PPL \downarrow ), and diversity, measured by Token Entropy ( \uparrow ). Tab. 10 presents the comprehensive evaluation on the OWT dataset. The data quantitatively confirms that k governs the model’s performance across different numbers of sampling steps, effectively interpolating between the behaviors of UDLM and MDLM. In the few-step regime (e.g., 8 to 32 sampling steps), increasing k significantly improves performance, aligning XDLM with the efficiency of UDLM. For instance, with only 8 sampling steps, XDLM ( k=0.9 ) yields a low perplexity of 189.750, which is highly competitive with UDLM (183.991) and vastly superior to the mask-based baselines, MDLM (711.382) and GIDD (654.841). Conversely, in the multi-step regime (e.g., 512 to 1024 sampling steps), retaining mask-like properties (lower k ) remains advantageous for achieving high-fidelity generation. XDLM ( k=0.1 ) demonstrates an optimal balance. It dramatically improves upon MDLM when sampling budget are limited, while retaining the capacity to reach very low perplexity scores (52.609 at 1024 steps) when more sampling steps are available, all without sacrificing the diversity indicated by entropy. These findings are corroborated by the results on the LM1B dataset, detailed in Tab. 11 . We observe a consistent trend where a higher k dominates when the number of sampling steps is small. At 4 sampling steps, XDLM ( k=0.5 ) achieves a perplexity of 232.702, effectively matching UDLM (232.403) and significantly outperforming MDLM (377.177). As the number of sampling steps increases, the performance gaps narrow, yet XDLM remains robust across the spectrum. Collectively, these results demonstrate that by tuning k , XDLM bridges the gap between the two paradigms: it captures the efficiency of uniform noise at few sampling steps while preserving the high-quality potential of masking at larger steps. Table 10 : Quality (PPL) vs. Diversity (Entropy) on OWT. The results demonstrate that the optimal mixing ratio k depends on the sampling budget. In the few-step regime (e.g., 8 steps), a higher k is essential for efficiency, allowing XDLM to match UDLM performance. Conversely, in the multi-step regime, a lower k becomes advantageous for achieving high fidelity. Notably, XDLM ( k=0.1 ) strikes an optimal balance, dramatically outperforming MDLM at low step counts while reaching superior perplexity at high step counts. Model step 8 step 16 step 32 step 64 step 128 step 256 step 512 step 1024 Perplexity \downarrow MDLM 711.382 288.157 156.576 106.098 80.198 64.376 51.756 41.545 GIDD 654.841 273.829 151.331 102.558 77.548 62.077 49.822 39.981 XDLM(k=1e-3) 613.647 253.897 146.567 104.090 80.301 65.245 52.650 42.797 XDLM(k=0.1) 301.640 161.724 114.795 94.837 83.205 73.824 63.888 52.609 XDLM(k=0.5) 218.572 127.335 98.817 86.956 81.051 77.155 73.503 66.136 XDLM(k=0.9) 189.750 117.234 93.239 84.068 80.614 79.456 78.707 78.238 UDLM 183.991 111.237 88.632 79.509 76.611 74.019 73.345 106.666 Entropy \uparrow MDLM 5.872 5.767 5.687 5.614 5.543 5.472 5.387 5.287 GIDD 5.832 5.748 5.675 5.605 5.534 5.462 5.370 5.269 XDLM(k=1e-3) 5.837 5.746 5.673 5.613 5.546 5.475 5.393 5.298 XDLM(k=0.1) 5.662 5.630 5.600 5.573 5.548 5.516 5.467 5.390 XDLM(k=0.5) 5.574 5.565 5.555 5.547 5.538 5.525 5.514 5.476 XDLM(k=0.9) 5.546 5.547 5.548 5.548 5.548 5.547 5.544 5.542 UDLM 5.536 5.539 5.543 5.535 5.536 5.531 5.524 5.769 Table 11 : Quality (PPL) vs. Diversity (Entropy) on LM1B. Corroborating the OWT findings, a higher k proves dominant when the sampling budget is strictly limited. At 4 steps, XDLM ( k=0.5 ) effectively matches UDLM. Across the full spectrum of sampling steps, XDLM bridges the gap between paradigms, capturing the efficiency of uniform noise for small budgets while preserving the high-quality potential of masking for larger budgets. Model step 4 step 8 step 16 step 32 step 64 step 128 Perplexity \downarrow MDLM 377.177 215.579 152.939 124.731 112.049 102.142 GIDD 392.002 215.873 151.824 124.014 110.247 101.211 XDLM(k=1e-3) 343.246 199.457 146.612 124.444 111.960 103.544 XDLM(k=0.1) 246.134 155.487 123.129 113.006 105.509 101.983 XDLM(k=0.5) 232.702 141.121 116.063 106.182 103.419 100.754 UDLM 232.403 136.914 110.575 102.072 98.571 96.385 Entropy \uparrow MDLM 4.405 4.379 4.366 4.357 4.353 4.347 GIDD 4.401 4.373 4.361 4.359 4.354 4.348 XDLM(k=1e-3) 4.382 4.363 4.357 4.358 4.354 4.351 XDLM(k=0.1) 4.326 4.329 4.334 4.342 4.345 4.346 XDLM(k=0.5) 4.309 4.315 4.325 4.335 4.340 4.342 UDLM 4.303 4.305 4.320 4.327 4.334 4.336 Appendix I Detailed Image Generation Results We present an extended analysis of XDLM’s image generation capabilities on ImageNet-1K and CIFAR-10, specifically examining the impact of the mixing ratio parameter k . As defined in our methodology, k governs the balance between the mask-based and uniform-noise processes. A higher k increases the proportion of uniform noise (effectively approaching the UDLM setting), while a lower k retains more mask-like properties. The results in Tab. 12 , 13 , 14 , 15 empirically validate the utility of this interpolation. We observe two key behaviors: first, a consistent convergence of XDLM towards UDLM behavior as k increases; and second, the fact that the optimal performance often lies within the interpolated regime ( 00.0 , the algorithm calculates a ‘refine_priority‘ by identifying tokens that are currently unmasked (‘ is_masked‘) but for which the model predicts a conflicting token ID (‘pred_tokens != token_ids‘) with a confidence score exceeding the current token’s confidence. These tokens are flagged in ‘update_mask‘ alongside the standard empty masks. This mechanism fundamentally transforms the generation from a purely additive process into a dynamic error-correction loop, allowing the model to stochastically “change its mind” and revise previous outputs during the denoising trajectory. The empirical results, summarized in Tab. 16 , demonstrate the superiority of this formulation across reasoning (GSM8K, MATH, BBH) and code generation (HumanEval, MBPP) benchmarks, particularly in low-compute regimes. At 32 sampling steps, LLaDA-XDLM significantly outperforms all baselines. The most distinct advantage is observed in the MBPP code generation task, where LLaDA-XDLM achieves a score of 15.00, effectively doubling the performance of the base LLaDA (6.80) and the continued MDLM control (4.40). A granular analysis of the MBPP failure modes reveals that this improvement is driven by a fundamental enhancement in generative fidelity; LLaDA-XDLM drastically reduces the incidence of “Failed” cases, defined as non-compilable or syntactically invalid code, from 429 (LLaDA) to 304. By successfully converting these previously unproductive attempts into “Passed” outputs (75 vs. 34), the model confirms that the XDLM objective, combined with the refinement sampling strategy, provides the necessary structural coherence to self-correct syntactic errors that would otherwise lead to compilation failures. As sampling steps increase to 64, 128, and 256 (the total generation length), LLaDA-XDLM maintains a competitive edge, validating that the performance gains stem directly from the internal representation learned during XDLM training rather than being an artifact of the sampling algorithm alone. Notably, we observe that the continual pretraining baseline (LLaDA-MDLM) exhibits a performance decline compared to the original LLaDA, likely due to the distribution shift between the tuning data and the original pretraining corpus. Consequently, the superior performance of LLaDA-XDLM over LLaDA-MDLM confirms that our gains stem from the effectiveness of the XDLM formulation rather than merely the additional 600 training steps. Furthermore, LLaDA-XDLM significantly outperforms the ablation baseline LLaDA-XDLM-infer, demonstrating that the improvement is intrinsic to the learned model rather than being an artifact of sampling tricks. Algorithm 1 Generation Strategy of LLaDA-XDLM ⬇ def generate ( steps , seq_length , mask_id , topk_absorb , topk_uniform , k =0.1): "" " Args: steps (int): Total number of sampling steps. seq_length (int): Target sequence length to generate. mask_id (int): Vocabulary index of the [MASK] token. topk_absorb (list[int]): Schedule for the number of tokens to decode at each step. topk_uniform (list[int]): Schedule for the number of tokens to refine at each step. k (float): The mixing ratio. Returns: token_ids (torch.Tensor): The generated sequence of token IDs with shape (seq_length,). " "" token_ids = torch . full (( seq_length ,), fill_value = mask_id , dtype = torch . long ) for i in range ( steps ): is_masked = ( token_ids == mask_id ) logits = model ( token_ids ). logits logits_with_noise = add_gumbel_noise ( logits ) pred_tokens = torch . argmax ( logits_with_noise , dim =-1) probs = torch . softmax ( logits , dim =-1) pred_confidence = torch . squeeze ( torch . gather ( probs , dim =-1, index = torch . unsqueeze ( pred_tokens , -1)), -1) mask_priority = torch . where ( is_masked , pred_confidence , - np . inf ) if k > 0.0: current_confidence = torch.squeeze(torch.gather(probs, dim=-1, index=torch.unsqueeze(token_ids, -1)), -1) can_be_refined = (~is_masked) & (pred_tokens != token_ids) & (pred_confidence >= current_confidence) refine_priority = torch.where(can_be_refined, pred_confidence, -np.inf) update_mask = torch . zeros_like ( pred_tokens , dtype = torch . bool ) _ , top_mask_indices = torch . topk ( mask_priority , k = topk_absorb [ i ]) update_mask [ top_mask_indices ] = True if k > 0.0: _, top_refine_indices = torch.topk(refine_priority, k=topk_uniform[i]) update_mask[top_refine_indices] = True token_ids [ update_mask ] = pred_tokens [ update_mask ] return token_ids Table 16 : Evaluation of LLaDA-XDLM on reasoning and code generation benchmarks. We compare the proposed method with the base LLaDA, an inference-only ablation (LLaDA-XDLM-infer), and an MDLM control (LLaDA-MDLM) across varying sampling steps. MBPP scores are decomposed into failure modes to illustrate structural correctness. Dataset BBH GSM8K MATH HumanEval MBPP Score Failed Wrong Pass 32 steps / 256 tokens LLaDA 42.68 24.49 4.80 5.49 6.80 429 37 34 LLaDA-XDLM-infer 41.74 24.26 4.86 1.22 5.40 438 35 27 LLaDA-MDLM 43.21 24.41 2.70 6.71 4.40 447 31 22 LLaDA-XDLM 42.76 29.26 4.72 10.98 15.00 304 121 75 64 steps / 256 tokens LLaDA 47.16 57.16 16.10 12.80 17.80 328 83 89 LLaDA-XDLM-infer 46.76 53.75 16.12 4.88 17.60 328 84 88 LLaDA-MDLM 48.32 54.97 13.42 14.63 13.40 348 84 67 LLaDA-XDLM 45.61 57.85 16.48 15.85 23.60 197 182 118 128 steps / 256 tokens LLaDA 47.49 67.93 26.12 24.39 27.60 161 199 138 LLaDA-XDLM-infer 47.89 66.41 26.70 8.54 27.40 187 174 137 LLaDA-MDLM 48.95 68.61 24.34 25.00 30.00 169 178 150 LLaDA-XDLM 46.41 68.01 25.06 25.00 30.80 130 215 154 256 steps / 256 tokens LLaDA 47.77 72.25 29.98 33.54 40.40 48 248 202 LLaDA-XDLM-infer 47.93 71.34 29.34 6.71 37.00 74 238 185 LLaDA-MDLM 49.84 72.93 29.70 32.32 37.60 51 260 188 LLaDA-XDLM 46.07 73.16 29.22 31.71 34.60 90 236 173 Appendix K Detailed Computational Efficiency Analysis We evaluate the computational efficiency of the proposed XDLM against baseline models by measuring speed and memory consumption across three distinct scenarios: the forward-only case (representing NELBO calculation for perplexity estimation), the forward-backward case (used in training), and the sampling case (used for generation). For the forward and forward-backward evaluations, we utilize randomly generated sequences with a batch size of 32 and a sequence length of 1024 to simulate real data processing. To ensure robust measurements, models are warmed up for 10 iterations, and we report the mean value calculated over the subsequent 100 runs. For the sampling scenario, we employ standard ancestral sampling to generate sequences of length 1024 over 32 steps. The evaluation batch size is set to 32. Models are warmed up for 1 iteration, and we report the mean value calculated over the subsequent 10 runs. As detailed in Tab. 17 , MDLM exhibits the highest throughput and lowest memory cost across all settings, a result expected due to the simplicity of its absorbing noise kernel. However, among models incorporating uniform noise distributions, XDLM achieves the highest throughput across the forward , forward-backward , and sampling regimes. This performance is directly attributed to our scalar reformulated efficient sampling and training strategy, which circumvents expensive matrix operations. In contrast, while UDLM maintains the third-highest throughput in forward and training modes, its sampling throughput is the lowest among the comparison group. Conversely, GIDD achieves the third-highest sampling throughput but performs poorly in NELBO calculation, reaching only half the forward throughput of XDLM. This trend is mirrored in memory consumption. While MDLM is the most lightweight overall, XDLM ranks first among methods whose noise kernel includes a uniform noise component, effectively avoiding the high memory overhead observed in GIDD and UDLM. Table 17 : Computational Efficiency Comparison. We report throughput (tokens/s, \uparrow ) and peak memory usage (GB, \downarrow ) for MDLM, GIDD, XDLM, and UDLM. The metrics cover three scenarios: forward (inference/perplexity estimation), forward-backward (training), and sample (generation). Leveraging the scalar reformulated strategy, XDLM achieves highly competitive throughput and memory efficiency, outperforming other baselines incorporating uniform noise kernels (GIDD, UDLM) in most metrics. Model Throughput (token/s) \uparrow Memory (GB) \downarrow forward forward-backward sample forward forward-backward sample MDLM 424294 141990 8789 6.285 35.354 18.848 GIDD 199516 95395 6336 25.131 51.060 40.856 XDLM (k=0.1) 396398 137372 7108 18.850 41.634 31.414 UDLM 370952 142276 2882 18.850 44.777 59.683 Appendix L Detailed Training Dynamics In this section, we provide the comprehensive numerical data corresponding to the training dynamics analysis discussed in Sec. 4.3 and visualized in Fig. 4 . These tables supplement the visual analysis by providing exact metric values for Perplexity, Entropy, FID, and IS across the training trajectory. Tab. 18 details the training dynamics on the LM1B benchmark over the course of 1 million training steps. The models were evaluated using a fixed budget of 128 sampling steps. As indicated in the main text, while MDLM exhibits superior performance in the initial phase, UDLM demonstrates strong scaling in later training stages, achieving the lowest perplexity among baselines by 1M steps (96.385). Furthermore, the table details the impact of varying the mixing ratio k within our proposed XDLM method. Complementing the text generation results, Tab. 19 presents the quantitative training dynamics for image generation on ImageNet-1K. Evaluations were conducted over 500k training steps using a fixed budget of 16 sampling steps. The numerical data corroborates the visual trends: XDLM establishes and maintains a decisive performance advantage. Notably, XDLM with k=0.5 achieves the lowest reported FID of 23.417 and the highest IS of 46.620 at the final checkpoint, consistently outperforming the MDLM and GIDD baselines. While UDLM starts with a higher FID, it converges rapidly to a competitive score (26.242), yet XDLM remains the leading model throughout the majority of the training phases. Table 18: Quantitative training dynamics on the LM1B benchmark, supplementing the visual analysis in Fig. 4 . We report Perplexity ( \downarrow ) and Entropy ( \uparrow ) over 1M training steps, utilizing a fixed budget of 128 sampling steps. The table details the impact of varying the mixing ratio k in XDLM compared to MDLM, GIDD, and UDLM baselines. Train Steps 100k 200k 300k 400k 500k 600k 700k 800k 900k 1M Perplexity \downarrow MDLM 113.218 107.864 106.735 105.839 104.594 104.106 103.559 103.174 102.769 102.142 GIDD 107.366 103.473 103.386 101.854 102.367 101.745 101.904 101.957 101.490 101.211 XDLM(k=1e-3) 112.628 108.381 106.406 104.913 105.156 103.817 103.624 102.731 103.638 103.544 XDLM(k=0.1) 119.632 110.803 109.017 106.265 105.428 104.572 103.645 103.348 102.223 101.983 XDLM(k=0.5) 117.846 111.024 108.130 105.955 104.953 103.048 102.464 101.040 102.015 100.754 UDLM 116.021 107.800 104.508 102.466 101.480 99.958 99.660 98.147 97.503 96.385 Entropy \uparrow MDLM 4.350 4.348 4.350 4.347 4.349 4.349 4.348 4.349 4.349 4.347 GIDD 4.352 4.350 4.351 4.349 4.351 4.349 4.350 4.350 4.351 4.348 XDLM(k=1e-3) 4.351 4.347 4.349 4.347 4.350 4.347 4.347 4.349 4.351 4.351 XDLM(k=0.1) 4.353 4.347 4.347 4.344 4.344 4.344 4.346 4.344 4.344 4.346 XDLM(k=0.5) 4.350 4.346 4.345 4.344 4.342 4.343 4.342 4.343 4.344 4.342 UDLM 4.349 4.343 4.341 4.340 4.338 4.336 4.338 4.336 4.335 4.336 Table 19 : Quantitative training dynamics on ImageNet-1K generation , corresponding to the curves in Fig. 4 . We report FID ( \downarrow ) and IS ( \uparrow ) over 500k training steps, utilizing a fixed budget of 16 sampling steps. The numerical data corroborates the visual trends: XDLM establishes and maintains a decisive performance advantage. Train Steps FID \downarrow IS \uparrow 100k 200k 300k 400k 500k 100k 200k 300k 400k 500k MDLM 40.060 33.723 31.090 29.436 28.785 28.978 36.136 39.804 42.893 44.656 GIDD 42.152 36.878 35.200 35.677 35.403 27.337 33.042 35.544 35.173 35.698 XDLM(k=1e-3) 45.381 38.967 36.061 36.287 35.046 24.967 30.512 33.948 34.031 35.097 XDLM(k=0.1) 38.358 31.464 27.524 26.853 25.774 27.949 35.839 41.150 42.222 43.903 XDLM(k=0.5) 37.567 30.098 26.687 23.790 23.417 27.854 35.682 40.724 45.976 46.620 UDLM 43.316 34.525 30.381 28.086 26.242 23.619 31.188 35.963 38.588 41.801