The Diffusion Duality, Chapter II: Ψ \Psi -Samplers and Efficient Curriculum Justin Deschenaux, Caglar Gulcehre, Subham Sekhar Sahoo Abstract Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2 Figure 1: Performance on Language Modeling and Image Modeling. \Psi -samplers generalize ReMDM ( 75 ) to arbitrary noise distributions. (Left): Generative perplexity (Gen. PPL; \downarrow ) as a function of NFEs, with nucleus sampling p=0.9 . \Psi -samplers consistently improve with more steps, unlike ancestral sampling which plateaus. Curves are annotated with the average unigram entropy per sequence as a proxy for diversity. (Right): On CIFAR-10, \Psi -samplers achieve better FID ( \downarrow ) than MDLM (with ReMDM). 1 Introduction Diffusion models are powerful generative algorithms that have achieved remarkable success in modeling continuous data domains, including images ( 25 ; 58 ) , audio ( 35 ; 41 ; 31 ) , and videos ( 27 ; 17 ; 5 ; 55 ) . Recent advances have extended diffusion models to categorical data, demonstrating their potential for language modeling ( 1 ; 44 ; 60 ; 67 ; 51 ; 61 ; 63 ) , graphs ( 40 ) , molecules ( 38 ) , and audio ( 36 ) . Unlike autoregressive models that generate tokens sequentially from left to right, diffusion language models can decode tokens in parallel and in any order while leveraging bidirectional contextual information. This capability enables the design of language models that can be significantly faster than their autoregressive counterparts while maintaining strong downstream performance ( 71 ; 37 ) . Discrete diffusion models primarily employ one of two noise distributions: a uniform prior or a masked prior that concentrates all probability mass on a special [ mask ] token. Unlike Masked Diffusion Models (MDMs), which update each token exactly once, Uniform-State Diffusion Models (USDMs) allow tokens to be revised multiple times during generation, enabling self-correction. This makes USDMs particularly effective for few-step ( 61 ) and guided generation ( 65 ) . However, the generation quality of USDMs has not yet matched that of MDMs in high-sampling-step regimes, and USDMs’ modeling capacity, as measured by likelihood, remains inferior to MDMs’. Although 61 proposed a curriculum learning strategy ( 2 ) that narrows the likelihood gap, this curriculum is computationally expensive. To address MDMs’ inability to remask tokens, ReMDM ( 75 ) introduced “Predictor-Corrector” (PC) samplers that generalize and outperform earlier PC methods ( 7 ; 20 ) . These samplers substantially improve the inference time scaling behavior of MDMs. However, PC methods for uniform-state diffusion remain underexplored. 7 proposed PC methods for samplers that take advantage of the rate change matrices of the continuous-time Markov chain (CTMC) formulation of discrete diffusion processes, but such samplers are known to perform worse than ancestral samplers ( 44 ; 65 ) . We propose \text{Duo}^{++} to address these challenges, which expands the design space of USDMs using non-Markovian superposition posteriors (or as we refer to them in this paper, \Psi -posteriors). These posteriors align with the intermediate marginals of discrete diffusion processes and give rise to \Psi -samplers with predictor-corrector capabilities that are crucial for improving sample quality. In addition, \text{Duo}^{++} introduces an efficient curriculum learning strategy that advances the approach of 61 by accelerating training and reducing memory usage. In summary, our contributions are threefold: (1) we propose a family of non-Markovian posteriors ( \Psi -posteriors) for discrete diffusion with arbitrary noise priors that share the same marginals as the Markovian discrete diffusion process (Sec. 3 ). (2) We demonstrate that the induced \Psi -samplers improve text and image generation quality and scale better than standard ancestral samplers in high NFE regimes, closing the performance gap with respect to MDMs coupled with remasking samplers in high NFE regimes for text generation (Sec. 5.1 ) and surpassing them on image generation tasks (Sec. 5.1.2 ). (3) We reformulate the curriculum learning strategy proposed in 61 , achieving a 2 \times speedup while reducing peak memory usage by 33% and end-to-end training time by 25%, while maintaining similar perplexity (Fig. 1 , right, Table 5 ) and downstream task accuracy (Table 1 ). 2 Background Notation Let \mathcal{V}:=\{\mathbf{v}\in\{0,1\}^{K}:\sum_{i=1}^{K}\mathbf{v}_{i}=1\} denote the set of one-hot encodings of discrete random variables over K categories. Let {\mathbf{x}}\in\mathcal{V}^{L} denote a sequence of L discrete variables in \mathcal{V} and {\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}\ell} denote the entry \ell^{\text{th}} in {\mathbf{x}} . We use boldface to denote both individual vectors and sequences; the context will make clear whether a symbol refers to a vector or a sequence. Let \Delta denote the K simplex. For \mathbf{v}\in\Delta , let \text{Cat}(\cdot;\mathbf{v}) denote a categorical distribution such that \mathbb{P}(\mathbf{u}_{i}=1)=\mathbf{v}_{i} , for \mathbf{u}\sim\text{Cat}(\cdot;\mathbf{v}),\mathbf{u}\in\mathcal{V} . Let \langle\mathbf{a},\mathbf{b}\rangle and \mathbf{a}\odot\mathbf{b} denote the dot and Hadamard products between two vectors respectively. Let {\bm{1}}=\{1\}^{K} denote the all-ones vector. Let \bm{\pi}\in\Delta be a designated categorical distribution referred to as the prior. 2.1 Discrete Diffusion Models Consider the clean data sequence {\mathbf{x}} of length L drawn from the data distribution q_{\text{data}} . Discrete diffusion models ( 68 ; 1 ) define a sequence of increasingly noisy distributions (q_{t})_{t\in[0,1]} , interpolating from q_{\text{data}} to a factorized prior distribution, which is a product of L independent \text{Cat}(.;\bm{\pi}) distributions, using Markovian transitions defined independently across input dimensions ( 7 ; 60 ; 67 ; 51 ; 65 ; 61 ) . Let {\mathbf{z}}_{t}\sim\prod_{\ell=1}^{L}{\color[rgb]{0.043,0.3242,0.5898}q_{t}}(.|{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}) denote the intermediate latents (sequence) at time step t . This work focuses on factorized, interpolating noise processes ( 60 ) , whose conditional marginal distribution takes the form: {\mathbf{z}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}\sim{\color[rgb]{0.043,0.3242,0.5898}q_{t}}(.|{\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}\ell};{\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}})=\text{Cat}(.;{\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}}{\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}+(1-{\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}})\bm{\pi}), (1) where {\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}}\in[0,1] is monotonically decreasing with t , and is known as the noise schedule . ( 1 ) defines the forward process , which progressively corrupts the data. The goal is to learn a generative process {p_{\theta}} , parameterized by a neural network with parameters \theta , that reverses this forward process to map from the noise prior back to q_{\text{data}} . The model is typically trained by minimizing the “Negative Evidence Lower Bound” (NELBO). The choice of token prior \bm{\pi} gives rise to two popular variants: Masked Diffusion Models (MDMs) and Uniform-state Diffusion Models (USDMs), which we discuss in the following. 2.1.1 Masked Diffusion Processes MDMs ( 60 ; 67 ; 51 ) use a masked prior, where \bm{\pi}={\mathbf{m}}\in\mathcal{V} is the one-hot representation of a special [ mask ] token ( 13 ) . During the forward process ( 1 ), tokens either remain unchanged or transition to the masked state {\mathbf{m}} , after which they stay masked. This behavior carries over to the reverse process. The posterior of the reverse process q^{\text{MDM}}_{s|t} for 0\leq s100{,}000 ( 73 ; 50 ) . Our key observation is that 61 use a very low temperature \tau=10^{-3} in ( 2.1.2 ). At such temperatures, the softmax concentrates almost all its mass on only a few entries, making most of the K weights negligible. We exploit this induced sparsity by approximating the full linear combination using only k\ll K embeddings (Fig. 3 , bottom). We illustrate the process hereby: Step 1: Generating top- k entries in {\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}} w/o materializing it Let o\in[K] be the nonzero coordinate of the one-hot vector {\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}} , i.e. ({\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})_{o}=1 . Recall that {\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}={\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}}{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}+{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}\bm{\epsilon} , with {\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}=\sqrt{1-{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}^{{\color[rgb]{0,0,0}2}}}} and \bm{\epsilon}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I}_{K}) . Therefore, ({\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})_{o}\sim{\mathcal{N}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}},{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) and ({\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})_{i\neq o}\sim{\mathcal{N}}(0,{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) . All coordinates in [K]\setminus{o} are exchangeable (i.i.d. with mean 0 ), while the coordinate o is the only one with a shifted mean. As a result, the top- k set falls into one of two cases: Case 1: o is not among the top- k , so all k winners lie in [K]\setminus{o} . Case 2: o is among the top- k , so the winners are {o} plus k-1 indices from [K]\setminus{o} . Next, we describe how to sample the top- k values (and later their indices) without ever forming the full K -dimensional vector. Let m=K{-}1 and consider m i.i.d. samples \tilde{w}_{1},\dots,\tilde{w}_{m}\sim{\mathcal{N}}(0,{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) . Rather than drawing all m values, we exploit the fact that order statistics of i.i.d. uniform random variables can be sampled recursively: the maximum of m i.i.d. \mathcal{U}[0,1] variables has CDF u^{m} , so it can be sampled directly. Conditioned on the maximum, the remaining values are also i.i.d. uniforms, so the next-largest can be sampled the same way, and so on. Applying the inverse normal CDF \Phi^{-1}(\cdot)\cdot{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}} converts the top- k uniform order statistics into the top- k Gaussian values (see Suppl. B.1.4 for details). We denote the result \mathcal{K}=(\mathcal{K}_{1}\geq\dots\geq\mathcal{K}_{k}) , where \mathcal{K}_{j} is the j -th largest among the K{-}1 zero-mean coordinates. We draw independently the “special” entry \tilde{w}\sim{\mathcal{N}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}},{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) , which matches the distribution of ({\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})_{o} . Now compare \tilde{w} to the current k -th largest value among the zero-mean coordinates: If \mathcal{K}_{k}>\tilde{w} , then o cannot enter the top- k . We are in Case 1, and \mathcal{K} already contains the correct top- k values. If \tilde{w}>\mathcal{K}_{k} , then o must be in the top- k . We are in Case 2. Let r=\big|\{j\in[k]:\mathcal{K}_{j}>\tilde{w}\}\big| be the number of values in \mathcal{K} that are larger than \tilde{w} . We insert \tilde{w} at rank r+1 and drop the previous smallest value: \mathcal{K}\leftarrow\mathcal{K}_{1:r}\|\tilde{w}\|\mathcal{K}_{r+1:k-1} , where \| denotes concatenation. For clarity, let \binom{S}{m} denote the set of all possible tuples with m distinct elements in the set S . We generate the index tuple \mathcal{I} corresponding to the top- k values \mathcal{K} as follows: Case 1: all top- k indices lie in [K]\setminus{o} ; hence, \mathcal{I}\sim\binom{[K]\setminus\{o\}}{k}. Case 2: the index o appears at position r+1 in \mathcal{K} . Sample the indices for the remaining k-1 values from [K]\setminus{o} , split into those above and below o : \mathcal{I}=L\|(o)\|R , where L\sim{\binom{[K]\setminus\{o\}}{r}},R\sim\binom{[K]\setminus\{o\}\setminus L}{k-r-1} . This produces both the top- k values and their matching indices while sampling only O(k) random variables, without constructing the full K -dimensional vector. Step 2: Approximating the softmax Given the top- k values and indices (\mathcal{K},\mathcal{I}) from Step 1, we approximate the softmax-weighted embedding vector by retaining only the k selected rows of the \texttt{embeddings}\in\mathbb{R}^{K\times d} ( d denotes the embedding size): \displaystyle{\color[rgb]{0.6055,0,0}\mathrm{softmax}}({\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})^{\top}\texttt{embeddings}\approx\sum_{i=1}^{k}\frac{\exp(\mathcal{K}_{i}/\tau)}{\tilde{Z}}\texttt{embeddings}[{\mathcal{I}_{i}}], (13) where \texttt{embeddings}[j] denotes the j -th row. The normalizer \tilde{Z} includes both sampled (top- k ) and unsampled terms. While each unsampled term is small, their sum may be non-negligible, hence we approximate it as (Suppl. B.2 ): \displaystyle\tilde{Z}\approx \displaystyle\underbrace{\sum_{i=1}^{k}\exp\left(\frac{\mathcal{K}_{i}}{\tau}\right)}_{\text{top-$k$ terms}}+\underbrace{\delta\exp\left(\frac{\tilde{w}}{\tau}\right)}_{\text{clean token}}+\underbrace{(K-k-\delta)\exp\left(\frac{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}}{2\tau^{2}}-\log\Phi\!\left(\frac{\mathcal{K}_{k}}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)+\log\Phi\!\left(\frac{\mathcal{K}_{k}-{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}/\tau}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)\right)}_{\text{unsampled zero-mean terms}}, (14) where \delta=1 if \tilde{w}\in\mathcal{K} ( case 2 ) and 0 otherwise. We provide the full derivation in Suppl. B.2 and pseudocode in Algo. 3 . Lastly, the curriculum objective in ( 9 ) requires evaluating the diffusion transformation operator {\color[rgb]{0.6055,0,0}\mathcal{T}}(\cdot) . Directly computing {\color[rgb]{0.6055,0,0}\mathcal{T}} via ( 6 ) is prohibitively expensive during training; 61 therefore precompute and cache many ({\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}},{\color[rgb]{0.6055,0,0}\mathcal{T}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}})) pairs, which is cumbersome. Instead, we compute {\color[rgb]{0.6055,0,0}\mathcal{T}}(\cdot) on the fly using its Taylor expansion; see Suppl. B.3.1 . Table 1: Accuracy on multiple-choice question answering datasets. Abbreviations: Arc-e (ARC-Easy), Arc-c (ARC-Challenge), HSwag (HellaSwag), WinoG (Winogrande), PIQA (Physical Intelligence Question Answering), OQA (OpenBookQA). † Results from 12 . \text{Duo}^{++} ( k=2 ) achieves slightly higher accuracy than Duo on 4 out of 6 tasks. Overall, \text{Duo}^{++} matches Duo’s performance while using 25% fewer flops . The highest accuracy among USDMs is bolded . The absolute best per column is underlined . Arc-e Arc-c HSwag WinoG PIQA MathQA OQA AR Transformer 44.95 23.04 30.55 52.80 63.71 22.24 19.00 MDLM † 34.26 24.66 31.54 51.93 57.89 20.70 28.60 Duo 28.11 25.43 26.46 47.20 51.14 20.00 23.40 \text{Duo}^{++} ( k=2 ) 27.32 26.11 26.26 49.64 52.12 20.40 27.80 \text{Duo}^{++} ( k=3 ) 28.28 25.00 25.89 47.36 50.65 21.01 23.00 \text{Duo}^{++} ( k=5 ) 28.03 25.77 26.90 50.12 51.25 20.20 25.40 5 Experiments We evaluate \text{Duo}^{++} with \Psi -samplers on language modeling (Sec. 5.1.1 ) and image generation (Sec. 5.1.2 ), showing that \Psi -samplers markedly improve text and image quality, making USDMs outperform MDMs in sample quality. In Sec. 5.2 , we show that \text{Duo}^{++} matches Duo ( 61 ) while using 33% less memory and training 25% faster, enabled by our efficient curriculum strategy (Sec. 4 ). 5.1 \Psi -Samplers 5.1.1 Language Modeling Takeaway 2: \Psi -samplers substantially improve Generative Perplexity for USDMs, with gains especially pronounced when NFEs exceed the sequence length. Takeaway 3: Unlike ancestral sampling, which plateaus, \Psi -samplers continue to improve with more sampling steps , closing the gap with Masked diffusion models. Experimental Settings Figure 4: Illustration of the possible evolution of t and the associated \kappa_{t} . In practice, we use \kappa_{t} close to 1 during the PC phase. We compare MDLM ( 60 ) and ReMDM ( 75 ) with \text{Duo}^{++} and \Psi -samplers. We use the original checkpoints of 60 , trained for 1M steps with a batch size of 512 on OpenWebText (OWT; 21 ) and context length L=1024 . \text{Duo}^{++} is trained with the same context length, batch size and number of steps, but with the efficient curriculum. Refer to the original works for more details. We measure the sample quality using the Gen. PPL ( \downarrow ) computed with GPT-2 Large ( 56 ) and the diversity using the unigram entropy ( \uparrow ) ( 16 ; 60 ; 61 ) . We cast logits to 64-bit precision for sampling ( 78 ) . See Suppl. C.1 for more details. Results and Ablation Fig. 1 (left) shows the Gen. PPL and entropy as a function of the NFE. \text{Duo}^{++} with \Psi -samplers outperforms MDLM with ReMDM and ancestral sampling across the entire range of NFEs. As the number of NFEs increases beyond the sequence length, ReMDM and \Psi -samplers further improve sample quality while ancestral sampling plateaus. We ablate on the \kappa_{t} schedule type (cap, rescale, loop, see Suppl. A.5 ), the step-size parameter \eta , the nucleus sampling threshold p\in\{0.9,0.95,1.0\} (Suppl. D.1 ). The rescale schedule with \eta=0.05 yields the best Gen. PPL while preserving the unigram entropy. Nucleus sampling ( p{=}0.9 ) consistently improves Gen. PPL for both MDLM and Duo, as observed in 75 . How to Pick \kappa_{t} ? We recommend the ReMDM-equivalent rescale schedule with \eta=0.05 and nucleus sampling ( p=0.9 ), using the log-linear noise schedule with linearly decreasing t , which outperforms the “loop” strategy (Suppl. A.5 ). 5.1.2 Image Modeling Takeaway 4: On CIFAR-10, \text{Duo}^{++} with \Psi -samplers achieves better FID and IS than MDLM with both ancestral sampling and ReMDM. Experimental Setup We train the same 35M-parameter U-Net as 1 on CIFAR-10 for 1.5M steps. Following 65 , the U-Net is made class conditional and we sample with Discrete Classifier-free Guidance (CFG; 28 ; 65 ). See Suppl. C.1 for full training details. We report the Fréchet Inception Distance (FID; 24 ) and Inception Score (IS; 64 ) between the training set and generated samples. Results and Ablation Fig. 1 (right) and Fig. 6 show that \Psi -samplers and ReMDM substantially improve FID and IS compared to ancestral sampling, with \text{Duo}^{++} reaching the best scores overall. We ablate on the sampling noise schedule (cosine vs. log-linear), \kappa_{t} , the activation range [t_{\text{off}},t_{\text{on}}] , nucleus sampling, and the \kappa_{t} schedules (including the ReMDM variants; see Suppl. A.5 ). Full results are in Suppl. D.1 . Using \kappa_{t} close to 1 (light noise injection) with a cosine schedule achieves the best FID and IS, and \text{Duo}^{++} tolerates stronger noise injection than MDLM. With ancestral sampling, nucleus sampling improves the FID in the low NFE regime for MDLM, and always helps for Duo. Since it is detrimental to MDLM at high NFE, we do not use nucleus sampling when using the \Psi -samplers, for both MDLM and Duo. How to Pick \kappa_{t} ? We recommend a cosine sampling schedule with \kappa_{t}=0.95 , t_{\text{on}}\in\{0.5,0.6\} , t_{\text{off}}=0.1 for \text{Duo}^{++} , and \kappa_{t}=0.99 , t_{\text{on}}=1.0 , t_{\text{off}}=0.1 for MDLM. We suggest using a piecewise constant \kappa_{t} with linearly decreasing t , rather than the ReMDM loop schedule (Suppl. A.5 ), setting \kappa_{t}<1 when t\in[t_{\text{off}},t_{\text{on}}] , since it outperforms the ReMDM schedules. 5.2 Fast Curriculum Takeaway 5: The efficient curriculum reduces peak memory by 33% and training time by 25% , while matching the performance of Duo on likelihood benchmarks and downstream tasks. Experimental Settings Table 2: Test perplexity (PPL) on LM1B and OWT. Lower is better. † Results from 61 . Best Uniform-state diffusion numbers are bolded . Duo and \text{Duo}^{++} achieve comparable performance across both datasets while requiring 25% fewer GPU-hours (Table 4 ), demonstrating the effectiveness of our memory-efficient curriculum. LM1B OWT Autoregressive Transformer † 22.3 17.5 Masked Diffusion SEDD Absorb † 32.7 24.1 MDLM † 27.0 23.2 Uniform-state Diffusion SEDD Uniform † 40.3 29.7 UDLM † 31.3 27.4 Duo † 29.9 25.2 \text{Duo}^{++} (Ours), k=2 30.0 25.2 \text{Duo}^{++} (Ours), k=3 30.1 25.3 \text{Duo}^{++} (Ours), k=5 30.2 25.4 We train \text{Duo}^{++} with the scalable curriculum (Sec. 4 ) on OpenWebText (OWT; 21 ) and LM1B ( 9 ) . We train all models for 1M steps, using a batch size of 512. For LM1B, we use the bert-base-uncased tokenizer with a context length of 128, padding shorter sequences. This setup follows previous work ( 60 ; 44 ; 23 ) . For OWT, we use the GPT-2 tokenizer ( 56 ) , and reserve the last 100k documents for validation, following 61 ; 60 . We follow 44 and use a modified diffusion transformer (DiT) ( 54 ) with rotary positional encoding ( 72 ) . We evaluate the impact of k=\{2,3,5\} during the efficient curriculum. All models are trained on 16 H100 GPUs with bfloat16 precision. Training uses the loss in ( 9 ), with \tau=0.001 and (\beta,\gamma)=(0.03,0.15) for the first 500K steps ( 61 ) . Likelihood Results Table 2 shows that on both LM1B and OWT, our efficient curriculum \text{Duo}^{++} matches the performance of Duo with its expensive curriculum. The lowest validation perplexity is achieved with k=2 , although k\in\{2,3,5\} performs similarly. We also compare the models trained on OWT in Zero-Shot perplexity, and find that \text{Duo}^{++} achieves a performance comparable to Duo. That is, we evaluate on the validation splits of the Penn Treebank ( 46 ) , WikiText ( 47 ) , LM1B ( 9 ) , LAMBADA ( 52 ) , AG News ( 76 ) and scientific articles from ArXiv and PubMed ( 10 ) . Table 5 shows that \text{Duo}^{++} reaches a zero-shot probability similar to that of Duo while requiring 25% less training GPU-hours . Likelihood-based Downstream Tasks In Table 1 , we compare the multiple-choice question (MCQ) accuracy of Duo, \text{Duo}^{++} , MDLM ( 60 ) , and an autoregressive transformer (1M training steps with a batch size of 512 on OWT, same hyperparameters as MDLM) using the lm-eval-harness suite ( 19 ) ; details in Suppl. C.3 ). We find that \text{Duo}^{++} achieves an accuracy similar to that of Duo, despite requiring 25% less training GPU-hours. However, it trails MDLM on most tasks, consistent with its higher perplexity. Throughput and Peak Memory Usage Table 4 reports the throughput and peak memory usage for Duo and \text{Duo}^{++} . \text{Duo}^{++} reduces the peak memory usage by about 33% and doubles the speed of the Curriculum Learning phase. When applying Curriculum Learning for half of the training steps, \text{Duo}^{++} trains 25% faster than Duo on the 138M-parameter scale. Notably, both peak memory usage and throughput remain stable over the full training run when k\in\{2,3,5\} . 6 Related work and Discussion Compatibility with General Discrete Diffusion Processes This work focuses on discrete diffusion with uniform or masked noise. However, our approach extends to more general discrete diffusion processes ( 66 ; 74 ; 29 ) featuring a combination of masked and uniform prior, since we provide a general predictor–corrector algorithm for discrete diffusion with arbitrary noise. Predictor-Corrector Samplers In the context of Masked diffusion, ReMDM ( 75 ) generalizes previous predictor-corrector methods ( 7 ; 8 ; 20 ) that were based on Continuous Time Markov Chain formulation of discrete diffusion processes. Our approach further generalizes ReMDM to support arbitrary diffusion processes. Unlike 39 ; 77 ; 42 ; 33 , who train an additional corrector module, our method does not introduce additional learned components. Comparison to Other Discrete Diffusion Samplers 53 uses noise-adaptive step sizes; while we use uniform steps, \Psi -samplers support any step-size schedule. 57 develops higher-order samplers; we use only first-order information, though the posterior in ( 11 ) could be approximated with higher-order methods. Thus, \Psi -samplers complement both lines of work. 7 Conclusion We introduced a unified and practical framework for predictor-corrector sampling in discrete diffusion language models through \Psi -posteriors. By linearly superposing the forward and reverse diffusion processes ( 11 ), the \Psi -posteriors preserve the marginals of standard diffusion models. Importantly, the \Psi -posteriors and associated \Psi -samplers subsume prior masked-diffusion PC samplers ( 7 ; 20 ; 75 ) as special cases, and naturally extend to discrete diffusion models with uniform prior. Empirically, \text{Duo}^{++} with \Psi -samplers matches the performance of MDMs on natural language generation and achieves stronger FID and IS scores on CIFAR-10. Moreover, they exhibit superior scaling: performance continues to improve with NFEs, unlike ancestral samplers, which plateau. Finally, we propose a scalable training curriculum ( 61 ) that reduces the peak memory usage by 33% and shortens the training time by 25%. Concurrently, 62 show that Duo surpasses an autoregressive model at the 1.7B scale on the math and reasoning benchmark (GSM8K). Taken together, these results challenge the view that Masked diffusion is categorically the future of diffusion language modeling. 8 Acknowledgements This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). We are grateful to Ricky T. Q. Chen and Zhihan Yang for insightful discussions and suggestions. We acknowledge the SCITAS team at EPFL for providing access to their cluster, and the Swiss National Supercomputing Centre for the Alps platform. We are grateful to Karin Gétaz for her administrative assistance. Appendix A \Psi -Posteriors A.1 Approximate Reverse Marginals We parameterize the (generative) \Psi -reverse marginals to have a similar form as the true posterior ( 11 ) . Therefore, the generative reverse marginals also factorize over the sequence length. Because {\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}1:L} is not available during sampling, there are two terms in ( 11 ) that are intractable. First, we choose to replace the posterior {{q_{s|t}}}(.|{\mathbf{z}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}},{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}) by {{q_{s|t}}}(.|{\mathbf{z}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}},{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}={\mathbf{x}}_{\theta}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}) . Additionally, as we cannot sample from {\color[rgb]{0.043,0.3242,0.5898}q_{s}}(.|{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}) without {\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}} , we replace {\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}\ell} by {\color[rgb]{0.043,0.3242,0.5898}q_{0|t}}(.|{\mathbf{z}}_{t},{\mathbf{x}}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}={\mathbf{x}}_{\theta}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}) , \forall\ell\in[L] . Replacing these two intractable terms yields our generative reverse marginals: \displaystyle\Psi^{\theta}_{s|t}(.|{\mathbf{z}}_{t})=\kappa_{t}{\color[rgb]{0.043,0.3242,0.5898}{q_{s|t}}}(.|{\mathbf{z}}_{t},{\mathbf{x}}=\mathbf{x}_{\theta}({\mathbf{z}}_{t},t))+(1-\kappa_{t})\left[\alpha_{s}{\color[rgb]{0.043,0.3242,0.5898}q_{0|t}}(.|{\mathbf{z}}_{t},{\mathbf{x}}=\mathbf{x}_{\theta}({\mathbf{z}}_{t},t))+(1-\alpha_{s})\pi\right]. (15) Note that for the masked posterior ( 2 ), {\color[rgb]{0.043,0.3242,0.5898}q_{0|t}}(.|{\mathbf{z}}_{t},{\mathbf{x}}={\mathbf{x}}_{\theta}({\mathbf{z}}_{t},t))={\mathbf{x}}_{\theta}({\mathbf{z}}_{t},t) . A.2 Proof that the \Psi -posteriors have the correct marginals Let \Psi_{s|t}(.|{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}},{\mathbf{z}}_{t}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}) denote the \Psi -posteriors defined in ( 11 ). Let s denote s(k)=t(k-1) and t denote t(k) . To prove that the \Psi -posteriors have the correct marginals, we proceed by (downwards) induction, similar to 69 . First, note that \Psi_{s}({\mathbf{z}}_{s}^{\color[rgb]{0.4688,0.4688,0.4688}\ell}|{\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}}) can be written as a marginalization over \tilde{\mathbf{z}}_{t}^{\color[rgb]{0.4688,0.4688,0.4688}\ell} , for st_{\text{off}} . The noise schedule \alpha_{t} is reparametrized to be piecewise linear. First, \alpha_{t} increases linearly from 0 to \alpha_{t_{\text{on}}} when t>t_{\text{on}} . Secondly, when t\in[t_{\text{off}},\,t_{\text{on}}] , it is held constant to \alpha_{t_{\text{on}}} , and finally increases from \alpha_{t_{\text{on}}} to 1 when t0 and j appears in S[0:t] then S[t]\leftarrow N-k+t {Use largest remaining value} else S[t]\leftarrow j end if end for return S B.2 Approximating the Weighted Sum of the Embeddings After extracting the top- k values and indices (\mathcal{K},\mathcal{I}) from {\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}} , we approximate the softmax-weighted embedding by retaining only the k selected rows of the \texttt{embeddings}\in\mathbb{R}^{K\times d} ( d denotes the embedding size): \displaystyle{\color[rgb]{0.6055,0,0}\mathrm{softmax}}({\mathbf{w}}_{t}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}})^{\top}\texttt{embeddings}\approx\sum_{i=1}^{k}\frac{\exp(\mathcal{K}_{i}/\tau)}{\tilde{Z}}\texttt{embeddings}[{\mathcal{I}_{i}}], (39) where \texttt{embeddings}[j] denotes the j -th row. The normalizer \tilde{Z} includes both sampled (top- k ) and unsampled terms. To account for this contribution, let \displaystyle\mu=\mathbb{E}[\exp(X/\tau)\mid X<\mathcal{K}_{k}] (40) denote the expectation that a r.v. X with pmf {\mathcal{N}}(0,{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) is less than \mathcal{K}_{k} . We approximate \tilde{Z} via two cases. Let o be the index corresponding to the clean-token category in {\mathbf{x}}^{{\color[rgb]{0.4688,0.4688,0.4688}\ell}} , and let \tilde{w}\sim{\mathcal{N}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}},{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) . Case 1. If o is not among the top- k indices (i.e., o\notin\mathcal{I} , and consequently \tilde{w}\notin\mathcal{K} ), then \tilde{Z} includes: (i) the k terms in \mathcal{K} , (ii) the explicit contribution from index o , and (iii) the remaining K-k-1 unsampled terms, each approximated by \mu . Thus, \displaystyle\tilde{Z}\approx\underbrace{\sum_{i=1}^{k}\exp\left(\frac{\mathcal{K}_{i}}{\tau}\right)}_{\text{top-$k$ terms}}+\underbrace{\exp\left(\frac{\tilde{w}}{\tau}\right)}_{\text{index $o$}}+\underbrace{(K-k-1)\mu}_{\text{unsampled terms}}. (41) Case 2. If o is among the top- k indices (i.e., o\in\mathcal{I} , and hence, \tilde{w}\in\mathcal{K} ), its contribution is already included in the top- k sum, leaving K-k unsampled terms. Hence, \displaystyle\tilde{Z}\approx\underbrace{\sum_{i=1}^{k}\exp\left(\frac{\mathcal{K}_{i}}{\tau}\right)}_{\text{top-$k$ terms}}+\underbrace{(K-k)\mu}_{\text{unsampled terms}}. (42) Next, we derive a closed-form expression for \mu in ( 40 ). \mu=\log\mathbb{E}[\exp(X)\mid X<\mathcal{K}_{k}]=\frac{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}}{2\tau^{2}}-\log\Phi\left(\frac{\mathcal{K}_{k}}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)+\log\Phi\left(\frac{\mathcal{K}_{k}-{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}/\tau}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right). (43) Proof. \displaystyle\mu= \displaystyle\log\mathbb{E}\left[\exp\left(\frac{X}{\tau}\right)\mid X<\mathcal{K}_{k}\right] Applying change of variables \bar{X}=X/\tau ; we get, \displaystyle=\log\mathbb{E}[\exp(\bar{X})\mid\bar{X}<\mathcal{K}_{k}/\tau] \displaystyle=\log\int_{-\infty}^{\mathcal{K}_{k}/\tau}\exp(x)\frac{f_{\bar{X}}(x)}{\mathbb{P}(\bar{X}<\mathcal{K}_{k}/\tau)}dx Substituting \sigma:={\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}/\tau , we get: \displaystyle=\log\left[\frac{1}{\Phi(\mathcal{K}_{k}/{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}})}\int_{-\infty}^{\mathcal{K}_{k}/\tau}\exp(x)\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{x^{2}}{2\sigma^{2}}\right)dx\right] \displaystyle=\log\left[\frac{1}{\Phi(\mathcal{K}_{k}/{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}})}\frac{1}{\sqrt{2\pi\sigma^{2}}}\int_{-\infty}^{\mathcal{K}_{k}/\tau}\exp\left(-\frac{x^{2}}{2\sigma^{2}}+x\right)dx\right] \displaystyle=\log\left[\frac{1}{\Phi(\mathcal{K}_{k}/{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}})}\frac{1}{\sqrt{2\pi\sigma^{2}}}\int_{-\infty}^{\mathcal{K}_{k}/\tau}\exp\left(-\frac{1}{2\sigma^{2}}(x^{2}-2\sigma^{2}x+\sigma^{4}-\sigma^{4})\right)dx\right] \displaystyle=\log\left[\frac{\exp(\sigma^{2}/2)}{\Phi(\mathcal{K}_{k}/{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}})}\frac{1}{\sqrt{2\pi\sigma^{2}}}\int_{-\infty}^{\mathcal{K}_{k}/\tau}\exp\left(-\frac{1}{2\sigma^{2}}(x-\sigma^{2})^{2}\right)dx\right] \displaystyle=\log\left[\frac{\exp(\sigma^{2}/2)}{\Phi(\mathcal{K}_{k}/{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}})}\Phi\left(\frac{\mathcal{K}_{k}/\tau-\sigma^{2}}{\sigma}\right)\right] \displaystyle=\frac{\sigma^{2}}{2}-\log\Phi\left(\frac{\mathcal{K}_{k}}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)+\log\Phi\left(\frac{\mathcal{K}_{k}/\tau-\sigma^{2}}{\sigma}\right) Substituting back \sigma:={\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}/\tau , we get: \displaystyle=\frac{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}}{2\tau^{2}}-\log\Phi\left(\frac{\mathcal{K}_{k}}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)+\log\Phi\left(\frac{\mathcal{K}_{k}-{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}/\tau}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right) (44) This concludes our proof. ∎ Finally, substituting the value of \mu from ( B.2 ) into ( 41 ) and ( 42 ), we obtain: \displaystyle\tilde{Z}\approx \displaystyle\underbrace{\sum_{i=1}^{k}\exp\left(\frac{\mathcal{K}_{i}}{\tau}\right)}_{\text{top-$k$ terms}}+\underbrace{\delta\exp\left(\frac{\tilde{w}}{\tau}\right)}_{\text{clean token}}+\underbrace{(K-k-\delta)\exp\left(\frac{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}}{2\tau^{2}}-\log\Phi\!\left(\frac{\mathcal{K}_{k}}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)+\log\Phi\!\left(\frac{\mathcal{K}_{k}-{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}/\tau}{{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}}}\right)\right)}_{\text{unsampled zero-mean terms}}, (45) B.3 Efficient computation of {\color[rgb]{0.6055,0,0}\mathcal{T}} During Training The curriculum objective in ( 9 ) requires evaluating the diffusion transformation operator {\color[rgb]{0.6055,0,0}\mathcal{T}}(\cdot) . Directly computing {\color[rgb]{0.6055,0,0}\mathcal{T}} via ( 6 ) is prohibitively expensive during training; 61 therefore precompute and cache many ({\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}},{\color[rgb]{0.6055,0,0}\mathcal{T}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}})) pairs, which is cumbersome. Instead, we compute {\color[rgb]{0.6055,0,0}\mathcal{T}}(\cdot) on the fly using its Taylor expansion, detailed below. This derivation relies on two propositions that justify swapping the order of (i) summation and integration (Prop. B.5 ) and (ii) differentiation and integration (Prop. B.6 ). Proposition B.5 (First Corollary of the Dominated Convergence Theorem ( 18 , Theorem 2.25)) . If the sum \sum_{n=0}^{\infty}f_{n}(x) exists for all x and there exists an integrable function g(x) such that \left|\sum_{n=0}^{k}f_{n}(x)\right|\leq g(x) (46) for all k , then \int_{-\infty}^{\infty}\sum_{n=0}^{\infty}f_{n}(x)dx=\sum_{n=0}^{\infty}\int_{-\infty}^{\infty}f_{n}(x)dx. (47) Proposition B.6 (Second Corollary of the Dominated Convergence Theorem ( 18 , Theorem 2.27)) . Let f(x,t) be differentiable in t and suppose there exists a function g(x,t) such that: 1. \left|\frac{\partial f(x,t)}{\partial t}\right|\leq g(x,t_{0}) for all x and t in some neighborhood |t-t_{0}|\leq\delta_{0} 2. \int_{-\infty}^{\infty}g(x,t)dx<\infty for all t Then \frac{d}{dt}\int_{-\infty}^{\infty}f(x,t)dx=\int_{-\infty}^{\infty}\frac{\partial f(x,t)}{\partial t}dx (48) B.3.1 Series Representation of {\color[rgb]{0.6055,0,0}\mathcal{T}} and \partial_{t}{\color[rgb]{0.6055,0,0}\mathcal{T}} Evaluating the series expansion on the fly is faster than precomputing and caching {\color[rgb]{0.6055,0,0}\mathcal{T}} for many ({\color[rgb]{0.043,0.3242,0.5898}\alpha_{t}},{\color[rgb]{0.6055,0,0}\mathcal{T}}({\color[rgb]{0.9023,0.5703,0.2188}\tilde{\alpha}_{t}})) pairs as done by 61 . We can further speed the evaluation up by fitting a low-degree polynomial to the series, which we use in practice (see Suppl. B.3.2 for the approximation error analysis). Below, we derive the series expansion for {\color[rgb]{0.6055,0,0}\mathcal{T}} (Prop. B.7 ) and its time-derivative \partial_{t}{\color[rgb]{0.6055,0,0}\mathcal{T}} (Prop. B.8 ): Proposition B.7 (Series Expansion of the Diffusion Transformation Operator) . The diffusion transformation operator {\color[rgb]{0.6055,0,0}\mathcal{T}} can be expressed as: {\color[rgb]{0.6055,0,0}\mathcal{T}}(\tilde{\alpha}_{t})=\frac{K}{K-1}\left[e^{-\nu_{t}^{2}/2}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}}{n!}M_{n}-\frac{1}{K}\right] (49) \nu_{t}=\frac{\tilde{\alpha}_{t}}{\sqrt{1-\tilde{\alpha}_{t}^{2}}} and M_{n}=\int_{-\infty}^{\infty}z^{n}\phi(z)\Phi^{K-1}(z)dz . Proof. Recall that the standard Gaussian PDF is given by \phi(x)=\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}. (50) For notational convenience, let \nu_{t}=\frac{\tilde{\alpha}_{t}}{\sqrt{1-\tilde{\alpha}_{t}^{2}}} . We can rewrite \phi(x-\nu_{t}) in terms of \phi(x) : \phi(x-\nu_{t})=\frac{1}{\sqrt{2\pi}}e^{-(x-\nu_{t})^{2}/2}=\frac{1}{\sqrt{2\pi}}e^{-(x^{2}-2\nu_{t}x+\nu_{t}^{2})/2}=\phi(x)e^{\nu_{t}x}e^{-\nu_{t}^{2}/2}. (51) Using the definition of the infinite series of e^{x} , we can expand e^{\nu_{t}x} : \phi(x-\nu_{t})=\phi(x)e^{-\nu_{t}^{2}/2}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}x^{n}}{n!}. (52) Substituting this into our original integral: \displaystyle\int_{-\infty}^{\infty}\phi\left(z-\nu_{t}\right)\Phi^{K-1}(z)dz \displaystyle=\int_{-\infty}^{\infty}\phi(z)e^{-\nu_{t}^{2}/2}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}z^{n}}{n!}\Phi^{K-1}(z)dz (53) Since Prop. B.5 is satisfied, as the sum is the Taylor series of the exponential function, we can exchange the order of integration and summation. This leads to our final result: \displaystyle\int_{-\infty}^{\infty}\phi\left(z-\nu_{t}\right)\Phi^{K-1}(z)dz \displaystyle=e^{-\nu_{t}^{2}/2}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}}{n!}\int_{-\infty}^{\infty}z^{n}\phi(z)\Phi^{K-1}(z)dz (54) \displaystyle=e^{-\nu_{t}^{2}/2}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}}{n!}M_{n}. ∎ Advantages At this point, one might ask what is gained by expressing {\color[rgb]{0.6055,0,0}\mathcal{T}} as a series expansion. There are two key advantages. First, since {\color[rgb]{0.6055,0,0}\mathcal{T}} is intractable, 60 resort to precomputing 100k evaluations, which can take up to two hours with the GPT-2 tokenizer. Second, they approximate the time derivative using finite differences. Crucially, observe that M_{n} and I_{n} in Prop. B.7 and B.8 are the only intractable components of the series expansion, and they are independent of the input \tilde{\alpha}_{t} . We find that the terms of the series decay to zero after roughly 150 terms (with slower decay as t\to 1 ). Thus, instead of pre-computing 100k evaluations of {\color[rgb]{0.6055,0,0}\mathcal{T}} , it suffices to cache M_{n} and I_{n} for n<150 . In practice, this takes only a few seconds and can be performed at the start of training. Proposition B.8 (Time-Derivative of the Diffusion Transformation Operator) . The time-derivative of the diffusion transformation operator {\color[rgb]{0.6055,0,0}\mathcal{T}} can be expressed as: \displaystyle\frac{d}{dt}{\color[rgb]{0.6055,0,0}\mathcal{T}}(\tilde{\alpha}_{t}) \displaystyle=\frac{K\cdot e^{-\nu_{t}^{2}/2}}{K-1}\frac{\tilde{\alpha}_{t}^{\prime}}{(1-\tilde{\alpha}_{t}^{2})^{3/2}}\sum_{n=0}^{\infty}\frac{\nu_{t}^{n}}{n!}\left[I_{n}-\nu_{t}M_{n}\right] (55) where \nu_{t} and M_{n} are defined as in Prop. B.7 . Finally, I_{n}=\int_{-\infty}^{\infty}z^{n+1}\phi(z)\Phi^{K-1}(z)dz , and \tilde{\alpha}_{t}^{\prime} denotes the time-derivative of the Gaussian noise schedule \tilde{\alpha}_{t} . Proof. We want to compute \displaystyle\frac{d}{d\nu_{t}}{\color[rgb]{0.6055,0,0}\mathcal{T}}(\tilde{\alpha}_{t})=\frac{K}{K-1}\frac{d}{d\nu_{t}}\int\phi(z-\nu_{t})\Phi^{K-1}(z)dz. (56) To justify passing the derivative under the integral, we verify the conditions of Prop. B.6 . Define f(z,t)=\phi\left(z-\frac{\tilde{\alpha}_{t}}{\sqrt{1-\tilde{\alpha}_{t}^{2}}}\right)\Phi^{K-1}(z)=\phi\left(z-\nu_{t}\right)\Phi^{K-1}(z), (57) which has time derivative \frac{\partial f(z,t)}{\partial t}=\frac{(z-\nu_{t})\phi(z-\nu_{t})}{(1-\tilde{\alpha}_{t}^{2})^{3/2}}\Phi^{K-1}(z). (58) We need to find a suitable dominating function g . Let 1>\delta_{0}>0 and choose t_{0}=\frac{1-\delta_{0}}{2} . When |t-t_{0}|\leq\delta_{0} , we have t\in[t_{0}-\delta_{0},t_{0}+\delta_{0}] . Since t_{0}-\delta_{0}\mathcal{K}_{k} then \triangleright Case 2 : clean token o belongs to the top- k \mu\leftarrow\mathbb{E}[\exp(X/\tau)\mid X<\mathcal{K}_{k-1}] , X\sim{\mathcal{N}}(0,{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) \triangleright Mean contribution unsimulated entry (Suppl. B.2 ) r\leftarrow\big|\{j\in[k]:\mathcal{K}_{j}>\tilde{w}\}\big| \triangleright Rank of \tilde{w} in \mathcal{K} \mathcal{K}\leftarrow(\mathcal{K}_{1:r},\;\tilde{w},\;\mathcal{K}_{r+1:k-1}) \triangleright Insert \tilde{w} , drop the smallest \mathcal{K}_{k} Sample k{-}1 indices \mathcal{J} uniformly w/o replacement from [K]\setminus\{o\} \triangleright Algo. 2 \mathcal{I}\leftarrow(\mathcal{J}_{1:r},\;o,\;\mathcal{J}_{r+1:k-1}) \tilde{Z}\leftarrow\sum_{i=1}^{k}\exp(\mathcal{K}_{i}/\tau)+(K-k)\,\mu \triangleright K{-}k unsimulated entries, each contributing \mu else \triangleright Case 1 : clean token o is not in the top- k \mu\leftarrow\mathbb{E}[\exp(X/\tau)\mid X<\mathcal{K}_{k}] , X\sim{\mathcal{N}}(0,{\color[rgb]{0.9023,0.5703,0.2188}\tilde{\sigma}_{t}^{{\color[rgb]{0,0,0}2}}}) \triangleright Mean contribution unsimulated entry (Suppl. B.2 ) Sample k indices \mathcal{I} uniformly w/o replacement from [K]\setminus\{o\} \triangleright Algo. 2 \tilde{Z}\leftarrow\sum_{i=1}^{k}\exp(\mathcal{K}_{i}/\tau)+(K{-}k{-}1)\,\mu+\exp(\tilde{w}/\tau) \triangleright \tilde{w} counted exactly, K{-}k{-}1 via \mu end if \triangleright Step 3: Normalized softmax weights over the k selected entries \bm{\lambda}_{i}\leftarrow\exp(\mathcal{K}_{i}/\tau)\;/\;\tilde{Z} , i=1,\ldots,k return \bm{\lambda},\;\mathcal{I} Appendix C Experimental Details C.1 \Psi -samplers C.1.1 OpenWebText To evaluate the samplers, we use the pre-trained MDLM ( 60 ) and Duo ( 61 ) checkpoints, as well as their distilled variants (using SDTT ( 11 ) and discrete consistency distillation, respectively, after 5 rounds of 10k steps). We re-state the training hyperparameters of both models in Suppl. C.2.1 . For ReMDM, we use both the official implementation of 75 and our re-implementation, which matches the original results while supporting additional sampling schedules beyond the log-linear one. See Suppl. D.1 for details on selecting \kappa_{t} . C.1.2 CIFAR10 (D3PM-like architecture) Table 3: Model architecture on CIFAR10 Component Value Vocab size 256 Number of ResNet blocks per scale 2 Base channels 128 Channel multiplier per scale (1,2,2,2) Attention resolutions 16 Conditional embedding dimension 128 Number of parameters 35.8M We train a U-Net backbone ( 59 ) for 1.5M steps with a batch size of 128, using class conditioning with a class-dropout rate of 0.1 (as in 65 ), and the default hyperparameters of 1 (Table 3 ). For both MDLM and Duo, we experiment with time-conditional and unconditional variants, and train models using either cosine or log-linear noise schedules. See Table 6 for the ancestral-sampling evaluation of all variants after pre-training. See Suppl. D.1 for details on selecting \kappa_{t} . C.2 Improved Curriculum C.2.1 Language modeling We adopt the same setup as prior work on discrete diffusion ( 44 ; 60 ; 61 ) , and restate it for completeness. LM1B We detokenize the One Billion Words ( 9 ) as in 44 ; 60 1 1 1 https://github.com/louaaron/Score-Entropy-Discrete-Diffusion/blob/main/data.py , and tokenize it using the bert-base-uncased tokenizer ( 13 ) , as 23 . We use a context length of 128 and pad shorter documents. OpenWebText We tokenize OpenWebText ( 21 ) with the GPT-2 tokenizer, concatenate sequences to a length of 1024, and insert an eos token between documents. Since the dataset lacks an official validation split, we reserve the last 100k documents for validation. Backbone We parameterize all models using the modified diffusion transformer architecture of 54 , following 44 ; 60 . Our models use 12 layers, a hidden dimension of 768, 12 attention heads, and a timestep embedding of size 128 for the uniform-state diffusion variants. Word embeddings are not tied between input and output. Curriculum Lookup For the Duo baseline, we train models using the original code. To implement the efficient curriculum, we replace the full linear combination of embeddings by a sparse lookup, implemented using torch.nn.functional.embedding_bag to avoid materializing intermediate tensors. The curriculum phase lasts for the first 500k steps, after which we perform regular embedding table lookups, just like 61 . Optimization We train all models with the AdamW optimizer ( 43 ) using a batch size of 512. The learning rate is linearly warmed up from 0 to 3\times 10^{-4} over 2,500 steps, then kept constant for the remainder of training. We apply a dropout rate of 0.1 throughout. C.3 Downstream Evaluation Protocol We evaluate downstream performance using the lm-eval-harness library ( 19 ) , following the protocol of 12 . We focus on multiple choice tasks, where the log-likelihood of each candidate answer, given a prompt, is computed and the answer with the highest score is selected. For diffusion language models, which optimize a variational bound on the log-likelihood of the full sequence, we adapt the evaluation by using Bayes’ rule: \log p({\mathbf{y}}_{i}|{\mathbf{x}})=\log p({\mathbf{x}},{\mathbf{y}}_{i})-\log p({\mathbf{x}})\propto\log p({\mathbf{x}},{\mathbf{y}}_{i}), (64) Since \log p({\mathbf{x}}) does not depend on the candidate {\mathbf{y}}_{i} , we simply select the answer that maximizes \log p({\mathbf{x}},{\mathbf{y}}_{i}) . In practice, we use the log-likelihood ELBO ( 4 ), estimated via Monte Carlo with 1024 samples, and choose the continuation {\mathbf{y}}_{i} with the highest estimated likelihood. C.4 Zero-Shot Likelihood Our setting is the same as used by 61 . Specifically, we measure the likelihood of the models trained on OpenWebText using the validation splits of seven diverse datasets: Penn Tree Bank (PTB; 46 ), Wikitext ( 47 ) , One Billion Words (LM1B; 9 ), Lambada ( 52 ) , AG News ( 76 ) , and Scientific Papers (Pubmed and Arxiv subsets; 10 ). The datasets are detokenized following the protocol of 44 ; 61 . We wrap all sequences to a maximum length of 1024 tokens and do not insert eos tokens between them. Table 5 shows that we reach similar performance as Duo. Appendix D Additional Experimental results In Suppl. D.1 , we elaborate on the impact of \kappa_{t} on the performance of the \Psi -samplers. In Suppl. D.2 , we show that our efficient curriculum produces weights with the same marginal distributions as 61 . D.1 Tuning \kappa_{t} for the \Psi -samplers Figure 6: \Psi -samplers, which generalize ReMDM, significantly improve the Inception Score on CIFAR-10, compared to ancestral sampling. As discussed in Sec. 5.1 , the choice of \kappa_{t} is critical for strong performance. With a poor choice of \kappa_{t} , \Psi -samplers can underperform ancestral sampling. Below, we report all of our hyperparameter sweeps across datasets. • We perform image modeling on CIFAR-10 using the U-Net architecture of 1 ; 65 , and use horizontal flipping as the sole data augmentation. • We evaluate \Psi -samplers on OpenWebText ( 21 ) using the original checkpoint of MDLM ( 60 ) and Duo ( 61 ) . D.1.1 CIFAR-10 We report FID ( 24 ) , computed between 50k generated samples and the training set. Before evaluating \Psi -samplers, we ablate on the training hyperparameters. Specifically, we train models with cosine and log-linear noise schedule, optionally with time-conditioning. We sample with both cosine and log-linear schedules. Finally, we check whether nucleus sampling ( 30 ) and greedy decoding on the final step can help, compared to vanilly ancestral sampling. Since nucleus sampling helps Duo but not MDLM, we compare the two models without nucleus sampling. Table 6 shows the validation perplexity and FID for a few number of sampling steps. Table 7 reports FID for ancestral sampling using step counts that are powers of two, from 32 up to 4096. Table 8 shows the results with ReMDM. Table 9 reports FID scores for \Psi -samplers using a stepwise-constant \kappa schedule. Table 11 shows the performance of \Psi -samplers using the \kappa schedule equivalent to ReMDM. We obtain similar results, which supports our theoretical claims. • MDLM (Ancestral). Training with cosine noise schedule and time conditioning yields the best validation perplexity and FID. • MDLM (ReMDM). We find that ReMDM improves the best FID over ancestral sampling, from 24.73 to 23.71 using 4096 sampling steps. Nucleus sampling can help at very low step counts, but the best performance is obtained with ancestral sampling. As the number of steps increases, nucleus sampling worsens the FID. • Duo (Ancestral). Cosine training without time conditioning yields the lowest perplexity, while log-linear training without time conditioning gives the best FID. We use the latter in downstream experiments. Nucleus sampling improves FID, and greedy decoding slightly worsens it. • Duo ( \Psi -samplers). \Psi -samplers further improve performance beyond ReMDM. With the log-linear sampling schedule (as used by ReMDM), \Psi -samplers reduce the FID from 23.71 to 20.71. Using a cosine sampling schedule further improves the FID. Overall, Duo improves from an FID of 25.63 (ancestral) to 15.05 with \Psi -samplers, and MDLM improves from 24.73 (ancestral) to 17.86 with \Psi -samplers. D.1.2 OpenWebText We report the generative perplexity using GPT-2 Large, following standard practice ( 60 ; 61 ) . Because language models can artificially lower the generative perplexity by producing repetitive text, we also report unigram entropy ( 16 ) , as a proxy. Some \Psi -samplers schedules reduce the unigram entropy more than others. Therefore, for figures, we select the \kappa schedule whose unigram entropy matches (or is closest to) the entropy of samples generated with ancestral sampling. If multiple schedules achieve the same entropy, we choose the one with the lowest generative perplexity. We indicate which schedule is used for plots by highlighting the corresponding row in blue in the tables. Overall, the \Psi -samplers can reduce the Gen. PPL of all models while retaining the unigram entropy. Best results are achieved using the rescale schedule with \eta\in\{0.01,0.02\} , for both MDLM and Duo. Table 12 shows the generative perplexity of MDLM and Duo after pre-training and after distillation with SDTT ( 11 ) or DCD ( 61 ) respectively, with and without nucleus sampling, using ancestral sampling. Table 13 shows the results when sampling with \Psi -samplers that are equivalent to ReMDM ( 75 ) , with the non-distilled models, while Table 14 shows the result for the distilled models. D.2 Distribution of the top k entries of the softmax To verify that our sparse implementation accurately approximates the curriculum weights of 61 , we compare the empirical distributions of the top- k largest entries between the original and our efficient implementation. While matching marginal distributions does not guarantee matching joint distributions, matching marginals are necessary for matching joints, and are easier to visualize. Recall that experimentally, our efficient implementation is sufficient to achieve strong performance (Sec. 5.2 ). Specifically, we show histograms using a tokenizer with 100k tokens in Figures 7 , 8 , 9 , 10 , and with the GPT-2 tokenizer in Figures 11 , 12 , 13 , 14 , with varying temperature and log signal-to-noise ratios. In all cases, the top k variables have matching distributions. D.3 Training Efficiency of Our Fast Curriculum Table 4: Training efficiency comparison between Duo and \text{Duo}^{++} on 138M parameter models. All measurements are conducted on a training job on 8 NVIDIA GH200-120GB GPU with batch size 32. We report the average throughput in sequence per second. The row “Duo (after CL)” denotes the resources consumption of Duo after the Curriculum phase. The impact of k is minimal when k\in\{2,3,5\} , and \text{Duo}^{++} uses similar resources. Method Throughput Peak Memory (samples/s) \uparrow (GiB) \downarrow Duo 81.8 94.3 Duo (after the CL) 122.4 63.3 \text{Duo}^{++} ( k\in\{2,3,5\} ) 121.9 63.4 As shown in Table 4 , our sparse curriculum achieves a 33% reduction in peak memory usage and reaches an average throughput 25% higher than Duo, at a context length of 1024. Table 5: Zero-shot perplexity (PPL) on seven datasets. Lower is better. † Results taken from 61 . \text{Duo}^{++} ( k=2 ) achieves a slightly lower zero-shot perplexity than Duo on 6 of 7 datasets. PTB Wiki LM1B LBD AG News PubMed ArXiv Autoregressive Transformer † 82.05 25.75 51.25 51.28 52.09 49.01 41.73 Diffusion (138M) SEDD Uniform † 105.51 41.10 82.62 57.29 82.64 55.89 50.86 UDLM † 112.82 39.42 77.59 53.57 80.96 50.98 44.08 Duo † 89.35 33.57 73.86 49.78 67.81 44.48 40.39 \text{Duo}^{++} ( k=2 ) 94.96 34.05 73.80 48.67 67.14 43.98 38.93 \text{Duo}^{++} ( k=3 ) 91.94 34.65 74.16 49.89 66.89 44.87 40.42 \text{Duo}^{++} ( k=5 ) 94.46 34.52 74.91 50.93 68.72 46.79 41.04 Figure 7: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 100, and log signal-to-noise ratio -2 . The histograms of the efficient and naive implementation match closely. Figure 8: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio -1 . The histograms of the efficient and naive implementation match closely. Figure 9: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio -2 . The histograms of the efficient and naive implementation match closely. Figure 10: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio -4 . The histograms of the efficient and naive implementation match closely. Figure 11: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 100, and log signal-to-noise ratio -2 . The histograms of the efficient and naive implementation match closely. Figure 12: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -1 . The histograms of the efficient and naive implementation match closely. Figure 13: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -2 . The histograms of the efficient and naive implementation match closely. Figure 14: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -4 . The histograms of the efficient and naive implementation match closely. Table 6: FID on CIFAR-10 with ancestral sampling. We train and sample with the log-linear and cosine scheduler. MDLM performs best with time-conditioning while Duo does not. We sample with discrete classifier-free guidance ( 65 ) with strength 1 , and greedy predictions on the last step. Scheduler Time PPL \downarrow FID \downarrow (Cosine) FID \downarrow (Log-linear) 64 256 1024 2048 64 256 1024 2048 MDLM Cosine ✗ 8.86 42.60 27.71 24.90 24.56 107.62 40.81 27.65 25.73 Cosine ✓ 8.72 41.89 27.03 24.67 24.24 114.56 40.60 27.08 25.50 Log-linear ✗ 8.76 43.95 29.01 26.11 25.67 111.77 42.15 28.85 26.89 Log-linear ✓ 8.75 49.36 32.10 28.76 28.21 122.70 41.79 27.89 26.02 MDLM (nucleus p=0.9) Cosine ✓ 8.72 34.81 44.04 47.84 48.37 41.73 33.33 43.12 45.98 MDLM (no greedy) Cosine ✓ 8.72 42.14 27.19 24.47 24.46 114.55 40.92 27.13 25.60 Duo Cosine ✗ 10.27 32.37 27.28 26.38 26.02 33.93 27.93 26.51 26.03 Cosine ✓ 10.32 33.74 27.98 26.81 26.96 36.23 28.77 27.08 26.79 Log-linear ✗ 10.49 31.78 27.03 26.00 25.75 33.44 27.46 26.08 25.87 Log-linear ✓ 10.45 34.05 27.74 26.58 26.37 36.46 28.49 26.60 26.22 Duo (nucleus p=0.9) Log-linear ✗ 10.49 23.13 22.21 22.58 22.49 24.24 22.41 22.35 22.54 Duo (no greedy) Log-linear ✗ 10.49 33.03 27.43 26.16 25.96 34.81 27.76 26.30 26.06 Table 7: FID on CIFAR-10 with ancestral sampling and a finer grid. We pick the variant with the best FID from Table 6 . Algo Train Sample p FID \downarrow 32 64 128 256 512 1024 2048 4096 Duo log-lin log-lin 1.0 42.71 33.44 29.18 27.46 26.62 26.08 25.87 25.79 Duo log-lin log-lin 0.9 28.53 24.24 22.89 22.41 22.56 22.35 22.54 22.41 Duo log-lin cos 1.0 39.65 31.78 28.55 27.03 26.03 25.89 25.75 25.63 Duo log-lin cos 0.9 25.96 23.13 22.68 22.21 22.26 22.58 22.49 22.49 MDLM cos log-lin 1.0 212.95 114.56 62.86 40.60 31.05 27.08 25.50 24.73 MDLM cos log-lin 0.9 84.85 41.73 31.28 33.33 38.49 43.12 45.98 55.37 MDLM cos cos 1.0 73.82 41.89 36.21 27.03 25.63 24.67 24.24 23.93 MDLM cos cos 0.9 58.31 34.81 37.91 44.04 45.32 47.84 48.37 49.23 Table 8: FID on CIFAR-10 with ReMDM (best checkpoints, as shown in Table 7 ). We sample with/without nucleus sampling, and with the 3 schedules of 75 (cap, loop, rescale). For the loop schedule, we use t_{\text{on}}=0.55 , t_{\text{off}}=0.05 , \alpha_{\text{on}}=0.9 , following ReMDM. Sampling experiments are executed in the original codebase of 75 . Number of steps 32 64 128 256 512 1024 2048 4096 ReMDM cap (p=1.0) \eta=0.005 215.67 116.24 63.37 40.82 31.40 27.28 24.97 24.78 \eta=0.010 218.41 118.25 64.50 41.77 32.40 28.68 27.91 33.68 \eta=0.020 224.20 122.61 66.95 44.54 36.26 35.39 46.01 92.48 \eta=0.050 242.25 143.21 84.41 64.10 73.89 132.13 210.60 203.14 ReMDM loop (p=1.0) \eta=0.01 307.56 234.55 138.56 80.50 55.86 47.05 45.44 50.44 \eta=0.02 307.81 237.28 142.21 83.68 59.96 53.88 60.50 87.54 \eta=0.04 308.24 242.70 152.28 94.63 76.93 88.53 135.05 196.58 \eta=0.06 308.88 248.76 165.79 114.92 113.26 157.92 223.70 237.16 ReMDM rescale (p=1.0) \eta=0.01 216.92 116.73 63.56 40.65 30.86 26.03 23.77 23.71 \eta=0.02 221.21 119.79 65.08 42.02 32.29 28.11 28.66 39.39 \eta=0.04 229.72 127.94 70.89 46.98 38.74 41.23 67.44 130.05 \eta=0.05 234.35 133.08 75.02 50.92 45.01 57.03 107.13 164.44 ReMDM cap (p=0.9) \eta=0.005 88.08 40.02 27.31 29.43 36.50 45.10 57.08 73.40 \eta=0.010 87.68 39.55 27.35 31.24 41.22 54.55 71.65 93.06 \eta=0.020 85.95 38.46 27.80 35.01 50.50 69.60 91.49 118.87 \eta=0.050 81.91 35.56 29.39 46.90 70.24 95.24 125.60 163.32 ReMDM loop (p=0.9) \eta=0.01 209.24 100.01 47.27 29.44 27.55 30.50 34.21 37.56 \eta=0.02 208.36 99.29 47.12 29.38 27.74 31.17 35.42 39.52 \eta=0.04 206.51 98.18 46.87 29.28 28.09 32.12 37.19 42.45 \eta=0.06 204.83 97.24 46.72 29.19 28.30 32.77 38.47 44.64 ReMDM rescale (p=0.9) \eta=0.01 87.31 39.51 27.25 30.74 40.22 53.30 70.24 91.79 \eta=0.02 85.94 38.45 27.45 34.13 49.00 67.89 90.61 118.10 \eta=0.04 83.47 36.44 28.29 41.76 63.40 87.03 115.60 153.03 \eta=0.05 82.26 35.69 28.99 44.69 68.80 94.07 125.42 165.62 ReMDM Best ( p=1.0 ) 215.67 116.24 63.37 40.65 30.86 26.03 23.77 23.71 Best ( p=0.9 ) 81.91 81.91 27.25 29.19 27.55 30.50 34.21 37.56 MDLM Ancestral (p=1.0) 212.95 114.56 62.86 40.60 31.05 27.08 25.50 24.73 Ancestral (p=0.9) 84.85 41.73 31.28 33.33 38.49 43.12 45.98 55.37 Table 9: FID on CIFAR-10 with \Psi -samplers, where \Psi -samplers are activated for steps with t\in[t_{\text{off}},t_{\text{on}}] , when \kappa_{t} is kept constant (according to the \kappa column, 1 otherwise). We use the same checkpoints as in Table 7 . Using a cosine sampling schedule and light noise injection ( \kappa close to 1) generally perform best. The CIFAR-10 curves in Fig. 1 show the best FID per number of steps. Algo \kappa Train Sample t_{\text{on}} t_{\text{off}} FID \downarrow 32 64 128 256 512 1024 2048 4096 Duo 0.02 log-lin cos 0.2 0.15 40.64 33.06 30.36 29.85 31.31 34.36 39.06 38.38 Duo 0.02 log-lin cos 0.5 0.45 41.81 33.67 29.50 26.55 24.83 25.12 31.63 51.83 Duo 0.02 log-lin cos 0.8 0.7 43.99 37.41 35.68 38.88 46.76 59.68 75.46 91.73 Duo 0.5 log-lin cos 0.2 0.1 39.95 32.14 28.86 27.18 26.57 26.46 27.29 28.35 Duo 0.5 log-lin cos 0.6 0.4 39.54 29.40 23.46 20.77 23.72 38.42 72.97 105.75 Duo 0.5 log-lin cos 0.9 0.65 43.00 34.68 31.85 34.73 45.68 64.97 88.07 107.36 Duo 0.95 log-lin cos 0.5 0.1 39.30 30.58 26.15 23.46 20.93 18.48 16.38 15.05 Duo 0.95 log-lin cos 0.6 0.1 39.19 30.15 25.14 21.54 18.64 16.70 16.30 18.99 Duo 0.95 log-lin cos 0.9 0.3 39.04 29.88 24.72 20.90 19.20 21.09 30.00 51.43 Duo 0.95 log-lin cos 0.9 0.4 39.21 30.29 25.26 21.57 19.92 21.50 30.03 50.88 Duo 0.98 log-lin cos 1.0 0.05 39.31 30.97 26.39 23.13 20.56 18.80 19.46 25.83 Duo 0.98 log-lin cos 1.0 0.1 39.31 30.99 26.40 23.14 20.58 18.83 19.48 25.82 Duo 0.99 log-lin cos 1.0 0.05 39.34 31.56 27.46 24.73 22.35 20.07 18.50 19.39 Duo 0.99 log-lin cos 1.0 0.1 39.35 31.57 27.46 24.73 22.37 20.09 18.51 19.41 Duo 0.02 log-lin log-lin 0.2 0.15 42.25 33.71 29.84 27.95 27.64 27.56 29.35 31.02 Duo 0.02 log-lin log-lin 0.5 0.45 43.86 36.29 33.35 33.24 34.74 36.97 36.77 37.30 Duo 0.02 log-lin log-lin 0.8 0.7 43.95 33.75 28.32 27.78 37.12 69.66 113.05 132.86 Duo 0.5 log-lin log-lin 0.2 0.1 42.10 33.40 29.19 27.14 26.22 25.52 25.10 24.71 Duo 0.5 log-lin log-lin 0.6 0.4 42.44 33.68 29.15 25.93 24.16 22.44 21.00 27.97 Duo 0.5 log-lin log-lin 0.9 0.65 42.87 31.04 26.37 31.86 61.36 121.64 155.77 151.48 Duo 0.95 log-lin log-lin 0.5 0.1 41.74 32.97 28.57 26.05 24.62 23.13 21.81 20.16 Duo 0.95 log-lin log-lin 0.6 0.1 41.46 32.47 27.74 24.97 22.94 20.83 18.87 16.82 Duo 0.95 log-lin log-lin 0.9 0.3 41.10 30.55 24.54 20.50 17.97 18.04 22.14 35.43 Duo 0.95 log-lin log-lin 0.9 0.4 41.18 30.58 24.71 20.59 18.08 18.02 22.07 35.44 Duo 0.98 log-lin log-lin 1.0 0.05 41.80 31.96 26.83 23.17 20.10 18.12 18.38 22.89 Duo 0.98 log-lin log-lin 1.0 0.1 41.81 31.98 26.85 23.17 20.12 18.15 18.40 22.94 Duo 0.99 log-lin log-lin 1.0 0.05 41.99 32.63 27.74 24.67 22.13 19.72 17.93 18.25 Duo 0.99 log-lin log-lin 1.0 0.1 41.99 32.63 27.75 24.67 22.13 19.75 17.95 18.28 MDLM 0.02 cos cos 0.2 0.15 75.63 49.18 45.02 54.67 83.47 181.18 280.42 297.52 MDLM 0.02 cos cos 0.5 0.45 117.57 89.53 111.75 200.49 283.55 310.51 314.98 313.93 MDLM 0.02 cos cos 0.8 0.7 172.24 197.61 232.36 262.87 269.22 267.86 264.57 259.88 MDLM 0.5 cos cos 0.2 0.1 73.13 46.10 38.47 39.71 48.49 75.27 173.09 266.36 MDLM 0.5 cos cos 0.6 0.4 134.11 114.88 144.25 217.74 268.03 274.83 270.53 256.03 MDLM 0.5 cos cos 0.9 0.65 151.90 131.04 147.67 177.75 198.33 201.97 193.77 184.76 MDLM 0.95 cos cos 0.5 0.1 73.03 44.15 33.68 30.50 29.93 31.50 35.72 51.53 MDLM 0.95 cos cos 0.6 0.1 74.57 45.00 34.07 30.32 29.16 31.03 37.46 64.74 MDLM 0.95 cos cos 0.9 0.3 79.25 47.02 33.97 27.84 24.24 23.43 26.96 42.58 MDLM 0.95 cos cos 0.9 0.4 78.18 46.36 33.06 26.69 22.67 20.91 21.90 28.82 MDLM 0.98 cos cos 1.0 0.05 74.05 43.85 32.32 26.69 23.22 20.81 19.41 20.20 MDLM 0.98 cos cos 1.0 0.1 74.05 43.85 32.31 26.65 23.17 20.76 19.26 19.98 MDLM 0.99 cos cos 1.0 0.05 72.39 42.87 31.79 26.65 23.72 21.07 19.24 17.94 MDLM 0.99 cos cos 1.0 0.1 72.38 42.87 31.78 26.64 23.69 21.04 19.19 17.86 MDLM 0.02 cos log-lin 0.2 0.15 217.56 118.08 68.02 51.76 55.02 78.21 171.72 275.25 MDLM 0.02 cos log-lin 0.5 0.45 247.31 157.61 124.97 162.92 256.01 298.74 305.05 310.28 MDLM 0.02 cos log-lin 0.8 0.7 298.96 294.71 298.95 312.49 317.03 312.60 308.42 302.37 MDLM 0.5 cos log-lin 0.2 0.1 216.08 116.99 65.73 45.72 41.32 45.95 68.60 152.77 MDLM 0.5 cos log-lin 0.6 0.4 266.16 195.76 171.73 212.68 273.48 281.96 272.45 260.26 MDLM 0.5 cos log-lin 0.9 0.65 296.08 268.98 265.73 278.38 281.68 275.20 265.49 247.21 MDLM 0.95 cos log-lin 0.5 0.1 216.90 117.05 64.76 43.50 36.06 34.84 37.06 44.92 MDLM 0.95 cos log-lin 0.6 0.1 218.58 118.21 65.33 44.32 37.14 36.09 39.42 55.34 MDLM 0.95 cos log-lin 0.9 0.3 225.19 124.03 67.82 44.06 35.20 33.97 42.48 80.34 MDLM 0.95 cos log-lin 0.9 0.4 223.84 123.04 67.19 43.29 33.85 32.00 37.23 63.89 MDLM 0.98 cos log-lin 1.0 0.05 218.15 118.08 63.97 40.97 30.67 25.69 23.64 25.40 MDLM 0.98 cos log-lin 1.0 0.1 218.14 118.09 63.96 40.96 30.65 25.64 23.57 25.29 MDLM 0.99 cos log-lin 1.0 0.05 215.41 116.02 63.30 40.42 30.43 25.37 22.45 20.77 MDLM 0.99 cos log-lin 1.0 0.1 215.40 116.03 63.27 40.41 30.43 25.35 22.42 20.71 Table 10: Inception Score on CIFAR-10 with \Psi -samplers, where \Psi -samplers are activated for steps with t\in[t_{\text{off}},t_{\text{on}}] , when \kappa_{t} is kept constant (according to the \kappa column, 1 otherwise). We use the same checkpoints as in Table 7 . The CIFAR-10 curves in Fig. 6 show the best Inception Score per number of steps. Algo \kappa Train Sample t_{\text{on}} t_{\text{off}} Inception Score \uparrow 32 64 128 256 512 1024 2048 4096 Duo 0.02 log-lin cos 0.2 0.15 7.02 7.25 7.35 7.48 7.52 7.47 7.38 7.63 Duo 0.02 log-lin cos 0.5 0.45 7.09 7.44 7.64 8.04 8.32 8.59 8.57 7.94 Duo 0.02 log-lin cos 0.8 0.7 6.84 6.99 7.00 6.91 6.64 6.16 5.67 5.19 Duo 0.5 log-lin cos 0.2 0.1 6.96 7.21 7.28 7.39 7.45 7.48 7.56 7.73 Duo 0.5 log-lin cos 0.6 0.4 7.31 7.73 8.14 8.51 8.46 7.91 6.40 5.39 Duo 0.5 log-lin cos 0.9 0.65 6.87 7.10 7.22 7.11 6.72 5.97 5.23 4.67 Duo 0.95 log-lin cos 0.5 0.1 6.98 7.26 7.45 7.53 7.67 7.89 8.06 8.29 Duo 0.95 log-lin cos 0.6 0.1 7.00 7.31 7.45 7.70 7.91 8.17 8.34 8.46 Duo 0.95 log-lin cos 0.9 0.3 7.08 7.37 7.54 7.84 8.01 8.07 7.72 6.84 Duo 0.95 log-lin cos 0.9 0.4 7.04 7.31 7.50 7.78 7.92 8.08 7.78 6.89 Duo 0.98 log-lin cos 1.0 0.05 7.00 7.25 7.40 7.55 7.73 7.97 8.10 7.91 Duo 0.98 log-lin cos 1.0 0.1 6.99 7.25 7.40 7.55 7.74 7.97 8.09 7.91 Duo 0.99 log-lin cos 1.0 0.05 6.98 7.22 7.37 7.45 7.58 7.77 7.96 8.08 Duo 0.99 log-lin cos 1.0 0.1 6.98 7.22 7.37 7.46 7.58 7.77 7.96 8.10 Duo 0.02 log-lin log-lin 0.2 0.15 6.82 7.09 7.22 7.30 7.36 7.44 7.46 7.43 Duo 0.02 log-lin log-lin 0.5 0.45 6.95 7.28 7.45 7.64 7.67 7.70 8.06 8.68 Duo 0.02 log-lin log-lin 0.8 0.7 7.00 7.54 8.02 8.18 7.89 6.46 5.03 4.55 Duo 0.5 log-lin log-lin 0.2 0.1 6.81 7.04 7.20 7.26 7.29 7.36 7.47 7.50 Duo 0.5 log-lin log-lin 0.6 0.4 7.04 7.45 7.73 7.93 8.20 8.51 9.00 9.50 Duo 0.5 log-lin log-lin 0.9 0.65 7.05 7.61 7.97 7.74 6.45 4.46 3.77 4.07 Duo 0.95 log-lin log-lin 0.5 0.1 6.80 7.10 7.25 7.31 7.35 7.43 7.55 7.63 Duo 0.95 log-lin log-lin 0.6 0.1 6.85 7.12 7.28 7.40 7.46 7.66 7.81 7.97 Duo 0.95 log-lin log-lin 0.9 0.3 6.89 7.27 7.58 7.78 8.10 8.22 8.20 7.67 Duo 0.95 log-lin log-lin 0.9 0.4 6.89 7.25 7.58 7.80 8.05 8.25 8.26 7.69 Duo 0.98 log-lin log-lin 1.0 0.05 6.85 7.19 7.36 7.49 7.72 7.96 8.05 8.03 Duo 0.98 log-lin log-lin 1.0 0.1 6.85 7.20 7.38 7.49 7.72 7.96 8.04 8.02 Duo 0.99 log-lin log-lin 1.0 0.05 6.81 7.13 7.32 7.45 7.61 7.71 7.98 8.12 Duo 0.99 log-lin log-lin 1.0 0.1 6.81 7.14 7.32 7.45 7.62 7.70 7.99 8.13 MDLM 0.02 cos cos 0.2 0.15 5.56 6.61 6.90 6.75 5.52 2.68 1.57 1.56 MDLM 0.02 cos cos 0.5 0.45 4.22 5.11 4.36 2.44 1.61 1.41 1.45 1.56 MDLM 0.02 cos cos 0.8 0.7 3.12 2.82 2.41 2.03 1.96 1.97 2.02 2.09 MDLM 0.5 cos cos 0.2 0.1 5.63 6.63 7.00 7.09 6.90 5.68 2.78 1.73 MDLM 0.5 cos cos 0.6 0.4 3.83 4.32 3.55 2.35 1.85 2.14 2.51 2.99 MDLM 0.5 cos cos 0.9 0.65 3.62 4.18 3.95 3.47 3.18 3.15 3.37 3.75 MDLM 0.95 cos cos 0.5 0.1 5.66 6.68 7.13 7.29 7.44 7.41 7.44 6.77 MDLM 0.95 cos cos 0.6 0.1 5.59 6.70 7.21 7.41 7.52 7.57 7.45 6.33 MDLM 0.95 cos cos 0.9 0.3 5.43 6.68 7.25 7.63 7.90 8.15 8.18 7.58 MDLM 0.95 cos cos 0.9 0.4 5.45 6.66 7.25 7.64 7.93 8.14 8.30 8.18 MDLM 0.98 cos cos 1.0 0.05 5.57 6.71 7.22 7.45 7.71 7.93 8.14 8.30 MDLM 0.98 cos cos 1.0 0.1 5.57 6.72 7.22 7.46 7.72 7.93 8.15 8.31 MDLM 0.99 cos cos 1.0 0.05 5.60 6.73 7.18 7.39 7.53 7.81 7.97 8.12 MDLM 0.99 cos cos 1.0 0.1 5.60 6.73 7.19 7.39 7.53 7.81 7.97 8.14 MDLM 0.02 cos log-lin 0.2 0.15 2.63 4.59 5.86 6.46 6.45 5.59 2.78 1.67 MDLM 0.02 cos log-lin 0.5 0.45 2.21 3.56 4.08 3.06 1.78 1.43 1.38 1.36 MDLM 0.02 cos log-lin 0.8 0.7 1.65 1.63 1.55 1.43 1.42 1.60 1.80 1.96 MDLM 0.5 cos log-lin 0.2 0.1 2.66 4.60 5.91 6.58 6.81 6.76 5.77 3.15 MDLM 0.5 cos log-lin 0.6 0.4 1.97 2.78 2.99 2.27 1.62 1.58 1.92 2.33 MDLM 0.5 cos log-lin 0.9 0.65 1.69 1.91 1.90 1.79 1.91 2.35 2.87 3.29 MDLM 0.95 cos log-lin 0.5 0.1 2.65 4.60 5.95 6.64 6.93 7.03 7.07 6.78 MDLM 0.95 cos log-lin 0.6 0.1 2.62 4.55 5.94 6.66 6.98 7.16 7.10 6.47 MDLM 0.95 cos log-lin 0.9 0.3 2.51 4.45 5.93 6.84 7.35 7.61 7.31 5.83 MDLM 0.95 cos log-lin 0.9 0.4 2.54 4.46 5.94 6.85 7.40 7.69 7.59 6.60 MDLM 0.98 cos log-lin 1.0 0.05 2.62 4.56 6.04 6.85 7.31 7.66 7.79 8.01 MDLM 0.98 cos log-lin 1.0 0.1 2.62 4.56 6.03 6.85 7.32 7.67 7.81 8.03 MDLM 0.99 cos log-lin 1.0 0.05 2.68 4.64 6.02 6.84 7.21 7.47 7.70 7.93 MDLM 0.99 cos log-lin 1.0 0.1 2.68 4.64 6.02 6.84 7.21 7.47 7.70 7.93 Table 11: FID on CIFAR-10 using \Psi -samplers whose \kappa_{t} schedulers are equivalent to ReMDM. We use no nucleus sampling, no temperature scaling, and \mathrm{cfg}=1 . As expected, with the log-linear scheduler, we reach a similar FID as when using the ReMDM codebase (Table 8 ). However, note that by using a log-linear scheduler, using a constant \kappa_{t}=0.99 , we reach a better FID than with the original ReMDM scheduler. Algo Train Sample FID \downarrow 32 64 128 256 512 1024 2048 4096 Duo with the ReMDM rescale schedule Duo log-lin cos 39.64 32.03 28.49 26.95 26.16 25.71 25.25 25.02 Duo log-lin log-lin 42.27 33.58 29.49 27.36 26.33 25.86 25.07 25.21 ReMDM Rescale ( \eta=0.01 ) MDLM cos cos 70.64 41.94 31.60 27.31 25.27 24.61 23.41 23.25 MDLM cos log-lin 213.22 114.24 62.51 40.51 30.28 26.21 23.61 23.40 ReMDM Cap ( \eta=0.005 ) MDLM cos log-lin 215.75 115.77 63.20 41.25 31.60 27.30 25.16 24.79 ReMDM Loop ( t_{\text{on}}=0.55,t_{\text{off}}=0.05,\alpha_{\text{on}}=0.9,\eta=0.01 ) MDLM cos log-lin 305.30 224.84 120.58 66.39 45.70 39.06 41.44 52.71 Table 12: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( 21 ) with ancestral sampling (no nucleus, no temperature scaling). We train using the log-linear noise scheduler, and sampling with the cosine scheduler is slightly better. We stick to the log-linear schedule for sampling in further experiments, to follow prior work, and since the cosine schedule only marginally reduces the Gen. PPL. Algo Dist. p Sched. Gen. PPL ( \downarrow ) 32 64 128 256 512 1024 2048 4096 Duo ✗ 1.0 cos 87.23 (5.54) 79.94 (5.55) 75.87 (5.53) 73.95 (5.54) 72.13 (5.54) 71.41 (5.53) 72.29 (5.53) 70.77 (5.52) Duo ✗ 1.0 log-lin 96.76 (5.57) 86.01 (5.56) 79.97 (5.55) 78.46 (5.53) 76.93 (5.54) 75.02 (5.53) 75.65 (5.52) 75.39 (5.52) Duo ✗ 0.9 cos 42.42 (5.36) 39.26 (5.37) 37.62 (5.35) 36.52 (5.35) 35.21 (5.34) 35.37 (5.34) 35.39 (5.34) 34.91 (5.33) Duo ✗ 0.9 log-lin 44.24 (5.40) 40.08 (5.40) 37.93 (5.39) 36.66 (5.37) 35.77 (5.37) 34.79 (5.35) 34.93 (5.35) 34.75 (5.35) Duo ✓ 1.0 cos 67.04 (5.47) 61.09 (5.45) 59.65 (5.42) 57.76 (5.42) 57.90 (5.42) 56.81 (5.43) 56.39 (5.41) 57.32 (5.42) Duo ✓ 1.0 log-lin 68.35 (5.54) 62.92 (5.54) 59.82 (5.50) 58.77 (5.46) 58.32 (5.46) 57.82 (5.45) 55.39 (5.43) 55.89 (5.42) Duo ✓ 0.9 cos 34.20 (5.31) 31.79 (5.29) 31.09 (5.25) 30.05 (5.25) 29.82 (5.26) 29.68 (5.27) 29.52 (5.24) 29.73 (5.23) Duo ✓ 0.9 log-lin 35.92 (5.41) 32.98 (5.40) 31.49 (5.36) 30.32 (5.31) 30.06 (5.29) 30.00 (5.28) 28.90 (5.25) 29.19 (5.25) MDLM ✗ 1.0 cos 168.66 (5.68) 131.55 (5.66) 115.74 (5.64) 111.72 (5.63) 106.63 (5.63) 104.56 (5.62) 103.12 (5.62) 104.73 (5.62) MDLM ✗ 1.0 log-lin 194.09 (5.74) 141.67 (5.69) 120.95 (5.67) 111.85 (5.65) 107.89 (5.64) 105.64 (5.64) 105.40 (5.63) 105.03 (5.62) MDLM ✗ 0.9 cos 58.33 (5.39) 46.71 (5.36) 40.66 (5.32) 39.43 (5.33) 37.64 (5.32) 37.39 (5.33) 36.98 (5.31) 36.87 (5.31) MDLM ✗ 0.9 log-lin 70.34 (5.49) 51.14 (5.43) 43.60 (5.39) 40.01 (5.37) 39.02 (5.35) 37.91 (5.34) 37.59 (5.32) 36.76 (5.31) MDLM ✓ 1.0 cos 63.04 (5.45) 52.72 (5.43) 47.83 (5.41) 45.94 (5.42) 44.67 (5.41) 44.60 (5.41) 44.50 (5.41) 44.42 (5.41) MDLM ✓ 1.0 log-lin 68.61 (5.48) 55.26 (5.45) 49.51 (5.44) 46.13 (5.42) 45.61 (5.42) 44.87 (5.42) 44.53 (5.41) 44.38 (5.42) MDLM ✓ 0.9 cos 31.47 (5.21) 26.52 (5.19) 24.14 (5.18) 23.49 (5.17) 22.93 (5.17) 22.64 (5.17) 22.38 (5.16) 22.49 (5.17) MDLM ✓ 0.9 log-lin 34.85 (5.26) 28.21 (5.23) 25.27 (5.21) 24.01 (5.19) 23.25 (5.18) 22.75 (5.17) 22.73 (5.17) 22.46 (5.16) Table 13: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( 21 ) with \Psi -samplers using \kappa_{t} schedules matching ReMDM (log-linear step size) and non-distilled models (as in Table 12 ). We experiment with nucleus sampling, following 75 . The rescale schedule is most effective to improve the Gen. PPL while retaining the unigram entropy. The lightblue rows are the ones plotted in Fig. 1 (left). Algo Eta Nucleus P Gen. PPL ( \downarrow ) 32 64 128 256 512 1024 2048 4096 Ancestral Sampling Duo N.A 1.0 96.76 (5.57) 86.01 (5.56) 79.97 (5.55) 78.46 (5.53) 76.93 (5.54) 75.02 (5.53) 75.65 (5.52) 75.39 (5.52) Duo N.A 0.95 56.65 (5.49) 50.78 (5.48) 48.68 (5.48) 47.26 (5.46) 45.42 (5.45) 45.11 (5.44) 45.12 (5.44) 44.84 (5.44) Duo N.A 0.9 44.24 (5.40) 40.08 (5.40) 37.93 (5.39) 36.66 (5.37) 35.77 (5.37) 34.79 (5.35) 34.93 (5.35) 34.75 (5.35) MDLM N.A 1.0 194.09 (5.74) 141.67 (5.69) 120.95 (5.67) 111.85 (5.65) 107.89 (5.64) 105.64 (5.64) 105.40 (5.63) 105.03 (5.62) MDLM N.A 0.95 106.28 (5.61) 77.06 (5.55) 68.34 (5.53) 63.19 (5.51) 58.80 (5.49) 56.94 (5.48) 57.54 (5.47) 56.44 (5.46) MDLM N.A 0.9 70.34 (5.49) 51.14 (5.43) 43.60 (5.39) 40.01 (5.37) 39.02 (5.35) 37.91 (5.34) 37.59 (5.32) 36.76 (5.31) Cap Schedule Duo 0.005 1.0 88.78 (5.58) 77.12 (5.57) 72.05 (5.56) 66.44 (5.54) 61.63 (5.53) 57.14 (5.51) 52.49 (5.51) 45.64 (5.45) Duo 0.01 1.0 86.89 (5.58) 75.23 (5.56) 68.98 (5.55) 63.66 (5.54) 57.34 (5.52) 52.06 (5.50) 46.04 (5.46) 39.48 (5.39) Duo 0.005 0.95 55.56 (5.49) 48.74 (5.47) 44.93 (5.46) 40.53 (5.43) 36.26 (5.41) 30.85 (5.37) 25.66 (5.32) 20.22 (5.22) Duo 0.01 0.95 54.07 (5.48) 46.27 (5.46) 41.93 (5.45) 36.60 (5.41) 30.98 (5.37) 25.53 (5.31) 20.10 (5.23) 15.19 (5.07) Duo 0.005 0.9 44.06 (5.41) 38.38 (5.39) 34.84 (5.37) 30.95 (5.33) 27.37 (5.30) 22.78 (5.24) 18.66 (5.16) 14.33 (5.03) Duo 0.01 0.9 43.05 (5.40) 36.75 (5.38) 32.27 (5.35) 27.83 (5.30) 23.38 (5.26) 18.74 (5.17) 14.40 (5.06) 10.88 (4.87) MDLM 0.005 1.0 195.83 (5.74) 142.25 (5.70) 121.99 (5.68) 113.94 (5.67) 110.75 (5.66) 112.78 (5.67) 119.61 (5.69) 131.85 (5.71) MDLM 0.01 1.0 198.02 (5.75) 144.89 (5.70) 125.25 (5.68) 117.84 (5.68) 116.62 (5.68) 126.32 (5.71) 143.96 (5.73) 186.72 (5.76) MDLM 0.005 0.95 106.40 (5.61) 74.97 (5.54) 63.15 (5.52) 55.82 (5.49) 50.31 (5.47) 43.78 (5.44) 37.04 (5.40) 30.46 (5.34) MDLM 0.01 0.95 105.45 (5.61) 73.92 (5.54) 61.41 (5.51) 52.81 (5.48) 46.03 (5.45) 38.85 (5.42) 31.30 (5.34) 24.31 (5.23) MDLM 0.005 0.9 69.20 (5.49) 49.59 (5.42) 41.08 (5.38) 35.19 (5.34) 31.49 (5.31) 26.33 (5.26) 21.16 (5.18) 15.87 (5.04) MDLM 0.01 0.9 68.57 (5.48) 48.30 (5.42) 38.80 (5.37) 32.38 (5.32) 27.66 (5.28) 21.57 (5.18) 16.26 (5.05) 11.67 (4.79) Rescale Schedule Duo 0.01 1.0 89.63 (5.58) 79.80 (5.57) 76.11 (5.56) 73.43 (5.55) 70.66 (5.54) 70.46 (5.53) 69.20 (5.54) 68.25 (5.53) Duo 0.02 1.0 89.55 (5.58) 79.44 (5.57) 75.98 (5.56) 72.99 (5.54) 69.85 (5.54) 68.39 (5.53) 66.60 (5.53) 63.70 (5.52) Duo 0.01 0.95 56.68 (5.49) 50.80 (5.48) 48.38 (5.47) 46.91 (5.46) 45.24 (5.45) 44.64 (5.44) 44.11 (5.44) 43.49 (5.43) Duo 0.02 0.95 56.68 (5.49) 50.66 (5.48) 48.09 (5.47) 46.19 (5.46) 44.17 (5.44) 42.71 (5.43) 41.47 (5.43) 38.06 (5.40) Duo 0.01 0.9 45.03 (5.41) 40.02 (5.40) 38.17 (5.39) 36.60 (5.36) 35.25 (5.35) 34.35 (5.34) 34.27 (5.35) 33.07 (5.33) Duo 0.02 0.9 45.04 (5.41) 40.00 (5.40) 38.05 (5.39) 36.15 (5.36) 34.74 (5.35) 33.13 (5.33) 31.79 (5.32) 29.08 (5.30) Duo 0.03 0.9 44.87 (5.41) 40.05 (5.40) 37.61 (5.39) 35.26 (5.36) 33.35 (5.34) 31.17 (5.32) 28.90 (5.31) 24.93 (5.26) Duo 0.04 0.9 44.43 (5.41) 39.67 (5.39) 37.21 (5.38) 34.75 (5.35) 32.47 (5.34) 29.30 (5.31) 26.15 (5.28) 22.05 (5.22) Duo 0.05 0.9 44.52 (5.41) 39.49 (5.40) 36.41 (5.38) 33.68 (5.35) 31.06 (5.34) 26.94 (5.28) 23.61 (5.25) 19.21 (5.17) MDLM 0.01 1.0 194.29 (5.74) 141.40 (5.69) 121.04 (5.67) 112.95 (5.65) 107.80 (5.64) 105.58 (5.64) 105.69 (5.63) 105.64 (5.63) MDLM 0.02 1.0 194.54 (5.74) 140.81 (5.69) 120.86 (5.67) 112.64 (5.65) 108.26 (5.64) 105.65 (5.64) 104.47 (5.63) 105.61 (5.64) MDLM 0.01 0.95 106.43 (5.61) 76.89 (5.55) 65.42 (5.52) 61.07 (5.50) 58.77 (5.49) 56.34 (5.47) 56.29 (5.47) 54.42 (5.45) MDLM 0.02 0.95 105.92 (5.60) 76.23 (5.55) 65.43 (5.52) 60.80 (5.50) 57.32 (5.49) 54.94 (5.47) 53.92 (5.46) 50.57 (5.45) MDLM 0.01 0.9 70.45 (5.49) 51.33 (5.43) 43.59 (5.39) 40.14 (5.36) 38.68 (5.35) 37.64 (5.34) 36.48 (5.32) 35.10 (5.31) MDLM 0.02 0.9 70.31 (5.49) 51.06 (5.43) 43.51 (5.39) 39.61 (5.36) 37.88 (5.35) 36.28 (5.33) 34.53 (5.31) 31.62 (5.29) MDLM 0.03 0.9 69.89 (5.49) 50.76 (5.42) 43.23 (5.39) 38.86 (5.36) 36.77 (5.34) 34.62 (5.32) 31.44 (5.29) 27.19 (5.25) MDLM 0.04 0.9 69.54 (5.49) 50.30 (5.42) 42.84 (5.39) 38.02 (5.35) 35.73 (5.33) 32.44 (5.31) 28.55 (5.27) 23.72 (5.21) MDLM 0.05 0.9 69.44 (5.48) 50.15 (5.42) 42.39 (5.38) 37.27 (5.35) 34.10 (5.33) 30.29 (5.30) 26.03 (5.25) 20.85 (5.16) Loop Schedule Duo 0.01 1.0 108.15 (5.58) 83.10 (5.58) 71.16 (5.56) 66.15 (5.55) 60.49 (5.55) 56.35 (5.53) 53.06 (5.51) 48.93 (5.48) Duo 0.02 1.0 103.48 (5.58) 79.75 (5.58) 67.99 (5.56) 63.05 (5.55) 56.92 (5.54) 52.69 (5.51) 48.63 (5.47) 43.28 (5.37) Duo 0.01 0.95 65.29 (5.49) 51.36 (5.48) 43.27 (5.46) 37.64 (5.43) 32.04 (5.40) 26.97 (5.35) 22.94 (5.30) 18.40 (5.20) Duo 0.02 0.95 61.61 (5.48) 47.46 (5.47) 38.78 (5.44) 32.69 (5.40) 27.26 (5.36) 22.35 (5.29) 18.43 (5.22) 14.31 (5.06) Duo 0.01 0.9 52.12 (5.40) 40.27 (5.39) 33.71 (5.37) 28.73 (5.33) 24.47 (5.29) 20.32 (5.23) 17.01 (5.16) 13.61 (5.05) Duo 0.02 0.9 49.08 (5.40) 37.00 (5.38) 30.08 (5.34) 24.88 (5.29) 20.59 (5.24) 16.69 (5.16) 13.61 (5.06) 10.77 (4.92) MDLM 0.01 1.0 340.32 (5.81) 192.48 (5.74) 140.70 (5.70) 127.32 (5.70) 119.34 (5.69) 127.63 (5.70) 149.13 (5.73) 198.48 (5.77) MDLM 0.02 1.0 338.82 (5.82) 193.71 (5.75) 144.92 (5.72) 140.73 (5.72) 136.30 (5.71) 162.47 (5.75) 246.89 (5.81) 354.65 (5.78) MDLM 0.01 0.95 182.65 (5.67) 101.56 (5.61) 71.76 (5.56) 58.43 (5.52) 51.33 (5.50) 45.27 (5.47) 39.08 (5.43) 33.48 (5.38) MDLM 0.02 0.95 177.31 (5.67) 97.61 (5.61) 68.49 (5.55) 55.21 (5.51) 47.71 (5.49) 41.64 (5.45) 34.91 (5.40) 29.63 (5.33) MDLM 0.01 0.9 117.28 (5.55) 65.24 (5.48) 46.91 (5.43) 37.62 (5.38) 31.93 (5.34) 27.80 (5.31) 23.38 (5.25) 19.78 (5.20) MDLM 0.02 0.9 112.21 (5.55) 61.93 (5.48) 43.89 (5.42) 34.69 (5.37) 28.99 (5.33) 24.58 (5.29) 20.09 (5.20) 16.68 (5.13) Table 14: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( 21 ) with \Psi -samplers using \kappa_{t} schedules matching ReMDM (log-linear step size) and distilled models (as in Table 12 ). We experiment with nucleus sampling, following 75 . Algo Eta Nucleus P Gen. PPLL ( \downarrow ) 32 64 128 256 512 1024 2048 4096 Ancestral Sampling Duo N.A 1.0 68.35 (5.54) 62.92 (5.54) 59.82 (5.50) 58.77 (5.46) 58.32 (5.46) 57.82 (5.45) 55.39 (5.43) 55.89 (5.42) Duo N.A 0.95 44.94 (5.47) 41.78 (5.46) 40.32 (5.43) 38.93 (5.39) 38.69 (5.37) 38.45 (5.36) 36.92 (5.33) 37.26 (5.33) Duo N.A 0.9 35.92 (5.41) 32.98 (5.40) 31.49 (5.36) 30.32 (5.31) 30.06 (5.29) 30.00 (5.28) 28.90 (5.25) 29.19 (5.25) MDLM N.A 1.0 68.61 (5.48) 55.26 (5.45) 49.51 (5.44) 46.13 (5.42) 45.61 (5.42) 44.87 (5.42) 44.53 (5.41) 44.38 (5.42) MDLM N.A 0.95 46.07 (5.37) 36.55 (5.33) 32.91 (5.31) 30.96 (5.30) 30.26 (5.29) 29.73 (5.29) 29.54 (5.28) 29.53 (5.28) MDLM N.A 0.9 34.85 (5.26) 28.21 (5.23) 25.27 (5.21) 24.31 (5.19) 23.25 (5.18) 22.75 (5.17) 22.73 (5.17) 22.46 (5.16) Cap Schedule Duo 0.005 1.0 66.13 (5.54) 58.49 (5.52) 53.61 (5.48) 47.85 (5.42) 41.59 (5.39) 34.05 (5.34) 25.67 (5.22) 19.25 (5.11) Duo 0.01 1.0 64.22 (5.53) 55.84 (5.51) 49.90 (5.48) 40.95 (5.39) 33.90 (5.34) 26.29 (5.24) 19.34 (5.11) 14.31 (4.96) Duo 0.005 0.95 43.68 (5.47) 38.77 (5.45) 35.55 (5.40) 31.36 (5.33) 26.74 (5.28) 21.84 (5.22) 16.22 (5.08) 12.00 (4.94) Duo 0.01 0.95 42.34 (5.46) 37.14 (5.44) 32.39 (5.38) 27.25 (5.30) 21.84 (5.22) 16.74 (5.10) 11.70 (4.92) 8.68 (4.72) Duo 0.005 0.9 34.80 (5.40) 30.95 (5.38) 28.15 (5.34) 24.47 (5.26) 21.25 (5.18) 17.02 (5.12) 12.86 (4.99) 9.48 (4.81) Duo 0.01 0.9 33.91 (5.40) 29.27 (5.37) 25.28 (5.31) 21.40 (5.21) 17.36 (5.13) 13.22 (5.00) 9.55 (4.82) 6.92 (4.56) MDLM 0.005 1.0 67.27 (5.48) 52.34 (5.45) 44.38 (5.42) 38.14 (5.40) 32.35 (5.37) 26.37 (5.34) 20.64 (5.27) 15.80 (5.19) MDLM 0.01 1.0 65.29 (5.47) 49.78 (5.44) 41.29 (5.40) 33.39 (5.38) 27.16 (5.34) 21.04 (5.28) 16.13 (5.19) 12.16 (5.08) MDLM 0.005 0.95 44.71 (5.36) 34.56 (5.32) 29.42 (5.30) 25.28 (5.27) 21.55 (5.23) 17.39 (5.18) 13.63 (5.09) 10.47 (4.98) MDLM 0.01 0.95 43.20 (5.36) 32.84 (5.32) 26.90 (5.29) 22.19 (5.24) 17.80 (5.19) 13.93 (5.11) 10.61 (4.98) 7.68 (4.76) MDLM 0.005 0.9 33.81 (5.26) 26.71 (5.22) 22.81 (5.19) 19.65 (5.16) 16.67 (5.11) 13.79 (5.06) 10.74 (4.94) 8.10 (4.78) MDLM 0.01 0.9 32.94 (5.25) 25.51 (5.22) 20.89 (5.18) 17.19 (5.13) 13.91 (5.05) 10.91 (4.95) 8.15 (4.78) 5.93 (4.54) Rescale Schedule Duo 0.01 1.0 68.33 (5.54) 62.77 (5.53) 59.65 (5.50) 57.89 (5.46) 57.43 (5.45) 56.18 (5.44) 53.13 (5.42) 51.93 (5.41) Duo 0.02 1.0 68.18 (5.54) 62.24 (5.53) 59.07 (5.50) 56.96 (5.46) 55.73 (5.44) 53.31 (5.43) 48.20 (5.40) 44.51 (5.38) Duo 0.01 0.95 45.04 (5.47) 41.74 (5.46) 39.99 (5.43) 38.80 (5.38) 38.10 (5.37) 37.51 (5.36) 35.43 (5.33) 34.71 (5.32) Duo 0.02 0.95 44.89 (5.47) 41.33 (5.46) 39.81 (5.43) 38.09 (5.38) 36.79 (5.36) 35.47 (5.35) 31.97 (5.31) 29.25 (5.28) Duo 0.01 0.9 35.91 (5.41) 33.05 (5.40) 31.55 (5.36) 30.39 (5.31) 29.94 (5.29) 29.70 (5.28) 27.73 (5.25) 27.43 (5.24) Duo 0.02 0.9 35.81 (5.41) 32.77 (5.40) 31.17 (5.36) 29.70 (5.30) 28.70 (5.28) 27.70 (5.26) 25.31 (5.22) 22.83 (5.19) MDLM 0.01 1.0 68.66 (5.48) 55.16 (5.45) 49.71 (5.43) 45.88 (5.42) 45.11 (5.42) 43.79 (5.41) 42.55 (5.40) 40.90 (5.40) MDLM 0.02 1.0 68.73 (5.48) 54.85 (5.45) 48.12 (5.43) 45.35 (5.42) 44.10 (5.42) 41.48 (5.41) 38.76 (5.39) 34.66 (5.38) MDLM 0.01 0.95 46.01 (5.37) 36.58 (5.33) 32.80 (5.31) 30.65 (5.30) 29.92 (5.29) 29.18 (5.28) 28.34 (5.28) 27.38 (5.27) MDLM 0.02 0.95 45.92 (5.37) 36.45 (5.33) 32.49 (5.31) 30.25 (5.29) 29.01 (5.28) 27.68 (5.28) 25.75 (5.26) 22.95 (5.24) MDLM 0.01 0.9 34.83 (5.26) 28.15 (5.23) 25.24 (5.21) 23.73 (5.19) 23.03 (5.18) 22.36 (5.17) 21.75 (5.17) 20.93 (5.15) MDLM 0.02 0.9 34.83 (5.26) 28.17 (5.23) 24.97 (5.21) 23.34 (5.19) 22.34 (5.17) 21.33 (5.17) 19.88 (5.15) 17.75 (5.12) Loop Schedule Duo 0.01 1.0 80.39 (5.55) 61.64 (5.54) 52.51 (5.52) 47.30 (5.48) 40.27 (5.44) 34.27 (5.40) 27.28 (5.32) 21.97 (5.26) Duo 0.02 1.0 75.97 (5.55) 57.36 (5.53) 47.47 (5.52) 41.33 (5.47) 34.18 (5.41) 28.72 (5.36) 22.16 (5.26) 17.67 (5.18) Duo 0.01 0.95 51.76 (5.48) 40.91 (5.47) 34.68 (5.44) 30.83 (5.39) 25.86 (5.34) 21.31 (5.27) 17.15 (5.18) 13.69 (5.10) Duo 0.02 0.95 48.78 (5.48) 37.61 (5.46) 30.95 (5.43) 26.55 (5.36) 21.60 (5.30) 17.64 (5.22) 13.84 (5.11) 11.15 (5.02) Duo 0.01 0.9 41.15 (5.42) 32.51 (5.40) 27.96 (5.38) 24.49 (5.32) 20.52 (5.25) 17.17 (5.19) 13.90 (5.10) 11.44 (5.02) Duo 0.02 0.9 38.73 (5.42) 30.04 (5.40) 24.99 (5.37) 21.24 (5.29) 17.40 (5.21) 14.25 (5.13) 11.51 (5.02) 9.51 (4.94) MDLM 0.01 1.0 99.76 (5.51) 62.76 (5.48) 47.50 (5.45) 39.07 (5.43) 32.85 (5.41) 28.01 (5.38) 23.18 (5.34) 19.32 (5.29) MDLM 0.02 1.0 93.99 (5.51) 58.00 (5.48) 43.00 (5.45) 33.84 (5.42) 28.60 (5.39) 24.13 (5.36) 19.81 (5.30) 16.32 (5.24) MDLM 0.01 0.95 65.09 (5.40) 41.85 (5.37) 31.76 (5.33) 26.11 (5.30) 22.21 (5.28) 19.19 (5.24) 16.12 (5.20) 13.59 (5.15) MDLM 0.02 0.95 61.24 (5.40) 38.68 (5.36) 28.92 (5.33) 23.21 (5.29) 19.45 (5.26) 16.60 (5.21) 13.84 (5.16) 11.73 (5.09) MDLM 0.01 0.9 48.86 (5.29) 32.03 (5.26) 24.51 (5.23) 20.56 (5.20) 17.79 (5.18) 15.42 (5.14) 13.19 (5.09) 11.29 (5.04) MDLM 0.02 0.9 46.12 (5.29) 29.77 (5.27) 22.52 (5.22) 18.46 (5.19) 15.86 (5.16) 13.57 (5.11) 11.54 (5.05) 9.85 (4.98)