What Exactly Does Guidance Do in Masked Discrete Diffusion Models
Ye He, Kevin Rojas, Molei Tao · 2025
Read What Exactly Does Guidance Do in Masked Discrete Diffusion Models by Ye He, Kevin Rojas, Molei Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10971.txt.
Introduction
Ye He, Kevin Rojas, and Molei Tao ask in this 2025 paper a precise question: what does guidance actually do inside a masked discrete diffusion model? These models build text and other discrete data by unmasking step by step, and classifier-free guidance steers the path. Assuming no scoring or stepping error, the authors solve the guided process exactly.
Guidance, they show, fattens the regions unique to the target class and thins the regions shared with other classes. The strength knob sets how sharp that split grows, and shapes the spread of samples in ways that differ between one and two dimensions. For strong guidance, a key distance shrinks at a double-exponential rate. Experiments back the geometry.
Catalog record
| CREATOR | Ye He, Kevin Rojas, Molei Tao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10971
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Exactly Does Guidance Do in Masked Discrete Diffusion Models Ye He, Kevin Rojas, Molei Tao Abstract We study masked discrete diffusion models with classifier-free guidance (CFG). Assuming no score error nor discretization error, we derive an explicit solution to the guided reverse dynamics, so that how guidance influences the sampling behavior can be precisely characterized. When the full data distribution is a mixture over classes and the goal is to sample from a specific class, guidance amplifies class-specific regions while suppresses regions shared with other classes. This effect depends on the guidance strength w w and induces distinct covariance structures in the sampled distribution. Notably, we observe quantitatively different behaviors in 1 1 D and 2 2 D. We also show that for large w w , the decay rate of the total variation ( TV \mathrm{TV} ) along the reverse dynamics is double-exponential in w w for both 1 1 D and 2 2 D. These findings highlight the role of guidance, not just in shaping the output distribution, but also in controlling the dynamics of the sampling trajectory. Our theoretical analysis is supported by experiments that illustrate the geometric effects of guidance and its impact on convergence. 1 Introduction Diffusion models have become an influential tool for generative modeling, offering a flexible framework that performs well across a range of data types including images, audio, and text ( Dhariwal and Nichol 2021 , Kong et al. 2021 , Li et al. 2022 , Ho et al. 2022 ) . Originally formulated in continuous state spaces ( Ho et al. 2020 , Song et al. 2021 ) , these models simulate a forward noising process—typically modeled by a stochastic differential equation and learn a reverse process to denoise and reconstruct the original data. More recently, the diffusion framework has been extended to discrete state spaces ( Campbell et al. 2022 , Lou et al. 2023 ) , where the forward process is defined via a continuous-time Markov chain over a finite state space. This has enabled generative modeling for discrete domains such as language modeling, molecule generation, and protein design ( Lou et al. 2023 , Nie et al. , Huang et al. 2023 , Gruver et al. 2023 ) . A key innovation that has enhanced the performance and flexibility of diffusion models is guidance, which introduces an auxiliary parameter to steer the reverse process toward desired outputs. In the continuous setting, classifier guidance ( Dhariwal and Nichol 2021 ) and classifier-free guidance ( Ho and Salimans 2021 , Nichol et al. 2022 ) are widely used for conditional generation based on class labels or text prompts, significantly improving sample quality and alignment with conditioning signals. This technique has been critical to the success of models such as GLIDE ( Nichol et al. 2022 ) and Imagen ( Saharia et al. 2022 ) . Theoretical analyses of guided diffusion models in continuous state spaces have examined how guidance modifies the reverse dynamics, most of which focus on simple settings such as low-dimensional and mixture of Gaussian models ( Bradley and Nakkiran 2024 , Wu et al. 2024 , Chidambaram et al. 2024 ) . Classifier-free guidance (CFG) has also been recently introduced to discrete diffusion models, for applications such as text generation and controlled molecule design ( Huang et al. 2023 , Nisonoff et al. 2024 ) . On the surface, the methodology appears to be very similar to the continuous diffusion case, but are guidance mechanism in continuous and discrete cases really similar? A closer look can actually reveal inherent differences: the lack of gradients and smooth geometry requires alternative strategies such as modifying transition probabilities or reweighting proposal distributions ( Nisonoff et al. 2024 , Sahoo et al. 2024 ) . While empirical results demonstrate that guidance improves sample quality and controllability ( Sahoo et al. 2024 , Xiong et al. 2025 ) , the theoretical understanding of how guidance affects the dynamics of the diffusion process in discrete state spaces remains limited. In this paper, we provide a rigorous and quantitative framework for analyzing the effects of CFG on the discrete diffusion generative process ( Nisonoff et al. 2024 ) . We focus on masked discrete diffusion models ( Campbell et al. 2022 , Shi et al. 2024 , Sahoo et al. 2024 , Ou et al. 2024 ) —a common subclass of discrete diffusion models. Assuming there is no errors from the score approximation and the numerical integration, we address the following two questions in the low-dimensional setting ( 1 D and 2 D). Q 1 . How does guidance affect the distribution of the generated samples? Towards this question, we derive explicit formulas for the sampled distributions. Assuming the full distribution is a mixture of different class distributions, i.e., the data distribution p satisfies Assumption 1.1 , the sampled distribution for a single class amplifies the probability mass in the area that is only supported in that class, and reduces the probability mass in the area that overlaps with other classes, eventually to zero as guidance strength w goes to \infty . The strength of the amplification/reduction depends on w . The covariance structure of the sampled distribution varies for 1 D and 2 D. Q 2 . How does guidance affect the rate of convergence of the reverse sampling dynamics? To answer this question, we quantify \mathrm{TV} between the distribution along the reverse sampling dynamics and the sampled distribution. Our results reflect that for both 1 D and 2 D, the decay rates of \mathrm{TV} along the reverse sampling dynamics exhibit a double-exponential dependency on the guidance strength w for w\gg 1 . By characterizing the above influence of guidance on sampling trajectories, distributional shifts and convergence rates, our work bridges the gap between practical heuristics and theoretical understanding in guided discrete diffusion generation. Assumption 1.1 . Let \{z_{k}\}_{k=1}^{M} be the set of M labels, each of which is associated with a class distribution p(\cdot|z_{k}) supported on \mathcal{X}_{k}\subsetneq S . The full data distribution p is a mixture of distributions \{p(\cdot|z_{k})\}_{k=1}^{M} with weights \{a_{k}\}_{k=1}^{M} , i.e., p(\cdot)=\sum_{k=1}^{M}a_{k}p(\cdot|z_{k}) . Paper Organization. The remainder of the paper is organized as follows. Section 2 introduces preliminaries on diffusion models relevant to our analysis. Section 3 quantifies the density evolution in masked discrete diffusion without guidance, which serves as a foundation for the guided case. Section 4 presents our theoretical analysis of the guided diffusion process, and Section 5 provides numerical examples supporting our findings. Conclusions are discussed in Section 6 , and additional related work is presented in Appendix A . 2 Preliminaries 2.1 Notations In this paper, for any x\in\mathbb{R}^{D} and A\subset\{1,2,\cdots,D\} , we use x_{A}\in\mathbb{R}^{|A|} to denote the vector by preserving dimensions whose indices are in A . \setminus i is used to denote \{1,2,\cdots,N\}\setminus\{i\} . For any distribution p , p(x_{A}) denotes the A -marginal density evaluated at x_{A} . For functions f,g , we use f(w)\sim g(w) to represent \lim_{w\to\infty}{f(w)}/{g(w)}=1 and f(w)=\Theta(g(w)) to indicate that c_{1}g(w)\leq f(w)\leq c_{2}g(w) for some c_{1},c_{2},w_{0}>0 and all w>w_{0} . 2.2 Discrete Diffusion Models We consider the probability state space S=\{1,2,\cdots,N\}^{D} . The data distribution p is represented as a vector in \mathbb{R}^{N^{D}} that sums