Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models

Kevin Rojas, Ye He, Chieh-Hsin Lai, Yuta Takida, Yuki Mitsufuji, Molei Tao · 2025

Read Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models by Kevin Rojas, Ye He, Chieh-Hsin Lai, Yuta Takida, Yuki Mitsufuji, Molei Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08965.txt.

Introduction

Kevin Rojas and co-authors fix a popular steering trick for masked diffusion models. These models build text or images by unmasking pieces step by step. Guidance nudges them toward a requested class, but the usual form backfires early, when most pieces are still hidden. A toy model analysis shows why: strong early guidance warps the balance of unmasking.

Late guidance helps, early guidance hurts. Their remedy smooths the path between the blank start and the data, and it takes a one line code change. Tests on image and text generation confirm cleaner samples. The work ties a practical fix to exact low dimensional math.

Catalog record

CREATORKevin Rojas, Ye He, Chieh-Hsin Lai, Yuta Takida, Yuki Mitsufuji, Molei Tao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models Kevin Rojas, Ye He, Chieh-Hsin Lai, Yuta Takida, Yuki Mitsufuji, Molei Tao Abstract Classifier-Free Guidance (CFG) is a widely used technique for conditional generation and improving sample quality in continuous diffusion models, and recent works have extended it to discrete diffusion. This paper theoretically analyzes CFG in the context of masked discrete diffusion, focusing on the role of guidance schedules. Our analysis shows that high guidance early in sampling (when inputs are heavily masked) harms generation quality, while late-stage guidance has a larger effect. These findings provide a theoretical explanation for empirical observations in recent studies on guidance schedules. The analysis also reveals an imperfection of the current CFG implementations. These implementations can unintentionally cause imbalanced transitions, such as unmasking too rapidly during the early stages of generation, which degrades the quality of the resulting samples. To address this, we draw insight from the analysis and propose a novel classifier-free guidance mechanism empirically applicable to any discrete diffusion. Intuitively, our method smoothens the transport between the data distribution and the initial (masked/uniform) distribution, which results in improved sample quality. Remarkably, our method is achievable via a simple one-line code change. The efficacy of our method is empirically demonstrated with experiments on ImageNet (masked discrete diffusion) and QM9 (uniform discrete diffusion). Figure 1 : We proposed an improved guidance mechanism through column normalization. Our method produces sharper images while being more stable to the guidance strength. Notably, it requires only a minor code modification. 1 Introduction Continuous-state diffusion models ( Ho et al. 2020 , Song et al. ) have proven effective in both unconditional and conditional generation tasks, such as generating data from natural language prompts. Prominent examples include text-to-image and text-to-video models like Stable Diffusion, Sora, and others ( Rombach et al. 2022 , Esser et al. 2024 , Brooks et al. 2024 ) . More recently, progress in discrete diffusion modeling ( Campbell et al. 2022 , Lou et al. 2023 , Huang et al. 2023 , Gruver et al. 2023 , Ou et al. 2024 , Shi et al. 2024 , Sahoo et al. 2024 ) has extended the applicability of diffusion-based generation to new domains, including molecular design, protein synthesis, and languages. Despite their success, these models often produce outputs that lack fine detail or strong alignment with conditioning inputs (e.g., text prompts). A widely adopted technique to address this issue is classifier-free guidance (CFG) ( Ho and Salimans 2021 ) , which improves fidelity but typically at the cost of reduced sample diversity ( Karras et al. 2024 ) . A growing body of work has sought to understand the theoretical foundations of CFG in diffusion models ( Chidambaram et al. 2024 , Pavasovic et al. 2025 , Bradley and Nakkiran 2024 , Ye et al. 2025 ) , while others have developed improved guidance algorithms ( Karras et al. 2024 , Li et al. 2024 ) . Classifier-free guidance has also been adapted to discrete diffusion models ( Nisonoff et al. 2024 , Schiff et al. 2024 ) , yielding promising empirical gains. Among these improvements, dynamic guidance schedules—where guidance strength varies over the generation trajectory—have shown especially effective. Strategies such as guidance intervals ( Kynkäänniemi et al. 2024 ) and gradually increasing schedules ( Xi et al. 2024 ) can significantly enhance sample quality and are increasingly adopted in practice ( Hoogeboom et al. 2024 , Yu et al. 2024 , Karras et al. 2024 ) . However, such scheduling techniques remain exclusive to the continuous setting. While recent adaptations of CFG to discrete diffusion have improved empirical performance, defining and optimizing effective guidance strategies in discrete spaces remains a fundamentally challenging and open research problem. In our work we aim to better understand the mechanisms by which guidance affects the sampling process in discrete diffusion. Specifically, we aim to answer the following questions: • How does the guidance schedule affect the distribution of the generated samples? • Is it possible to characterize properties of good guidance schedules? To do so, we start by deriving explicit formulas for the sampled distribution under varying guidance schedules in 1 and 2 dimensions. We use these expressions to study the effect of the guidance schedule. Our work leads to the following results: • The current guidance mechanisms suffer from a flaw that complicates the simulation. • To address the flaw, we propose a novel classifier-free guidance mechanism based on a simple yet principled column normalization of the rate matrix. This change is theoretically justified, easy to implement (pseudocode in Sec. 1 ), and comparing favorably to existing approaches. • We characterize good guidance schedules for discrete diffusion. ⬇ def normalized_guidance_euler_transition ( x , c , t , dt , w ): uncond = model ( x , cond = None ) cond = model ( x , cond = c ) logits = w * cond + (1 - w ) * uncond p_theta = logits . softmax ( dim =-1) s , s_bar = sigma ( t ), sigma_bar ( t ) change = dt * s * (1 - exp (- s_bar )) return sample ( delta ( x ) + change * p_theta ) Our guidance in the special case of masked diffusion using Euler transitions. Our method is a simple one line change. ⬇ def other_guidance_euler_transition ( x , c , t , dt , w ): uncond = model ( x , cond = None ) cond = model ( x , cond = c ) logits = w * cond + (1 - w ) * uncond p_theta = logits . exp () s , s_bar = sigma ( t ), sigma_bar ( t ) change = dt * s * (1 - exp (- s_bar )) return sample ( delta ( x ) + change * p_theta ) Unlocking/Simple guidance for the special case of masked diffusion using Euler transitions. 2 Preliminaries This paper considers a vocabulary of size M and state space S=\{1,2,\dots,M\}^{d} , with each element being a sequence of tokens. The number of tokens d will also be referred to as the dimension. Each probability distribution on S is represented as a vector in \mathbb{R}^{M^{d}} whose entries sum to one. 2.1 Discrete Diffusion Given an initial distribution p\in\mathbb{R}^{M^{d}} , discrete diffusion is defined by considering a continuous time Markov chain (CTMC) given by: \displaystyle\frac{\mathrm{d}p_{t}}{\mathrm{d}t}=R_{t}p_{t},\quad p_{0}=p. (1) Here R_{t}\in\mathbb{R}^{{M^{d}}\times{M^{d}}} is known as the rate matrix, and it is such that when t\to\infty , p_{t} converges to a simple distribution. Additionally, R_{t} must satisfy that its non-diagonal entries are non-negative and each column must add up to zero. The time reversal of this process can be found by considering a different CTMC given by: \displaystyle\frac{\mathrm{d}p_{T-t}}{\mathrm{d}t}=\overline{R}_{T-t}p_{T-t}. (2) This process is considered as the time reversal since it has the same law as ( 1 ) for all values of t . The reverse transition matrix can be found through the following identities: \displaystyle\overline{R}_{t}(y,x)=R_{t}(x,y)\cdot\frac{p_{t}(y)}{p_{t}(x)},\qquad\overline{R}_{t}(x,x)=-\sum_{y\neq x}\overline{R}_{t}(y,x). (3) The ratios \frac{p_{t}(y)}{p_{t}(x)} are called the score. Given this matrix, it is possible to generate new samples using different sampling strategies like Euler schemes, \tau -leaping ( Lou et al. 2023 ) or higher order

Read the complete text