Stage-wise Dynamics of Classifier-Free Guidance in Diffusion Models
Cheng Jin, Qitan Shi & Yuantao Gu · 2025
Read Stage-wise Dynamics of Classifier-Free Guidance in Diffusion Models by Cheng Jin, Qitan Shi & Yuantao Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22007.txt.
Introduction
Cheng Jin, Qitan Shi, and Yuantao Gu analyze a knob that nearly every image generator exposes: classifier-free guidance. Turning it up makes outputs follow the prompt more closely, but the cost in variety was poorly understood. Earlier studies mostly used simple prompts with one mode of answer. Their 2025 paper, accepted at ICLR 2026, studies guidance when the prompt allows many distinct modes.
The authors show sampling unfolds in three stages. First, guidance pulls samples toward the weighted average of the modes, adding bias and growth. Next, in the separation stage, that inherited bias quietly suppresses the weaker modes, cutting global diversity. Finally, guidance tightens each surviving mode, trimming fine detail. Experiments back the stages: strong guidance early kills broad diversity, while strong guidance late kills detail. The theory points to a schedule that varies guidance over time, which improved both quality and diversity in tests.
Catalog record
| CREATOR | Cheng Jin, Qitan Shi & Yuantao Gu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22007
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stage-wise Dynamics of Classifier-Free Guidance in Diffusion Models Cheng Jin, Qitan Shi & Yuantao Gu Abstract Classifier-Free Guidance (CFG) is widely used to improve conditional fidelity in diffusion models, but its impact on sampling dynamics remains poorly understood. Prior studies, often restricted to unimodal conditional distributions or simplified cases, provide only a partial picture. We analyze CFG under multimodal conditionals and show that the sampling process unfolds in three successive stages. In the Direction Shift stage, guidance accelerates movement toward the weighted mean, introducing initialization bias and norm growth. In the Mode Separation stage, local dynamics remain largely neutral, but the inherited bias suppresses weaker modes, reducing global diversity. In the Concentration stage, guidance amplifies within-mode contraction, diminishing fine-grained variability. This unified view explains a widely observed phenomenon: stronger guidance improves semantic alignment but inevitably reduces diversity. Experiments support these predictions, showing that early strong guidance erodes global diversity, while late strong guidance suppresses fine-grained variation. Moreover, our theory naturally suggests a time-varying guidance schedule, and empirical results confirm that it consistently improves both quality and diversity. 1 Introduction Diffusion models have driven remarkable progress in generative modeling, achieving state-of-the-art results across images, video, audio, and multimodal domains ( Rombach et al. 2022 ; Podell et al. 2023 ; Cao et al. 2024 ; Zhang et al. 2025 ) . A central requirement in practice is conditional generation, where outputs must follow prompts, class labels, or other structured signals. Among existing strategies, Classifier-Free Guidance (CFG) ( Ho & Salimans 2021 ) has become the de facto standard due to its simplicity and effectiveness, powering nearly all large-scale diffusion pipelines. Numerous variants further extend its influence ( Jin et al. 2025 ; Gao et al. 2025 ; Malarz et al. 2025 ; Chung et al. 2025 ; Sadat et al. 2025 ; Castillo et al. 2025 ) . Yet despite its widespread adoption, the theoretical underpinnings of CFG remain poorly understood. While recent work has advanced the theory of diffusion sampling ( Benton et al. 2023 ; Chen et al. 2023 ; Cai & Li 2025 ; Li et al. 2025a ) , the analysis of CFG itself is still in its infancy, hindered by its heuristic formulation. Prior studies fall into two categories. The first assumes a unimodal conditional distribution—typically a single Gaussian—yielding clean derivations but overlooking the multimodal nature of real-world tasks ( Chidambaram et al. 2024 ; Wu et al. 2024 ; Bradley & Nakkiran 2024 ; Xia et al. 2024 ; Jin et al. 2025 ; Pavasovic et al. 2025 ; Li et al. 2025b ) . The second relaxes these assumptions but imposes only weak conditions, producing broad qualitative insights without sharp predictions ( Li & Jiao 2025 ) . Consequently, several well-documented empirical phenomena remain theoretically elusive, most notably the collapse of diversity under large guidance weights ( Ho & Salimans 2021 ) . Explaining this diversity loss is both a key theoretical challenge and a practical necessity for improving conditional generation. In this work, we move beyond the restrictive unimodality assumption by modeling conditional distributions as Gaussian mixtures. This perspective reveals that guided sampling follows a natural three-stage structure . In the Direction Shift stage (early, high-noise regime), trajectories are drawn toward the class-weighted mean, and guidance amplifies this attraction, inducing initialization bias and norm inflation. In the Mode Separation stage (intermediate regime), the dynamics remain locally neutral, but the inherited bias suppresses weaker modes, reducing global diversity. In the Concentration stage (late regime), contraction within modes is intensified by strong guidance, suppressing local variability and fine-grained diversity. We empirically validate these predictions on state-of-the-art diffusion models rather than toy settings. To test our framework, we first compare early-high, late-high, and all-high schedules, confirming that strong early guidance reduces global diversity. We then probe the late stage by initializing trajectories with identical noise and perturbations, finding that large late weights enforce excessive similarity in fine details and diminish local diversity. Beyond these validation studies, we further conduct an additional experiment with a time-varying guidance schedule. The results show that adjusting guidance strength over time can yield improved overall outcomes—a practical insight that, while secondary to our main theoretical contribution, highlights the broader value of our analysis. Our implementation is publicly available at https://github.com/sqt24/tvcfg . Contributions. This work makes three contributions: (i) We provide the first theoretical framework for analyzing CFG under multimodal conditional distributions, revealing a natural three-stage structure of the sampling dynamics. (ii) We empirically validate the central predictions of this framework: strong early guidance reduces global diversity, while strong late guidance suppresses local variability. (iii) As a byproduct of our analysis, we observe that varying guidance strength over time can improve the quality–diversity trade-off. Together, these results unify theoretical and empirical perspectives and establish a foundation for the principled design of guided diffusion samplers. 2 Preliminary 2.1 Diffusion Model Diffusion models generate data by inverting a forward noising process that progressively transforms samples from the target distribution p_{\text{data}}({\bm{x}}) into a simple Gaussian reference. The forward corruption can be written as {\rm d}{\bm{x}}_{t}=-\alpha(t){\bm{x}}_{t}{\rm d}t+\sqrt{2\beta(t)}{\rm d}{\bm{w}}_{t},\quad{\bm{x}}_{0}\sim p_{\text{data}},\quad t\in[0,T], (1) where \alpha(t) and \beta(t) control the drift and diffusion, and {\bm{w}}_{t} is a Wiener process. As t grows, the distribution p_{t} converges toward an isotropic Gaussian, ensuring that sampling can begin from a tractable prior. By time-reversal theory ( Anderson 1982 ) , sampling is performed through the reverse SDE {\rm d}{\bm{x}}_{t}=\big[-\alpha(t){\bm{x}}_{t}-2\beta(t)\nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t})\big]{\rm d}t+\sqrt{2\beta(t)}{\rm d}\bar{{\bm{w}}}_{t}, (2) where \nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}) is the score function , representing the gradient of the log-density, and \bar{{\bm{w}}}_{t} is a backward Wiener process. Intuitively, the score term provides a restoring drift that counteracts noise, gradually steering trajectories back toward high-density regions of the data distribution. Modern implementations typically adopt the equivalent probability flow ODE ( Song et al. 2021 ) , which eliminates randomness while preserving the same marginal distributions: \frac{{\rm d}{\bm{x}}_{t}}{{\rm d}t}=-\alpha(t){\bm{x}}_{t}-\beta(t)\nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}). (3) Its deterministic nature allows the use of high-order ODE solvers ( Lu et al. 2022 ; Zhao et al. 2023 ) , enabling efficient generation with few function evaluations and making theoretical analysis more tractable. This ODE view is particularly important for understanding how guidance mechanisms reshape the underlying dynamics. Conditional generation. When conditioning on external information y (e.g., a class label or a text prompt), one simply replaces the unconditional score with the conditional score: \frac{{\rm d}{\bm{x}}_{t}}{{\rm d}t}=-\alpha(t){\bm{x}}_{t}-\beta(t)\nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}\mid y). (4) This conditional probability flow ODE produces samples consistent with y while retaining the computational benefits of the ODE formulation. However, in practice the learned conditional score may be weak or under-confident, leading to poor semantic alignment with the conditioning signal. 2.2 Classifier-Free Guidance While the conditional ODE equation 4 incorporates side information y , its influence in practice is often insufficient. Classifier-Free Guidance (CFG) addresses this issue by extrapolating between the unconditional score \nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}) and the conditional score \nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}\mid y) predicted by the same model: \hat{\bm{s}}_{t}({\bm{x}}_{t};y,\omega)=(1-\omega)\,\nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t})+\omega\,\nabla_{{\bm{x}}_{t}}\log p_{t}({\bm{x}}_{t}\mid y),\quad\omega>1. (5) Here the guidance scale \omega controls the trade-off: \omega=1 reduces to the plain conditional model, whereas larger values enforce stronger alignment with y . While this interpolation is simple and effective, the resulting score no longer corresponds to any valid probabilistic model. Plugging the guided score equation 5 into the probability flow