TCFG: Tangential Damping Classifier-free Guidance
Mingi Kwon, Shin seong Kim, Jaeseok Jeong, Yi Ting Hsiao, Youngjung Uh · 2025
Read TCFG: Tangential Damping Classifier-free Guidance by Mingi Kwon, Shin seong Kim, Jaeseok Jeong, Yi Ting Hsiao, Youngjung Uh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18137.txt.
Introduction
Mingi Kwon, Shin seong Kim, Jaeseok Jeong, Yi Ting Hsiao, and Youngjung Uh improve how AI image tools follow text prompts in a 2025 paper. The tools are diffusion models that build images from noise step by step. A method called classifier-free guidance steers them toward the prompt. But part of that steering can push the image off course.
The team adds a step they call tangential damping to the guidance. It filters parts of the guide signal with a math split called singular value decomposition. The filter lines up the plain guide with the prompt-based guide. Image quality rose with almost no extra compute cost. The work also sheds new light on how guide signals act inside diffusion models.
Catalog record
| CREATOR | Mingi Kwon, Shin seong Kim, Jaeseok Jeong, Yi Ting Hsiao, Youngjung Uh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TCFG: Tangential Damping Classifier-free Guidance Mingi Kwon, Shin seong Kim, Jaeseok Jeong, Yi Ting Hsiao, Youngjung Uh Abstract Diffusion models have achieved remarkable success in text-to-image synthesis, largely attributed to the use of classifier-free guidance (CFG), which enables high-quality, condition-aligned image generation. CFG combines the conditional score (e.g., text-conditioned) with the unconditional score to control the output. However, the unconditional score is in charge of estimating the transition between manifolds of adjacent timesteps from x t x_{t} to x t − 1 x_{t-1} , which may inadvertently interfere with the trajectory toward the specific condition. In this work, we introduce a novel approach that leverages a geometric perspective on the unconditional score to enhance CFG performance when conditional scores are available. Specifically, we propose a method that filters the singular vectors of both conditional and unconditional scores using singular value decomposition. This filtering process aligns the unconditional score with the conditional score, thereby refining the sampling trajectory to stay closer to the manifold. Our approach improves image quality with negligible additional computation. We provide deeper insights into the score function behavior in diffusion models and present a practical technique for achieving more accurate and contextually coherent image synthesis. 1 Introduction Diffusion models [ 12 , 31 ] have shown remarkable progress in image generation [ 19 , 30 , 27 ] . In particular, the emergence of classifier-free guidance [ 11 , 6 ] (CFG) has attracted significant attention because it allows us to provide desired guidance by leveraging the conditional estimated score directly within the diffusion model. Figure 1 : (a) Classifier-free guidance. When the unconditional score {\bm{s}}_{\theta}({\bm{z}}_{t}) and the conditional score {\bm{s}}_{\theta}({\bm{z}}_{t},y) are misaligned, the result of CFG tends to fall off the manifold. (b) Our proposed method reduces the misalignment between the unconditional score {\bm{s}}_{\theta}({\bm{z}}_{t}) and the conditional score {\bm{s}}_{\theta}({\bm{z}}_{t},y) , ensuring sampling aligns with the target manifold. The classifier-free guidance fundamentally computes the final score by combining the unconditional and conditional estimated scores. This approach ensures a generation that aligns well with the given condition. Additionally, using an appropriate guidance scale has been shown to enhance image quality across various tasks, further driving improvements in applications like text-to-image generation. Let us say the guided score \tilde{{\bm{s}}}_{\theta} as \tilde{{\bm{s}}}_{\theta}={\bm{s}}_{\theta}^{\text{uncond}}+\omega_{\text{scale}}({\bm{s}}_{\theta}^{\text{cond}}-{\bm{s}}_{\theta}^{\text{uncond}}) . In text-to-image models, the text condition ( {\bm{s}}_{\theta}^{\text{cond}} ) is randomly replaced with a null condition ( {\bm{s}}_{\theta}^{\text{uncond}} ) during training (e.g., with a probability p = 0.1), enabling the null condition to act as a general estimator for any sample. It means that {\bm{s}}_{\theta}^{\text{uncond}} is the score estimated from z_{t} to z_{t-1} for all samples in the sampling trajectory of diffusion models. The unconditional score for any sample has certainly enabled the successful use of classifier-free guidance. However, we argue that there can be a misalignment between the unconditional and conditional estimated scores (See Eq. 2 in Sec. 2 ), which hinders the approximation toward the manifold by the given condition. Fig. 1 (a) conceptually illustrates the potential issue that arises when the manifold of the unconditional score differs from that of the conditional score. In this paper, we show that this misalignment can be resolved with a simple algorithm, which significantly reduces the tendency of CFG to generate off-manifold samples, as illustrated in Fig. 1 (b). Our approach is based on the following insights. First, the score predicted by the diffusion model estimates the intrinsic dimension of the data manifold [ 32 ] . Additionally, this intrinsic dimension can be captured by the tangent space of the target manifold [ 9 , 3 ] . Instead of directly estimating the intrinsic dimension, we focus on utilizing the tangential component inherent in the unconditional score during classifier-free guidance. By reducing its misalignment with the conditional score, we enhance the alignment and ultimately improve the quality of the generated outputs. Specifically, we push the score \tilde{{\bm{s}}}_{\theta} toward the normal direction of the conditional manifold by eliminating the values of column vectors with small singular values using the orthogonal matrix V obtained through the singular value decomposition of the conditional and unconditional scores. In this paper, we propose a novel sampling method that leverages the unconditional score within CFG. To support our approach, we first lay out the theoretical foundation in section Sec. 2 and Sec. 3 , discussing the manifold hypothesis and its connection to diffusion models. In Sec. 4 , we provide a comprehensive explanation of our proposed method. This is followed by a detailed analysis using a toy example in Sec. 5 , and we demonstrate the practical applicability of our method on real-world text-to-image models in Sec. 6 . Our experiments show a significant improvement in the MS-COCO Fréchet Inception Distance (FID) across various models that utilize classifier-free guidance, e.g., diffusion models (Stable Diffusion v1.5 [ 26 ] and SDXL [ 23 ] ) and rectified flow (Stable Diffusion 3 [ 8 ] ). Additionally, our method improves DiT [ 21 ] FID on ImageNet. Notably, our method helps mitigate the overexposure bias problem, leading to resulting images that better align with the underlying data distribution, as supported by improved quantitative metrics. 2 Background Diffusion models Diffusion models learn the score that reverses the forward noising process. This forward process from the real data distribution p({\bm{x}}_{0}) to a latent distribution p({\bm{z}}_{1})\sim N(0,\sigma_{\text{max}}^{2}I) along timesteps t\in[0,1] is defined by a Gaussian kernel: {\bm{z}}_{t}={\bm{x}}_{0}+\sigma(t){\mathbf{\epsilon}} . The function \sigma(t) is a noise schedule where \sigma(0)=0 and \sigma(1)=\sigma_{\text{max}} , determining the amount of noise to be added at each timestep t to erase information from x . A generative process is represented as its reverse with a stochastic differential equation (SDE): \displaystyle\mathrm{d}{\bm{z}} \displaystyle=-\dot{\sigma}(t)\sigma(t)\nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})\,\mathrm{d}t \displaystyle\quad-\beta(t)\sigma(t)^{2}\nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})\,\mathrm{d}t+\sqrt{2\beta(t)}\sigma(t)\,\mathrm{d}\omega_{t}, where \mathrm{d}\omega_{t} is a standard Wiener process. Alternatively, it can be expressed as an ordinary differential equation: \mathrm{d}{\bm{z}}=-\dot{\sigma}(t)\sigma(t)\nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})\,\mathrm{d}t. Diffusion models approximate the score function \nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t}) with a neural network {\bm{s}}_{\theta}({\bm{z}}_{t},t) . They are trained to predict the clean data from the noisy {\bm{z}}_{t} . The trained model performs the reverse process using: \nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})\approx\frac{{\bm{s}}_{\theta}({\bm{z}}_{t},t)-{\bm{z}}_{t}}{\sigma(t)^{2}}. Classifier guidance (CG) and classifier-free guidance (CFG) For an arbitrary class label y , CG defines the class-conditional sampling distribution \tilde{p}_{\theta}({\bm{z}}_{t}\mid y) as: \tilde{p}_{\theta}({\bm{z}}_{t}\mid y)\propto p_{\theta}({\bm{z}}_{t}\mid y)\,p_{\theta}(y\mid{\bm{z}}_{t})^{\gamma}, where p_{\theta}(y\mid{\bm{z}}_{t}) is the classifier distribution and \gamma is a scaling parameter. [ 6 ] When \gamma>0 , it is known to reduce sample diversity but enhance quality. However, CG requires a classifier that can predict label y from the noisy {\bm{z}}_{t} . CFG proposes a method to sample from the conditional distribution by expressing the classifier distribution p_{\theta}(y\mid{\bm{z}}_{t}) in terms of the conditional distribution p_{\theta}({\bm{z}}_{t}\mid y) and the unconditional distribution p_{\theta}({\bm{z}}_{t}) : \tilde{p}_{\theta}({\bm{z}}_{t}\mid y)\propto p_{\theta}({\bm{z}}_{t}\mid y)^{1+\gamma}\,p_{\theta}({\bm{z}}_{t})^{-\gamma}. As a result, the final score \nabla_{{\bm{z}}_{t}}\log\tilde{p}_{\theta}({\bm{z}}_{t}\mid y) is approximated by: \displaystyle\nabla_{{\bm{z}}_{t}}\log\tilde{p}_{\theta}({\bm{z}}_{t}\mid y) \displaystyle=(1+\gamma)\,{\bm{s}}_{\theta}({\bm{z}}_{t},y)-\gamma\,{\bm{s}}_{\theta}({\bm{z}}_{t}) \displaystyle={\bm{s}}_{\theta}({\bm{z}}_{t})+\omega\,({\bm{s}}_{\theta}({\bm{z}}_{t},y)-{\bm{s}}_{\theta}({\bm{z}}_{t})), where \omega=1+\gamma . [ 11 ] In practice, both {\bm{s}}_{\theta}(x_{t},y) and {\bm{s}}_{\theta}({\bm{z}}_{t}) are approximated by a single neural network that is jointly trained to estimate both the conditional and unconditional scores. Text-to-image models use the null condition y_{\text{null}}=\varnothing as a class label to train {\bm{s}}_{\theta}({\bm{z}}_{t})\approx{\bm{s}}_{\theta}({\bm{z}}_{t},y_{\text{null}}) . This approach allows {\bm{s}}_{\theta}({\bm{z}}_{t},y)-{\bm{s}}_{\theta}({\bm{z}}_{t}) to provide guidance similar to the gradient of an implicit classifier. Hereafter, we will simply denote {\bm{s}}_{\theta}({\bm{z}}_{t},y_{\text{null}}) as {\bm{s}}_{\theta}({\bm{z}}_{t}) . Diffusion models and data manifold encoding The manifold hypothesis suggests that high-dimensional data lies on or near