Enhancing Diffusion Model Guidance through Calibration and Regularization

Seyed Alireza Javid, Amirhossein Bagheri, Nuria González-Prelcic · 2025

Read Enhancing Diffusion Model Guidance through Calibration and Regularization by Seyed Alireza Javid, Amirhossein Bagheri, Nuria González-Prelcic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05844.txt.

Introduction

Seyed Alireza Javid, Amirhossein Bagheri, and Nuria González-Prelcic fix a flaw in guided image making: early in cleanup, the guide is too sure of itself and its signal fades. Guided diffusion steers pictures toward a named class using a sorting model's slopes. When that model is too sure too early, the steering fades out.

Two fixes are offered. A light fine-tune based on a smooth scoring error makes the model's confidence honest, which lifted image quality scores. New sampling tricks then work with stock models and no retraining: tilted sampling, a spread-keeping rule, and a scheme based on gaps between spreads. On a standard image set their best guidance beat prior methods with no retraining of the base model.

Catalog record

CREATORSeyed Alireza Javid, Amirhossein Bagheri, Nuria González-Prelcic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Diffusion Model Guidance through Calibration and Regularization Seyed Alireza Javid, Amirhossein Bagheri, Nuria González-Prelcic Abstract Classifier-guided diffusion models have emerged as a powerful approach for conditional image generation, but they suffer from overconfident predictions during early denoising steps, causing the guidance gradient to vanish. This paper presents two complementary contributions to enhance classifier-guided diffusion models. First, we introduce a differentiable Smooth Expected Calibration Error (Smooth ECE) loss that improves classifier calibration with minimal fine-tuning, achieving approximately a 3% improvement in Fréchet Inception Distance (FID) scores. Second, we propose enhanced sampling guidance methods that operate on off-the-shelf classifiers without requiring retraining. Our approach includes: (1) tilted sampling that leverages batch-level information to control outlier influence, (2) adaptive entropy-regularized sampling to maintain diversity, and (3) a novel divergence-regularized sampling method that adds a class-aware, mode-covering correction that strengthens movement toward the target class while maintaining exploration. Theoretical analysis reveals that our methods effectively combine enhanced target direction guidance with controlled diversity exploration, mitigating gradient vanishing. Experimental results on ImageNet demonstrate that our best divergence-guided sampling achieves an FID of 2.13 while maintaining competitive precision and recall metrics. Our methods provide a practical solution for improving conditional generation quality without the computational overhead of classifier and diffusion model retraining. Code is available at: https://github.com/ajavid34/guided-info-diffusion . \workshoptitle Structured Probabilistic Inference & Generative Modeling workshop of NeurIPS 2025 1 Introduction Denoising diffusion probabilistic models (DDPMs) have achieved state-of-the-art results in unconditional image generation, producing high-quality and diverse images by progressively reversing a noising process 11 ; 27 ; 35 . These models are built on a solid probabilistic foundation, which allows for stable training and scalability across a wide range of datasets. A key strength of DDPMs is their adaptability: by incorporating external information like class labels or text embeddings, they can be extended to conditional image generation. Conditional image generation enables the creation of images that adhere to specific constraints, such as class labels or textual descriptions 31 ; 32 ; 22 . DDPMs have proven to be highly effective in this area, particularly for generating class-conditioned images of exceptional quality 11 ; 8 ; 27 . The theoretical underpinnings of diffusion models, similar to score-based generative models, allow for this conditional extension through Bayes’ theorem without the need for retraining the entire model 36 ; 38 ; 39 . One of the most effective techniques for conditional generation involves using an independent, noise-aware classifier to guide the reverse diffusion process 8 ; 34 ; 23 . This classifier directs the generation process toward the desired output by estimating the gradient of the log-probability of the target label with respect to the noisy input, all without requiring retraining of the generative model itself. Despite these advances, a critical limitation of classifier-guided diffusion models is that they suffer from overconfident predictions during early denoising steps, causing the guidance gradient to vanish 23 ; 42 . Our main contributions to this paper are as follows: • We propose a calibration-aware classifier finetuning loss based on the Smooth Expected Calibration Error (Smooth ECE) 26 . This loss improves calibration and achieves better downstream performance while introducing only minimal fine-tuning overhead. • We introduce an entropy-regularized guidance method that prevents premature overconfidence, promoting diversity during sampling while preserving class relevance. • We propose a divergence regularized guidance method, which enhances mode coverage and preserves fidelity to the target class distribution. • We design a tilted-loss-based batch-aware sampling strategy, which leverages information across generated samples to balance quality and diversity without additional complexity. We conduct experimental analysis on ImageNet 128\times 128 , demonstrating that our proposed methods outperform existing classifier-guided diffusion approaches in terms of Fréchet inception distance (FID) 10 , precision, and recall. 2 Classifier design As introduced by 23 , the ECE (details in Appendix C.1 ) has a direct connection to the FID. Motivated by this connection, we define a differentiable Huber-like calibration loss termed Smooth ECE , which operates over confidence bins and applies a smooth absolute error at the individual sample level. The loss is formulated as \mathcal{L}_{\mathrm{ECE}}=\frac{1}{n}\sum_{b=1}^{B}\sum_{i:\hat{p}^{(i)}\in\mathcal{B}_{b}}\sqrt{\left(\hat{p}^{(i)}-a^{(i)}\right)^{2}+\beta}. (1) where \hat{p}^{(i)}=\max_{y}p_{\phi}(y\mid x^{(i)}) is the predicted confidence for sample i , and a^{(i)}=\mathbb{I}[\hat{y}^{(i)}=y^{(i)}] is the correctness of the prediction, where \hat{y}^{(i)}=\arg\max_{c}p_{\phi}(c\mid x^{(i)}) . The term \beta>0 is a smoothing constant that ensures differentiability. The interval \mathcal{B}_{b}=\left(\frac{b-1}{B},\frac{b}{B}\right] defines the b -th confidence bin, and each sample contributes only to the bin corresponding to its confidence level. The loss is computed per sample and aggregated over all bins to form the final scalar value. The relationship between smooth ECE and Huber loss is explained in Appendix C.1 . Remark 1 . Our method differs from Meta-Calibration 5 by directly introducing smoothness into the ECE loss via a Huber-like function, avoiding the need for soft binning or differentiable ranking. Unlike their meta-learning framework, we apply our smooth ECE loss directly during training without outer-loop optimization. This results in a simpler, more efficient, and plug-and-play approach. 3 Sampling guidance We proposed a calibration-aware classifier design that leverages the ECE. However, fine-tuning classifiers can be costly and time-consuming. In what follows, we focus on off-the-shelf classifier-guidance methods that operate directly on existing checkpoints and require no additional fine-tuning. Since off-the-shelf classifiers are typically not robust to Gaussian noise and are not time-dependent 23 , we select \widehat{x}_{0}(t)=\left(\widehat{x}_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}\left(\widehat{x}_{t},t\right)\right)/\sqrt{\bar{\alpha}_{t}} as the classifier input. 3.1 Regularized sampling The sampling guidance from the classifier during the denoising steps of a diffusion model is helpful for increasing both the diversity and quality of the generated image 8 . However, a critical flaw emerges in this process. Even when an image is only partially generated and still lacks fine-grained features, a standard classifier can often classify it with excessively high confidence 42 . This causes the classifier’s predicted distribution for the noisy image to prematurely converge to a one-hot distribution. As a result, the conditional gradient guidance vanishes early in the process, and the conditional generation degrades into a less effective unconditional one for the remaining steps. To address this limitation, 42 proposed entropy-constrained training and scalar weight based entropy on the classifier guidance. While they show some improvement compared to the baseline, their scheme requires costly training of a classifier from scratch and cannot be generalized to off-the-shelf classifiers. As Figure 1 shows, the top 25% most confident samples maintain near-perfect confidence throughout most of the sampling process, demonstrating the overconfidence problem that leads to less effective guidance. While the average confidence increases more gradually, we can still see the overconfidence problem for most of the denoising process. This motivates our analysis of two general methods for more robust classifier guidance. Figure 1 : The visualization of the denoising sampling process. Overconfidence in classifier-guided diffusion sampling without regularization is mostly effective in the last steps where the probability is close to one. 3.1.1 Entropy-regularized sampling 23 demonstrated that the classifier guidance can be interpreted as the gradient of the joint E_{\tau_{1}}(x,y) and marginal energy E_{\tau_{2}}(x) difference as follows: \displaystyle\log p_{\tau_{1},\tau_{2}}(y\mid x) \displaystyle=\log\frac{\exp\left(\tau_{1}f_{y}(x)\right)}{\sum_{i=1}^{N}\exp\left(\tau_{2}f_{i}(x)\right)}=\tau_{1}f_{y}(x)-\log\sum_{i=1}^{N}\exp\left(\tau_{2}f_{i}(x)\right) (2) \displaystyle: \displaystyle=-E_{\tau_{1}}(x,y)+E_{\tau_{2}}(x), \displaystyle\nabla_{x}\log p_{\tau_{1},\tau_{2}}(y\mid x) \displaystyle=\nabla_{x}\left(\tau_{1}f_{y}(x)-\log\left(\sum_{i=1}^{N}\exp\left(\tau_{2}f_{i}(x)\right)\right)\right):=-\nabla_{x}E_{\tau_{1}}(x,y)+\nabla_{x}E_{\tau_{2}}(x), (3) where \tau_{1} and \tau_{2} represent joint and marginal temperatures, respectively. Based on the discussion in Section 3.1 , we rewrite the regularized guidance as \mathcal{S}_{\text{entropy}}(x,y):=\log p_{\tau_{1},\tau_{2}}(y|x)+\lambda_{t}H(p(\cdot|x)), (4) where H(p(\cdot|x))=-\sum_{i=1}^{N}p(i|x)\log p(i|x) is the entropy of

Read the complete text