Entropy Rectifying Guidance for Diffusion and Flow Models
Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari · 2025
Read Entropy Rectifying Guidance for Diffusion and Flow Models by Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13987.txt.
Introduction
Tariq Berrada Ifriqi and co-authors steer image generators with an attention trick in this 2025 paper. Diffusion models use guidance to sharpen images and honor prompts. The standard trick forces a trade: better quality costs variety, or consistency costs one of the two. Their method, entropy rectifying guidance, adjusts attention at inference time instead.
It needs no extra model and no added sampling passes. It also works without a prompt, where the classic trick cannot run. Tests show gains in image quality, variety, and prompt match together. It stacks cleanly with two recent guidance methods for further gains. The authors offer it as a simple, general upgrade for image models.
Catalog record
| CREATOR | Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Entropy Rectifying Guidance for Diffusion and Flow Models Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari Abstract Guidance techniques are commonly used in diffusion and flow models to improve image quality and input consistency for conditional generative tasks such as class-conditional and text-to-image generation. In particular, classifier-free guidance (CFG) is the most widely adopted guidance technique. It results, however, in trade-offs across quality, diversity and consistency: improving some at the expense of others. While recent work has shown that it is possible to disentangle these factors to some extent, such methods come with an overhead of requiring an additional (weaker) model, or require more forward passes per sampling step. In this paper, we propose Entropy Rectifying Guidance (ERG), a simple and effective guidance method based on inference-time changes in the attention mechanism of state-of-the-art diffusion transformer architectures, which allows for simultaneous improvements over image quality, diversity and prompt consistency. ERG is more general than CFG and similar guidance techniques, as it extends to unconditional sampling. We show that ERG results in significant improvements in various tasks, including text-to-image, class-conditional and unconditional image generation. We also show that ERG can be seamlessly combined with other recent guidance methods such as CADS and APG, further improving generation results. † † correspondence: First Author at [email protected] CFG ERG (ours) CFG ERG (ours) Figure 1 : Qualitative comparison of classifier-free guidance (CFG) and our Entropy Rectifying Guidance (ERG). The images generated using ERG (bottom rows) exhibit greater quality and diversity than standard CFG. Images are generated using 50 Euler steps; each column corresponds to a different random seed for the generations. 1 Introduction Diffusion ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song et al., 2021 ; Dhariwal and Nichol, 2021 ) and flow models ( Lipman et al., 2023 ; Ma et al., 2024 ; Esser et al., 2024 ) are state-of-the-art generative modeling tools for various modalities, ranging from images ( Rombach et al., 2022 ; Podell et al., 2024 ; Chen et al., 2024 ; Esser et al., 2024 ) , to audio ( Wang et al., 2023 ; Levy et al., 2023 ) , and video ( Jin et al., 2025 ; Polyak et al., 2024 ) . These models generate data by starting with a simple prior and iteratively removing noise – a process called “denoising”. See Kingma and Gao (2023) ; Lipman et al. (2023) for details on diffusion and flow matching. These models can be conditioned on various inputs to control the generative process, e.g. , in text-to-image models. Guidance techniques such as classifier guidance ( Dhariwal and Nichol, 2021 ) and classifier-free guidance ( Ho and Salimans, 2021 ) are commonly used to improve sample quality and consistency with input conditioning. In the sampling process, these techniques combine a conditional signal with an unconditional one, and control the strength of conditioning through a scaling parameter. Although such guidance techniques are a crucial component in achieving state-of-the-art results, they are also known to negatively affect the diversity of generated samples given a particular prompt ( Sadat et al., 2024 ; Karras et al., 2024 ; Kynkäänniemi et al., 2024 ; Saharia et al., 2022 ) . Moreover, too high guidance scales may lead to overly saturated images, affecting the quality of generated images ( Saharia et al., 2022 ) . An extensive analysis on the trade-offs of image generation quality, diversity, and consistency was presented by Astolfi et al. (2024) . To mitigate these quality-diversity-consistency trade-offs, more advanced guidance techniques have recently been proposed, see e.g. Sadat et al. (2025) ; Karras et al. (2024) . Most guidance techniques , however, require spending part of the training cycles on unconditional generation for the guidance to work, even if unconditional generation is not a goal in itself, and are not applicable to unconditional sampling ( Sadat et al., 2025 ; Ho and Salimans, 2021 ) . Others rely on a second model with weaker performance than the main model, thereby increasing memory requirements ( Karras et al., 2024 ) . In our work, we build upon the work of Karras et al. (2024) and Hong (2024) , and propose Entropy Rectifying Guidance (ERG): a simple and effective method to obtain both a strong and a weak predictive signal from a single model that leverages attention layers, where the model may be conditional or unconditional. In particular, our method uses the Hopfield energy formulation of attention ( Ramsauer et al., 2021 ; Hong, 2024 ) and applies a temperature scaling to the softmax function of the attention layers in order to obtain the weak predictive signal. This scaling does not require any adaptations in model training, and may be applied to pre-trained denoising models, as well as their accompanying text encoders. Moreover, motivated by this energy interpretation of attention layers, we also consider iterative re-application of attention layers, and rescaling the residual attention update. Our experiments show that manipulating the attention layers in this manner results in simultaneous improvements in sample quality, diversity, and consistency, contrary to the trade-offs observed by Astolfi et al. (2024) for classifier-free guidance. In summary, our contributions are the following: 1. We propose Entropy Rectifying Guidance (ERG), a guidance mechanism based on modifying the energy landscape of the attention layers. 2. Since our guidance mechanism does not require unconditional inference, it is directly applicable to any attention-based diffusion or flow model, including unconditional, class-conditional, and text-to-image models. 3. Experimentally, we find that ERG significantly improves image quality and diversity while retaining the same prompt consistency as the standard classifier-free guidance ( +30 points in density, +4 points in VQAScore) on COCO at 512 resolution using our 1.9B text-to-image model. 2 Related work and background In this section, we briefly review background material on diffusion and flow models, and related work on guidance techniques to improve sampling as well as the Hopfield energy formulation of attention. 2.1 Diffusion and flow matching models Diffusion models ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song et al., 2021 ) form a flexible class of generative models whose underlying principle is to map samples \boldsymbol{\epsilon} from a trivial unit Gaussian prior p_{0}=\mathcal{N}(0,{\bf I}) to samples from a learned model p_{1} of the data distribution. The forward process is defined as: {\bf x}_{t}=\alpha_{t}{\bf x}_{1}+\sigma_{t}\boldsymbol{\epsilon} with t\in[0,1] , where {\bf x}_{1}\sim p_{1} , and \alpha_{t} is a decreasing function of “time” t while \sigma_{t} an increasing function of t . Flow matching methods ( Lipman et al., 2023 ) assume that \alpha_{0}=\sigma_{1}=1 and \alpha_{1}=\sigma_{0}=0 . Using these assumptions, during the reverse process {\bf x}_{t} interpolates between \boldsymbol{\epsilon} at t=0 and {\bf x}_{1} at t=1 . In contrast, score-based diffusion models ( Ho et al., 2020 ; Dhariwal and Nichol, 2021 ) set \alpha_{t} and \sigma_{t} implicitly through different formulations of stochastic differential equations (SDE) where \mathcal{N}(0,{\bf I}) is the equilibrium distribution. Additionally, they consider t\in[0,T] with T large enough so that {\bf x}_{T} is approximately distributed as a unit Gaussian random variable. 2.2 Guidance mechanisms Classifier guidance. To enable high quality conditional generation, Dhariwal and Nichol (2021) proposed to guide the sampling process by leveraging gradients from pre-trained auxiliary classifier p(c|{\bf