Token Perturbation Guidance for Diffusion Models

Javad Rajabi Soroush Mehraban Seyedmorteza Sadat Babak Taati, [2mm] University of Toronto, [1.5mm] KITE Research Institute · 2025

Read Token Perturbation Guidance for Diffusion Models by Javad Rajabi Soroush Mehraban Seyedmorteza Sadat Babak Taati, [2mm] University of Toronto, [1.5mm] KITE Research Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10036.txt.

Introduction

Javad Rajabi, Soroush Mehraban, Seyedmorteza Sadat, and Babak Taati guide diffusion image models by perturbing tokens inside the network, in a paper from 2025. Standard guidance needs special training and only works for conditional generation. A training-free method that works for any prompt state would widen use.

Their method shuffles token representations in a norm-preserving way to create a guidance signal during sampling. Analysis shows the signal behaves much like the standard one. On two leading image models, unconditional quality improved nearly twofold by their score, while conditional prompt alignment nearly matched standard guidance. No architecture change is needed.

Catalog record

CREATORJavad Rajabi Soroush Mehraban Seyedmorteza Sadat Babak Taati, [2mm] University of Toronto, [1.5mm] KITE Research Institute
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10036

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Token Perturbation Guidance for Diffusion Models Javad Rajabi Soroush Mehraban Seyedmorteza Sadat Babak Taati, [2mm] University of Toronto, [1.5mm] KITE Research Institute Abstract Classifier-free guidance (CFG) has become an essential component of modern diffusion models to enhance both generation quality and alignment with input conditions. However, CFG requires specific training procedures and is limited to conditional generation. To address these limitations, we propose Token Perturbation Guidance (TPG), a novel method that applies perturbation matrices directly to intermediate token representations within the diffusion network. TPG employs a norm-preserving shuffling operation to provide effective and stable guidance signals that improve generation quality without architectural changes. As a result, TPG is training-free and agnostic to input conditions, making it readily applicable to both conditional and unconditional generation. We further analyze the guidance term provided by TPG and show that its effect on sampling more closely resembles CFG compared to existing training-free guidance techniques. Extensive experiments on SDXL and Stable Diffusion 2.1 show that TPG achieves nearly a 2 × \times improvement in FID for unconditional generation over the SDXL baseline, while closely matching CFG in prompt alignment. These results establish TPG as a general, condition-agnostic guidance method that brings CFG-like benefits to a broader class of diffusion models. The code is available at https://github.com/TaatiTeam/Token-Perturbation-Guidance 1 Introduction Diffusion models ho2020denoising ; song2020denoising ; song2020score have emerged as the main methodology behind many successful generative models for images esser2024scaling ; podell2023sdxl , videos blattmann2023stable ; bar2024lumiere ; yang2024cogvideox ; yu2024efficient ; tian2024videotetris , audio tian2025audiox ; xue2024auffusion , and 3D objects cui2024neusdfusion ; liu2024pi3d ; yan2024dreamdissector . Despite their theoretical capacity to produce high-fidelity data, unguided diffusion models often suffer from poor sample quality, manifesting as visual artifacts, lack of semantic consistency, and insufficient sharp details ( karras2024guiding ) . To mitigate these issues, classifier-free guidance (CFG) ho2022classifier has become the de facto approach to steer the generation process toward higher quality and more semantically aligned outputs. However, CFG is inherently limited to conditional generation and requires a specific training strategy that randomly replaces the input condition with a null condition. In response to these constraints, several alternative guidance techniques have emerged, aiming to extend CFG-like benefits to broader settings hong2024smoothed ; karras2024guiding ; ahn2024self ; hong2023improving ; sadat2024no . These approaches often manipulate components of the denoiser network, such as attention layers, to construct effective guidance signals. However, they either require additional specialized training or offer limited improvements in prompt alignment and generation quality (particularly w.r.t. unconditional generation). Accordingly, there remains a need for a training-free guidance mechanism that works across both conditional and unconditional settings while improving generation quality and semantic alignment similar to CFG. In this paper, we revisit existing attention-based guidance techniques and aim to bridge the gap between the effectiveness of CFG and that of training-free, condition-agnostic methods. We observe that CFG effectively recovers global structure and coarse details during the early denoising steps, whereas existing training-free methods tend to produce over-smoothed results at the same stage (see highlighted regions in Figure 1 ). This is problematic because early denoising steps are critical for both image quality and prompt alignment, as they establish global structure, major shapes, and coarse semantics before the network begins refining fine details wang2023diffusion ; dhariwal2021diffusion . If the model fails to capture correct semantics, object placement, or overall composition at this stage, it may never fully recover from that high-level mismatch in later refinements. This lack of sufficient early-step guidance likely explains why existing methods often yield only marginal improvements in prompt alignment and generation quality compared to CFG. Figure 1: Visualization of the denoising process over time for different guidance strategies: CFG ho2022classifier , PAG ahn2024self , SEG hong2024smoothed , and our proposed TPG. Each row shows generated images at various denoising time steps, from t=981 (left) to t=1 (right). The red box highlights the early-to-middle denoising stage ( t=821 to t=741 ), where CFG and TPG demonstrate clearer structure (e.g. horse face) and consistency. The text prompt used is "a female in a black jacket is riding a brown and white horse" . Motivated by these insights, we introduce Token Perturbation Guidance (TPG), a novel method that directly perturbs intermediate token representations within the diffusion network, without requiring additional training or architectural changes. TPG employs token shuffling as the core operation to provide effective guidance signals. Specifically, token shuffling is (i) linear, (ii) preserves token norms, and (iii) disrupts local structure while maintaining global statistics. As shown in Figure 1 , TPG exhibits behavior similar to CFG and, compared to other training-free methods, more faithfully recovers both global structure and fine details at early denoising stages. We evaluate the effectiveness of TPG on both conditional and unconditional image generation using SDXL podell2023sdxl and Stable Diffusion 2.1 rombach2022high . Our results show that TPG achieves nearly a 2 \times improvement in FID for unconditional generation compared to the SDXL baseline and also significantly outperforms existing perturbation-based guidance techniques. Furthermore, we observe that TPG closely mirrors CFG in terms of alignment and its effect on the sampling process, i.e., positively aligning with ground-truth noise in low-frequency bands, remaining largely orthogonal at other frequencies, and following a similar norm profile throughout denoising. These findings establish TPG as a general, condition-agnostic guidance method that extends CFG-like benefits to a broader class of diffusion models, including those for unconditional generation. 2 Related work Score - based diffusion models sohl2015deep ; ho2020denoising ; song2019generative ; song2020score reverse a forward noising process by learning the score function, the gradient of the log data density, at multiple noise levels to progressively transform pure Gaussian noise into realistic samples ( song2019generative ) . This principled estimator of the data distribution has outperformed previous generative modeling methods in both fidelity and mode coverage dhariwal2021diffusion . Since the score represents an explicit gradient field, the sampling trajectory can be guided by incorporating auxiliary gradients, leading to powerful guidance techniques such as classifier guidance (CG) dhariwal2021diffusion and classifier - free guidance (CFG) ho2022classifier . These guidance methods significantly enhance image quality and prompt alignment, albeit at the cost of oversaturation ( sadat2025eliminating ) and reduced diversity ( ho2022classifier ; sadat2024cads ) . Although CFG improves image fidelity and alignment with the input condition, it is inherently restricted to conditional generation. Moreover, since its guidance signal is defined as the difference between conditional and unconditional denoising outputs, CFG requires specific training procedures and its sampling trajectory can overshoot the desired conditional distribution, leading to skewed or oversimplified images kynkaanniemi2024applying . Autoguidance karras2024guiding builds on CFG by introducing a deliberately weaker, also known as the “bad version” of the noise predictor, i.e., a less-trained denoiser network, to produce the guidance signal. While this avoids CFG’s reliance on the unconditional score, identifying an effective “bad version” is non-trivial and still requires training and careful tuning of the model. By contrast, our method requires no additional training or architectural changes. Recently, attention-based perturbation methods have shown promising results in improving the quality of generated images by leveraging or modifying the attention maps within the diffusion model’s attention blocks. Self-Attention Guidance (SAG) hong2023improving uses the model’s own attention map to blur the denoiser input. Perturbed Attention Guidance (PAG) ahn2024self replaces

Read the complete text