Spectral Guidance for Flexible and Efficient Control of Diffusion Models

Gabriel Moreira, Manuel Marques, João Paulo Costeira, Chenyan Xiong · 2026

Read Spectral Guidance for Flexible and Efficient Control of Diffusion Models by Gabriel Moreira, Manuel Marques, João Paulo Costeira, Chenyan Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28900.txt.

Introduction

Gabriel Moreira and colleagues steer image-making diffusion models by learning the few features that stay useful as noise corrupts data. They identify these features, learned without labels, as singular functions of a conditional expectation operator. The learned basis then projects guidance signals such as labels or masks onto the sampling path. The method needs no retraining and no denoiser backpropagation, and it lifts accuracy on CIFAR-10 by 37 percentage points over the best training-free baseline while sampling four times faster.

Catalog record

CREATORGabriel Moreira, Manuel Marques, João Paulo Costeira, Chenyan Xiong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94042
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spectral Guidance for Flexible and Efficient Control of Diffusion Models Gabriel Moreira, Manuel Marques, João Paulo Costeira, Chenyan Xiong Abstract We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering 4 × 4\times faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance. 1 Introduction Generative modeling has seen tremendous progress with diffusion-based approaches ( 29 ) , which now produce high-fidelity samples across images, audio, and other modalities ( 14 ; 33 ; 27 ) . These models operate by reversing a progressive corruption process, gradually transforming structured data into noise. While the forward and reverse dynamics of generation are well-understood, the utility of these models hinges on guidance : sampling according to user specifications, which can be in the form of labels ( 9 ) , text prompts ( 28 ) , or custom objectives ( 39 ) . The key challenge is how to impose such guidance in practice. Successful mainstream approaches rely on training models to be conditional from the outset ( 9 ; 15 ) , so that desired guidance can be injected directly during sampling. This strategy yields strong and stable control, but tightly couples the model to a fixed set of conditions and requires retraining or additional models whenever the specification changes. An alternative is to start from an unconditional model and enforce the desired behavior only at sampling time by optimizing a user-defined objective ( 5 ; 35 ) . While this offers greater flexibility, it typically requires differentiating through the denoiser during sampling and approximating an intractable posterior distribution. In practice, this leads to higher computational cost and often unstable control, especially for complex objectives. In this work, we propose Spectral Guidance, a framework that enables flexible control by leveraging the intrinsic structure of the diffusion process. As data is gradually corrupted by noise, fine-grained details are lost while coarse semantic features persist. We show that these features form a natural, low-dimensional coordinate system that tracks how information propagates through the diffusion. By learning this representation, we can project arbitrary guidance objectives, from simple labels to masks, directly into the generative trajectory. This allows for stable control without the need for task-specific retraining or denoiser gradients. Our technical approach is based on a low-rank approximation of the conditional expectation operator across diffusion timesteps. Because high-frequency details are progressively destroyed, the leading singular functions of this operator form a time-indexed low-dimensional basis that captures persistent axes of variation over diffusion time. We show that a self-supervised learning (SSL) objective with orthogonality constraints ( 2 ) is a variational estimator for the operator’s leading singular functions, with independently diffused views of the same sample acting as augmentations. Once these representations are learned, guidance reduces to a simple linear projection onto this basis. This formulation provides an efficient and flexible guidance mechanism. It removes the need for backpropagation through the denoiser during sampling, requiring only a one-time training to learn the intrinsic coordinates of the diffusion process. These representations reveal which features are recoverable and when guidance is effective. Further, being task-agnostic, they support arbitrary downstream control objectives without retraining. Empirically, we achieve consistent gains across label, attribute, and CLIP guidance; notably, on CIFAR-10, we surpass the strongest training-free baseline by 37 percentage points in accuracy while improving FID and sampling 4 \times faster. In addition, our representations generalize to spatial control like mask-guided generation without auxiliary models. Finally, they reveal a spectral phase transition that pinpoints the optimal time window for effective guidance. Our contributions are summarized as follows: • We propose Spectral Guidance, which reframes diffusion guidance as a projection onto a coordinate system aligned with the generative dynamics, enabling flexible control without retraining the diffusion model. • We introduce an SSL objective that estimates the spectral decomposition of the diffusion operator. This representation yields a lightweight guidance algorithm, decoupled from the gradients of the denoiser. • We outperform training-free baselines by over 37 percentage points in accuracy and 4\times in speed, while enabling complex controls like mask guidance without auxiliary models. Further, our representations reveal a phase transition during the reverse process that aligns with the optimal time window for effective guidance. Code available at https://github.com/gabmoreira/spectralguidance 2 Related Work We review key advances in conditional generation below. Diffusion guidance. To introduce conditioning or amplify specific signals during sampling, Classifier Guidance (CG) ( 9 ) employs an external classifier trained on noisy data, using its gradients to steer the generative trajectory. Classifier-Free Guidance (CFG) ( 15 ) alleviates the need for external classifiers by jointly training a conditional and unconditional model, subsequently interpolating their score estimates during sampling. While CFG has become the de facto standard for modern architectures, ranging from Stable Diffusion ( 27 ) to Flow Matching models ( 23 ) , it typically applies a constant guidance scale, remaining agnostic to the dynamics of the diffusion process. For a comprehensive survey, we refer to 38 . To address the limitations of static guidance, recent works ( 19 ; 21 ) leverage insights into diffusion dynamics and phase transitions ( 10 ; 26 ) to target time windows where features are most controllable, thereby optimizing generation quality. Training-free guidance. A distinct line of research focuses on controlling pre-trained diffusion models via external loss functions, in a plug-and-play fashion, eliminating the need for retraining. Initially developed for inverse problems ( 18 ; 4 ; 17 ) , this paradigm includes Diffusion Posterior Sampling (DPS) ( 5 ) , which guides sampling with the gradients of a loss function evaluated on a point-estimate of the posterior distribution. For general control, Loss-Guided Diffusion (LGD) ( 31 ) refines DPS by estimating the conditional expectation via Monte Carlo sampling, while MPGD ( 12 ) leverages the manifold hypothesis to constrain guidance to low-dimensional data manifolds. Universal Guidance for Diffusion (UGD) ( 1 ) and FreeDoM ( 36 ) further strengthen guidance through iterative “time-travel” strategies and adaptive schedules across diffusion timesteps. More recently, Training-Free Guidance (TFG) ( 35 ) unifies many of these approaches under a common guidance algorithm. Editing directions in diffusion models. NoiseCLR ( 7 ) discovers interpretable directions in the noise space of pre-trained diffusion models via a contrastive self-supervised objective; while it shares the use of SSL-style training, it targets latent-space editing rather than information-preserving structure. More closely related, 24 and 3 use the spectral decomposition of the denoiser’s Jacobian as a post hoc

Read the complete text