Colored Noise Diffusion Sampling

Hadar Davidson, Noam Issachar, Sagie Benaim · 2026

Read Colored Noise Diffusion Sampling by Hadar Davidson, Noam Issachar, Sagie Benaim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30332.txt.

Introduction

Hadar Davidson, Noam Issachar and Sagie Benaim set out a view of SDE sampling as split energy transfer. Diffusion models exhibit spectral bias, fixing low-end structure early and high-end detail later. They present Colored Noise Sampling (CNS), a training-free solver using a tuned schedule sending added energy toward unresolved bands. By using the model's spectral bias, CNS steers the spread toward the data pattern. Tests show CNS beats ODE and SDE baselines as a direct sampler swap across designs. On ImageNet-256, CNS achieves large unguided FID cuts.

Catalog record

CREATORHadar Davidson, Noam Issachar, Sagie Benaim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100837
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30332

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Colored Noise Diffusion Sampling Hadar Davidson, Noam Issachar, Sagie Benaim Abstract Diffusion models achieve state-of-the-art image synthesis, with their generative trajectories fundamentally exhibiting a spectral bias, resolving low-frequency global structures early and high-frequency fine details later. Conventional stochastic differential equation (SDE) solvers fail to account for this dynamic, naively injecting uniform white noise throughout the entire process and misusing the finite energy budget. In this work, we establish a mathematical framework that reconsiders SDE inference as a targeted, frequency-decoupled energy transfer. Leveraging this framework, we introduce Colored Noise Sampling (CNS), a novel, training-free stochastic solver. Rather than injecting uniform white noise, CNS utilizes a dynamic, timestep- and frequency-dependent schedule that more efficiently allocates injected energy toward structurally unresolved frequency bands. By actively exploiting the model’s inherent spectral bias, CNS systematically steers the generated distribution toward the true data manifold. Extensive experiments demonstrate that CNS significantly outperforms standard ODE and SDE baselines as a strictly plug-and-play, inference-time sampler substitution across diverse architectures (SiT, JiT, FLUX). Compared to standard sampling on ImageNet-256, CNS achieves substantial unguided FID reductions, improving from 8.26 to 6.27 on SiT-XL/2, 32.39 to 26.69 on JiT-B/16, and 11.88 to 8.31 on JiT-H/16, while yielding consistent relative FID improvements with Classifier-Free Guidance. Project page is available at https://hadardavidson.github.io/CNS/ . 1 Introduction Diffusion models have established a new standard in photorealistic image synthesis, defining the state-of-the-art in high-fidelity generation [ 15 , 57 , 26 ] . Crucially, the sampling trajectory of these models exhibits a spectral bias [ 60 , 20 ] . This inductive property dictates that diffusion models inherently resolve low-frequency global structures during early sampling steps, while filling in high-frequency fine details in later steps. Current sampling algorithms [ 56 , 57 , 55 ] fail to account for this phenomenon. Standard stochastic methods based on Stochastic Differential Equations (SDEs) naively inject uniform white noise, completely disregarding how the frequency spectrum of the generated image dynamically evolves over time. To address this inefficiency at its root, we introduce a new class of stochastic solvers that actively leverages this spectral bias. By tailoring the injected noise to the specific denoising timestep, we improve generation fidelity without requiring any additional training. Recognizing the importance of spectral bias, several recent works have attempted to exploit it. One line of research alters the training framework, interpolating from spectrally non-uniform or temporally evolving noise distributions [ 8 , 53 , 17 ] . Other approaches operate at inference time, introducing ad-hoc modifications such as frequency-decoupled operations [ 43 , 66 ] , internal activation reweighting [ 54 ] , or step-size schedule adjustments [ 28 ] . While these methods yield measurable improvements, they remain fundamentally constrained by their underlying use of spectrally uniform solvers. This naturally leads to our guiding question: How can we actively exploit the spectral bias of diffusion models to design a fundamentally new, general-purpose sampler that improves generation fidelity? ODE SDE CNS (Ours) Figure 1 : Colored Noise Sampling (CNS) . Samples from SiT-XL/2 on ImageNet-256 (with CFG) for different sampling strategies. While standard SDEs inject uniform white noise, our Colored Noise Sampling (CNS) dynamically reallocates injected stochastic energy to unresolved frequency bands. This actively leverages the network’s spectral bias to systematically steer the output toward the true data manifold, outperforming standard ODE and SDE solvers. To answer this, we first establish a mathematical framework to control the generated distribution via frequency-aware noise injection. Geometrically, sampling trajectories resemble non-orthogonal rotations toward the data manifold [ 60 ] . This implies that diffusion models do not arbitrarily discard initial noise; rather, a significant structural component of this signal is preserved and mapped into final image features [ 63 , 36 ] . A key observation of our work is that this signal-preserving transfer also applies to the continuous noise injected by SDE solvers throughout the trajectory. Furthermore, this process is frequency-decoupled: injected noise in a specific frequency band maps directly to spatial features in that same band. By ensuring our frequency-aware adjustments remain strictly variance-preserving, requiring only that the total injected energy per step remains normalized, we demonstrate that the classic Langevin requirement for uniform white noise [ 72 ] can be safely relaxed without pushing intermediate states out of distribution. Building on this framework, we construct a timestep- and frequency-dependent noise schedule. By analyzing the progression rates of different frequency bands during generation, we propose that the network’s ability to convert injected noise into coherent image features significantly depends on how structurally “resolved” that specific band is at a given timestep. This insight allows us to reconsider SDE sampling as a targeted energy injection process. Rather than uniformly distributing the finite injected noise budget, our approach utilizes a dynamic schedule based on the expected evolution of the trajectory, allocating energy to the frequency bands where it is most needed. This principled allocation steers the output toward the true data manifold, yielding strictly higher-fidelity generation. To validate our approach, we conduct extensive experiments across diverse architectures and modalities, including latent-space generation (SiT [ 35 ] ), pixel-space generation (JiT [ 29 ] ), and state-of-the-art text-to-image synthesis (FLUX [ 25 , 26 ] ). Evaluated primarily via the Fréchet Inception Distance (FID) [ 14 ] , empirical results demonstrate that our method significantly outperforms standard ODE and SDE baselines. On ImageNet-256 [ 49 ] , we achieve substantial FID reductions under both unguided and Classifier-Free Guidance (CFG) [ 16 ] settings, while maintaining robust stability across varying discretization steps. We visually highlight the superiority of our approach over standard baselines in Fig. 1 . Furthermore, our sampler proves effective when integrated into text-to-image pipelines like FLUX, improving automatic human-preference scores. To summarize, our main contributions are: • We establish a mathematical framework that reframes SDE noise injection as a targeted energy transfer, and demonstrate that the standard Langevin requirement for spectrally uniform white noise can be safely relaxed to resolve spectral gaps. • We introduce Colored Noise Sampling (CNS), a novel, training-free stochastic solver that actively leverages spectral bias by dynamically allocating injected noise energy toward structurally unresolved frequency bands. • We validate CNS as a robust, general-purpose sampler across diverse architectures (SiT, JiT, FLUX). On ImageNet-256, CNS achieves substantial unguided FID reductions (e.g., 8.26 to 6.27 for SiT-XL/2, 32.39 to 26.69 for JiT-B/16) and relative CFG improvements ranging from \sim 8% to \sim 50% over standard ODE and SDE baselines. 2 Related Work Samplers for Diffusion Models. Sampling in diffusion models is a highly researched domain primarily focused on numerically mitigating discretization errors. Prominent advancements include higher-order solvers [ 61 , 64 , 34 ] that maintain fidelity at low step counts, dynamic solver alternation [ 32 , 71 ] , and state reparameterizations that smooth integration pathways [ 69 , 68 , 5 ] . While these methods successfully reduce truncation errors and accelerate generation, they remain agnostic to the evolving spatial structure of the state. Our approach is fundamentally orthogonal: rather than strictly optimizing numerical precision, we optimize the allocation of stochastic energy by explicitly exploiting the model’s spectral bias. Leveraging Spectral Bias in Diffusion Models. A prominent line of research exploits the spectral bias of diffusion models during training by altering noise distributions.

Read the complete text