A Fourier Space Perspective on Diffusion Models

Fabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar · 2025

Read A Fourier Space Perspective on Diffusion Models by Fabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11278.txt.

Introduction

Fabian Falck and co-authors look at image-making models in 2025 from a new angle: how they treat fine detail versus broad shapes. These diffusion models learn by adding noise to data and then removing it. In the frequency view the team uses, fine detail sits in high bands and broad shapes in low bands, and real data fade fast toward the high bands.

Standard training noise corrupts fine detail much sooner than broad shapes, so models build broad shapes first. The team shows this skew breaks a key math assumption and hurts fine detail quality. Their alternate process noises all bands at the same rate. It did far better where fine detail matters most and matched standard results on common image tests.

Catalog record

CREATORFabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11278

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Fourier Space Perspective on Diffusion Models Fabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar Abstract Diffusion models are state-of-the-art generative models on data modalities such as images, audio, proteins and materials. These modalities share the property of exponentially decaying variance and magnitude in the Fourier domain. Under the standard Denoising Diffusion Probabilistic Models (DDPM) forward process of additive white noise, this property results in high-frequency components being corrupted faster and earlier in terms of their Signal-to-Noise Ratio (SNR) than low-frequency ones. The reverse process then generates low-frequency information before high-frequency details. In this work, we study the inductive bias of the forward process of diffusion models in Fourier space. We theoretically analyse and empirically demonstrate that the faster noising of high-frequency components in DDPM results in violations of the normality assumption in the reverse process. Our experiments show that this leads to degraded generation quality of high-frequency components. We then study an alternate forward process in Fourier space which corrupts all frequencies at the same rate, removing the typical frequency hierarchy during generation, and demonstrate marked performance improvements on datasets where high frequencies are primary, while performing on par with DDPM on standard imaging benchmarks. 1 Introduction Figure 1 : [Left] The Fourier power law observed in (top-left) images [ Krizhevsky et al., 2009 ] , (top-right) videos [ Kay et al., 2017 ] , (bottom-left) audio [ Tzanetakis, 1999 ] , and (bottom-right) Cryo-EM derived protein density maps [ wwPDB Consortium, 2023 ] . [Center] A DDPM forward process on these modalities noises high-frequency components substantially faster (SNR changes more per time increment), and earlier than low-frequency components. [Right] The alternate EqualSNR forward process noises all frequencies at the same rate, disrupting DDPM’s generation hierarchy. The GIFs are best viewed in Adobe Reader. Diffusion models are the state-of-the-art generative model on data modalities such as images [ Rombach et al., 2022 , Baldridge et al., 2024 ] , videos [ Brooks et al., 2024 , Ho et al., 2022 , Blattmann et al., 2023 ] , proteins [ Watson et al., 2023 , Lewis et al., 2024 ] , and materials [ Zeni et al., 2023 ] . They excel on a wide range of tasks and applications on these modalities, such as generating high-resolution images and videos given a text prompt [ Rombach et al., 2022 ] , sampling the distribution of conformational states of proteins [ Lewis et al., 2024 ] , or generating novel materials under property constraints [ Zeni et al., 2023 ] . Why do diffusion models work so well on these modalities , possibly even surpassing the performance and efficiency of autoregressive models? In this work, we study this question via the forward (or noising) process of diffusion models in Fourier space. The forward process of standard diffusion models such as DDPM [ Song et al., 2020a ] corrupts data by progressively adding white Gaussian noise until all information is destroyed. 1 1 1 White noise has equal variance across all frequency components. Diffusion models then learn a denoiser which reverses this forward process by starting from Gaussian noise and iteratively refining it to approximate the original data [ Song et al., 2020b , Sohl-Dickstein et al., 2015 ] . While one might assume that the forward process destroys all information in data uniformly, this is not the case. In fact, for the modalities mentioned above, a DDPM forward process does not treat all frequency components equally. These modalities have in common that they exhibit a power law in their Fourier representation: low-frequency components have orders of magnitudes higher variances (and magnitudes) than high-frequency components (see Figure 1 [left], [ Van der Schaaf and van Hateren, 1996 ] ). This data property has two important implications: The DDPM forward process noises high-frequency components both substantially earlier [ Rissanen et al., 2022 ] , and faster than low-frequency components (see Figure 1 [centre]), which we will discuss in Section 2 and theoretically characterise with the Signal-to-Noise Ratio (SNR) (see Definition 4 ). Intuitively speaking, the forward process in DDPM corrupts the high frequency information—the fine details such as edges—in fewer timesteps than low-frequency features, such as the larger structures and overall colour of an image (see Figure 4 for an illustration). What is the inductive bias of this forward process on the learned reverse process? In theory (i.e. with unlimited resources and arbitrarily accurate estimates of the scores of the noised distributions), diffusion models can learn to express any continuous distribution (see for e.g. Theorem 2 in [ Chen et al., 2023 ] ). In practice, however, diffusion models are constrained, for instance by a limited number of intermediate steps (discretisation) and the expressiveness of the neural network (score estimation), resulting in approximation errors. Since DDPM applies noise more aggressively to high-frequency components, corrupting them in fewer steps, we hypothesise that their approximation error is larger, resulting in a lower generation quality. As a result, DDPM prioritises low-frequency components within its resource constraints. The forward process also imposes a hierarchy of the frequencies during generation: as the generative process learns to reverse the forward process (which in DDPM and on the modalities of interest noises high frequencies before low frequencies), the reverse process generates low frequencies first, and generates high frequencies conditional on low frequencies (see Figure 4 for an illustration). Previous work has observed this phenomenon of a soft-conditioning or “approximately autoregressive” generation in DDPM diffusion models [ Dieleman, 2024 , Gerdes et al., 2024 ] , drawing an important parallel with Large Language Models (LLMs). However, in spite of this ‘hidden structure’ in diffusion models, we lack understanding of the degree to which it is essential. Can diffusion models also generate all frequencies at the same rate, i.e. without any hierarchy (see Figure 1 [right]), and how does this perform? Can we also generate high-frequency details first, then low frequencies conditional on them (see Figure 16 in Appendix B )? In this work, we study the forward process of diffusion models on data exhibiting the Fourier power law and its effect on the learned reverse process in Fourier space. We theoretically and empirically analyse the impact of noising rate and hierarchy on the assumptions of the reverse process, and in turn on generation quality in diffusion models. We organise our contributions as follows: • In Section 2 , we formally describe the DDPM forward process and define the SNR. We observe that this forward process results in a non-uniform rate of noising across frequencies: specifically, high-frequency components are noised faster (in the sense that the SNR descreases more per time increment) and earlier (in the sense that the SNR of high-frequency components is smaller than the SNR of low-frequency components at all timesteps). • In Section 3 , we theoretically explain and empirically demonstrate that faster noising as measured by SNR of high-frequency components violates the Gaussian assumption in the backward process, leading to worse generation quality for these frequencies. We then propose a framework for forward processes based on frequency-specific SNR, including a training and sampling algorithm, a loss based on the Evidence Lower Bound (ELBO), and a mutual calibration procedure. In

Read the complete text