On the Anisotropy of Score-Based Generative Models

Andreas Floros, Seyed-Mohsen Moosavi-Dezfooli & Pier Luigi Dragotti · 2025

Read On the Anisotropy of Score-Based Generative Models by Andreas Floros, Seyed-Mohsen Moosavi-Dezfooli & Pier Luigi Dragotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22899.txt.

Introduction

A model that makes new data may favor some trends over others. Andreas Floros wrote a 2025 paper on that bias. His co-authors are Seyed-Mohsen Moosavi-Dezfooli and Pier Luigi Dragotti. The models are score based. A score is a slope field that guides how new data is made.

The team name the key trends Score Anisotropy Directions. In short, they call them SADs. Each network has its own set of SADs. The SADs seem to form a base that fits the network shape. That fit may hint at how well a model will learn. It can be read before training starts. Tests on fake data and on image sets back this claim. SADs also track with downstream scores.

Catalog record

CREATORAndreas Floros, Seyed-Mohsen Moosavi-Dezfooli & Pier Luigi Dragotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Anisotropy of Score-Based Generative Models Andreas Floros, Seyed-Mohsen Moosavi-Dezfooli & Pier Luigi Dragotti Abstract We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture’s output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models. 1 Introduction Figure 1: Sphere modeling in subspaces of \mathbb{R}^{D} ( D=256 ) via DiT ( Peebles & Xie 2023 ) . The only difference is the choice of subspace: the left “sphere” lies in a subspace aligned with the network’s geometry, \mathbf{G}_{\mathcal{F}} , while the right is in a non-aligned subspace. Despite identical setups, their quality differs consistently across repeated trials, suggesting that alignment with architectural geometry controls generalization. We formalize these ideas in Sections 3.2 and 3.3 . Neural networks generalize through inductive biases, i.e., biases that guide learning beyond training data ( Goyal & Bengio 2020 ; Wilson & Izmailov 2020 ) . For discriminative tasks, they are partially characterized through the Neural Anisotropy Directions (NADs), which reveal the architecture’s directional preferences in the input space ( Ortiz-Jimenez et al. 2020 ) . However, generative modeling lacks a cohesive theory that explains how architectural geometry interacts with data manifolds ( Kadkhodaie et al. 2024 ; An et al. 2025 ) . In this work, we present a unified approach to explaining and interpreting inductive biases of score-based generative models by examining anisotropy in the output space, where networks exhibit preferential learning along certain directions. As motivated in Figure 1 , we posit that generalization ability is largely characterized by the alignment of the data with the architecture’s “geometry” at initialization. Our contribution is making this notion of “geometry” precise and decomposing the output space in terms of the anisotropy directions induced by it, i.e., the Score Anisotropy Directions (SADs) that we introduce below: Definition 1 ( Score Anisotropy Directions ) . The Score Anisotropy Directions (SADs) of an architecture are the ordered set of orthonormal vectors of the output space, \{\bm{u}_{i}\}_{i=1}^{D} , that are ranked in terms of the preference of the network to generate data along those particular directions via score-based generative modeling. 2 Background Here we give a brief overview of score-based generative models and provide some context on existing work that examines inductive biases of deep neural networks. We defer an extended discussion and comparison with related work to Section 4.1 . 2.1 Score-Based Generative Models At the heart of score-based modeling is the score function , i.e., the gradient of the log-density of the underlying data distribution p . Given an estimate of the score, one can, in theory, sample from p via a gradient ascent procedure. Starting with an arbitrary prior, \bm{x}_{0}\sim\pi , we have the following Langevin dynamics iterations: \bm{x}_{k+1}=\bm{x}_{k}+\frac{\eta}{2}\overbrace{\nabla_{\bm{x}_{k}}\log p(\bm{x}_{k})}^{\text{``score"}}+\sqrt{\eta}\bm{z}_{k},\quad k=0,1,\dots,K (1) where \bm{z}_{k}\sim\mathcal{N}(\bm{0},\bm{I}) is standard Gaussian and \eta>0 . If \eta\to 0 and K\to\infty , under certain technical conditions, the iterates in Equation 1 converge to a sample from p ( Welling & Teh 2011 ) . However, a practical limitation of the above setup is that estimated scores are inaccurate in low-density regions (e.g., in early iterations where learning is intractable). Moreover, score functions may be undefined in the case of data residing in low-dimensional manifolds. That is, the overall approach breaks down under the commonly adopted manifold hypothesis ( Song & Ermon 2019 ) . The research community has therefore largely moved on to Denoising Score Matching (DSM) ( Vincent 2011 ) and annealed Langevin dynamics, i.e., diffusion models, which are also the main focus of this work. 1 1 1 Without loss of generality, we will standardize our notation to the variance exploding formulation of Song & Ermon 2019 . Specifically, consider noise scales, \sigma\in[\sigma_{\text{min}},\sigma_{\text{max}}] , and associated densities, p_{\sigma}=p*\mathcal{N}(\bm{0},\sigma^{2}\bm{I}) , where * represents convolution. Here, \sigma_{\text{min}} is small enough such that p_{\sigma_{\text{min}}}\approx p and \sigma_{\text{max}} is large enough so we can write p_{\sigma_{\text{max}}}\approx\mathcal{N}(\bm{0},\sigma^{2}_{\text{max}}\bm{I}) . With estimates of scores of the perturbed distributions, \nabla_{\bm{x}_{\sigma}}\log p_{\sigma}(\bm{x}_{\sigma}) , one samples from p by decaying \sigma from \sigma_{\text{max}} to \sigma_{\text{min}} ( Ho et al. 2020 ; Song et al. 2021 ) . This way, accurate modeling along sampling trajectories is tractable and the noise ensures support over the entirety of the ambient space, overcoming the limitations of naive Langevin dynamics. In particular, the scores are equivalent to minimum mean squared error Gaussian denoisers ( Efron 2011 ) . That is, for neural networks, \mathcal{F}_{\bm{\theta}}:\mathbb{R}^{D}\times\mathbb{R}\to\mathbb{R}^{D} , parameterized by \bm{\theta} , one can approximate the scores via the following DSM optimization: \min_{\bm{\theta}}\mathbb{E}_{\bm{x}\sim p,\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}),\sigma}\left[\left\|\mathcal{F}_{\bm{\theta}}(\bm{x}+\sigma\bm{\epsilon},\sigma)+\frac{\bm{\epsilon}}{\sigma}\right\|^{2}_{2}\right]. (2) 2.2 Inductive Biases in Deep Learning There is a vast literature on understanding the inductive biases of deep neural networks ( Goyal & Bengio 2020 ; Wilson & Izmailov 2020 ) . Of particular interest is the work of Ortiz-Jimenez et al. 2020 , who identify directional biases in classifiers, i.e., the Neural Anisotropy Directions (NADs). More recently, Movahedi et al. 2025 have extended the NAD framework by formalizing the concept of input-space architectural geometry and exploring its evolution during training. Specifically, they propose the geometric invariance hypothesis , which posits that NADs persist through training. To our knowledge, contrary to the discriminative case, there is no unified theory that explains the preferred modeling directions of diffusion. For example, a recent study by Kadkhodaie et al. 2024 argues that convolutional diffusion models are biased towards Geometry-Adaptive Harmonic Bases (GAHBs). However, they acknowledge that a mathematically precise definition of such bases remains an open question. More fundamentally, follow-up work by An et al. 2025 suggests that the theory of GAHBs does not extend to transformer-based diffusion models. Specifically, they instead resort to a transformer-specific analysis by inspecting and manipulating attention maps. In contrast to the above-mentioned works on the inductive biases of diffusion models, we present a more unified treatment of directional biases that is agnostic to the denoising network’s architecture. In particular, our key insight is recognizing that the NAD framework can be adapted and extended to score-based modeling by assuming an underlying data log-density that assigns high probability to “on-manifold” data and low probability otherwise. With this approach, our analysis amounts to understanding the anisotropy directions of such implicitly induced discriminative models. 3 Directional Inductive Biases of Diffusion Models Figure 2: \text{MSW}_{2} distance (computed over 10k test samples and 16384 projections) of iDDPM U-Net ( Nichol & Dhariwal 2021 ) architecture. Each pixel corresponds to a rank-one dataset of 16 \times 16 images (with 10k training samples) that is aligned with a basis element of the canonical basis, DCT, DST, (ordered) Hadamard transform or Haar wavelet transform. That is, for a given location (canonical) or frequency / sequency (DCT, DST, Hadamard) or scale, channel and location (Haar), we visualize the performance on the corresponding dataset. For ease of visualization, in the case

Read the complete text