Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling

2025

Read Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05743.txt.

Introduction

Xiao Li and co-authors study a puzzle from 2025 about diffusion models. These models were built to make images. Yet they also learn useful image features on their own. Feature quality first rises as noise grows. It then falls again at high noise.

The team use the fact that image data sit near a low size shape. On that base, they prove the peak appears when the model truly catches the data shape. Tests on class tasks back the claim. The peak shows when the model makes fresh images. It fades as the model starts to copy its training set.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling Abstract Xiao Li † , Zekai Zhang † , Xiang Li , Siyi Chen , Zhihui Zhu 1 , Peng Wang , Qing Qu ‡ University of Michigan \cdot 1 Ohio State University † Joint first author ‡ Corresponding author Abstract Diffusion models, though originally designed for generative tasks, have demonstrated impressive self-supervised representation learning capabilities. A particularly intriguing phenomenon in these models is the emergence of unimodal representation dynamics, where the quality of learned features peaks at an intermediate noise level. In this work, we conduct a comprehensive theoretical and empirical investigation of this phenomenon. Leveraging the inherent low-dimensionality structure of image data, we theoretically demonstrate that the unimodal dynamic emerges when the diffusion model successfully captures the underlying data distribution. The unimodality arises from an interplay between denoising strength and class confidence across noise scales. Empirically, we further show that, in classification tasks, the presence of unimodal dynamics reliably reflects the generalization of the diffusion model: it emerges when the model generates novel images and gradually transitions to a monotonically decreasing curve as the model begins to memorize the training data. Keywords: Diffusion Models, Representation Learning, Denoising AutoEncoders, Generalization Date: August 24, 2026 Correspondence: [email protected] Resources: To be released https://deepthink-umich.github.io (a) Classification (b) Segmentation Figure 1 : Unimodal representation dynamics in diffusion-based representation learning tasks. This unimodal representation pattern has been previously observed in diffusion-based representation learning tasks; see [ 8 , 96 , 86 ] . To verify this, we train diffusion models on various datasets and evaluate downstream performance using noisy images \bm{x}_{t} at different timesteps t . In both classification and segmentation tasks, the performance consistently follows a unimodal trend, peaking at intermediate noise levels. In (b), "mIoU" denotes mean Intersection over Union, a standard metric used in segmentation tasks. 1 Introduction Diffusion models, a class of score-based generative models, have achieved great empirical success in various tasks such as image and video generation, speech and audio synthesis, and solving inverse problems [ 3 , 31 , 72 , 101 , 7 , 30 , 43 , 44 , 71 , 74 , 12 , 15 , 80 , 53 , 97 ] . In general, these models, consisting of forward and backward processes, learn data distributions by simulating the non-equilibrium thermodynamic diffusion process [ 79 , 31 , 81 ] . Specifically, the forward process progressively adds Gaussian noise to training samples until they are fully corrupted, while the backward process involves learning a score-based model to generate samples from the noisy inputs [ 34 , 81 ] . Beyond their strong generative capabilities, recent studies have revealed that diffusion models also possess remarkable representation learning capabilities [ 8 , 96 , 59 , 13 , 86 , 26 ] . In particular, the internal feature extractors of trained diffusion models have been shown to serve as powerful self-supervised learners, achieving strong performance on downstream tasks such as classification [ 96 , 59 ] , semantic segmentation [ 8 ] , and image alignment [ 86 ] . In many cases, these representations match or even surpass those from specialized self-supervised methods, implying that diffusion models have the potential to unify generative and discriminative learning in vision. Alongside the empirical success, a prevalent phenomenon, which we refer to as the unimodal representation dynamics , has been widely observed in diffusion models for representation learning [ 8 , 96 , 86 ] : the quality of learned representations, as measured by downstream task performance, follows a unimodal trend across noise levels. Specifically, the most effective features consistently appear at an intermediate noise level, while performance degrades as inputs become either fully noisy or entirely clean (see Figure 1 ). Despite being widely observed, the underlying cause of this unimodal representation dynamic remains poorly understood. Our contributions. In this work, we investigate the emergence of unimodal representation dynamics in diffusion models and its cause. Our analysis characterizes how representation quality varies across noise levels and offers new perspectives on how diffusion models, despite their generative design, can excel at representation learning. Specifically, we conduct a comprehensive theoretical and empirical study of unimodal representation dynamics in diffusion models. Motivated by the well-established observation that natural image data typically reside on a low-dimensional manifold [ 25 , 67 , 82 ] , we examine the representation dynamics through how diffusion models learn from noisy mixtures of low-rank Gaussian ( MoLRG ) distributions. We show that the emergence of unimodal representations is intrinsically linked to the model’s ability to capture the underlying low-dimensional structure of the data. Our key contributions are as follows: • Mathematical framework for studying representation learning in diffusion models. To analyze the representation dynamics, we provide a mathematical study of how diffusion models learn from MoLRG distributions, where data lie on a union of low-dimensional subspaces. We adopt a simplified yet analytically tractable network architecture that mimics key structural properties of U-Net. Under this setting, we quantify the quality of learned representations via the Signal-to-Noise Ratio ( \mathrm{SNR} ) within the target subspace, enabling us to characterize how representation quality evolves across timesteps in the diffusion process. • Theoretical explanation for the emergence of unimodal dynamics. Leveraging the structures of the MoLRG model, we prove that the unimodal pattern in representation quality naturally arises when the model effectively captures the low-dimensional data distribution. We show that this unimodal pattern is driven by an interplay between denoising strength and class separability that varies with noise levels. There exists an intermediate diffusion timestep where class-irrelevant components are maximally suppressed and class-relevant features are best preserved, resulting in optimal representation quality. • Empirical connection between unimodal dynamics and generalization. Empirically, we demonstrate that the presence of unimodal representation dynamics serves as a reliable indicator of model generalization in classification tasks. Specifically, the unimodal pattern consistently emerges when the model generalizes well, but progressively vanishes as the model shifts toward memorizing the training data. Relationship to prior results. Recent empirical advances in leveraging diffusion models for downstream representation learning have gained significant attention. However, a theoretical understanding of how diffusion models learn representations across different noise levels remains largely unexplored. Here, we focus on the results most relevant to our work and defer a more comprehensive survey to Appendix A . A recent study by [ 26 ] takes initial steps in this direction by analyzing the optimization dynamics of a two-layer CNN trained with diffusion loss on binary class data. Their focus is on contrasting the learning behavior under denoising versus classification objectives, without examining how representation quality evolves across timesteps. In contrast, our work characterizes and compares representations learned at different timesteps, provides a deeper understanding of diffusion-based representation learning and also extends to multi-class settings. A recent study by [ 100 ] also investigates the influence of timesteps in diffusion-based representation learning, focusing on attribute classification and counterfactual generation. In contrast, our work provides a theoretical explanation and practical metric for the emergence of unimodal representation dynamics and shows its relationship with data and model complexity and training iterations. 2 Problem Setup Basics of diffusion models. Diffusion models are a class of probabilistic generative models that aim

Read the complete text