Elucidating Representation Degradation Problem in Diffusion Model Training

Zhipeng Yao, Dazhou Li, Zitong Zhang, Durude Mahee, Fan Zhu, Wenbin Zhang Xinwei He Yeying Jin Rui Yu · 2026

Read Elucidating Representation Degradation Problem in Diffusion Model Training by Zhipeng Yao, Dazhou Li, Zitong Zhang, Durude Mahee, Fan Zhu, Wenbin Zhang Xinwei He Yeying Jin Rui Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10790.txt.

Introduction

Zhipeng Yao and eight colleagues diagnose a training bottleneck in diffusion models, the generators behind much of today's image synthesis. They call it representation degradation: as noise levels rise, model outputs warp structurally, which destabilizes training and degrades what the model makes.

Their analysis ties the trouble to mismatched target recoverability, linked to weakening spectral structure in the neural tangent kernel. Their fix, Elucidated Representation Diffusion, shifts optimization effort toward noise levels where targets can still be recovered. Across diffusion backbones it speeds convergence without outside supervision. Model builders get a plug-and-play remedy for a failure mode they may have blamed on tuning.

Catalog record

CREATORZhipeng Yao, Dazhou Li, Zitong Zhang, Durude Mahee, Fan Zhu, Wenbin Zhang Xinwei He Yeying Jin Rui Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120369
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10790

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Elucidating Representation Degradation Problem in Diffusion Model Training Zhipeng Yao, Dazhou Li, Zitong Zhang, Durude Mahee, Fan Zhu, Wenbin Zhang Xinwei He Yeying Jin Rui Yu Abstract Diffusion models have achieved remarkable success, yet their training remains inefficient due to a severe optimization bottleneck, which we term Representation Degradation. As noise levels increase, the outputs of the trained model exhibit progressive structural distortion, which can destabilize training and impair generation quality. Our analysis suggests that this instability is driven by mismatched target recoverability, which is associated with Neural Tangent Kernel (NTK) spectral weakening and effective low-rank behavior. To address this, we propose Elucidated Representation Diffusion (ERD), a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability. By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones. 1 Introduction Diffusion models [ 50 , 14 , 52 ] have become a dominant paradigm for generative modeling [ 34 , 16 , 28 , 67 ] . As the field increasingly shifts toward scalable Transformer-based architectures [ 43 , 2 ] , improving training efficiency has become a central challenge. Existing methods often accelerate training by heuristically reallocating the optimization effort across noise levels [ 10 , 23 , 55 , 65 , 11 , 66 ] , but they remain largely agnostic to the quality of the internal features learned. More recently, Representation Alignment (REPA) [ 62 ] has shown that aligning diffusion features with pre-trained priors can substantially accelerate training. This empirical success raises a fundamental question: “what intrinsic bottleneck makes standard diffusion training inefficient in the first place?” To answer this question, we empirically examine how the output of the trained model evolves across noise levels. We find that although the predicted geometry aligns well with the target distribution at low noise levels, it becomes increasingly distorted and eventually collapses under high noise. As illustrated in Fig. 1 , the predicted distribution, when projected onto a 2D plane, preserves meaningful structure at low noise but loses it severely at high noise. We refer to this phenomenon as Representation Degradation [ 7 ] . These observations reveal an intrinsic instability in diffusion training, demonstrating that as noise increases, the model becomes progressively less able to preserve semantically meaningful structure. To explain this phenomenon, we analyze diffusion training in the Neural Tangent Kernel (NTK) regime as a continuous-time, noise-conditioned regression process [ 19 , 25 ] . Our analysis shows that optimization at each noise level is jointly shaped by three factors: the allocation of optimization effort, the local kernel geometry induced by corrupted inputs, and the recoverability of the target itself. As noise increases, corrupted inputs lose informative structure, which weakens the local NTK spectrum and reduces learnability along semantically meaningful directions. Only a small fraction of the target remains effectively recoverable within finite training time, while standard training may still allocate substantial optimization effort to these weakly recoverable regions [ 3 ] . This mismatch is consistent with spectral degeneration [ 44 ] , effective low-rank behavior, and contributes to representation degradation [ 21 ] in the extreme-noise regime. Given \bm{x}_{t} input network, where t\in 0\to T Figure 1 : We visualized pre-trained model predictions as forward diffusion progressively corrupts the inputs. Fig. 1(a) shows that the image distribution becomes increasingly stretched and flattened, while Fig. 1(b) shows the predicted noise shifting from an isotropic to an anisotropic distribution. (a) x_{\theta} -prediction. (b) \epsilon_{\theta} -prediction. These findings identify representation degradation as an intrinsic optimization bottleneck rather than a purely architectural limitation. To address it, we propose Elucidated Representation Diffusion (ERD) , a training framework that reallocates optimization effort according to target recoverability without relying on external alignment networks. Specifically, ERD adopts a target-adaptive weighting rule that emphasizes noise levels with stronger recoverable signal and downweights regions that receive excessive optimization mass despite limited learnability. In this way, ERD improves optimization stability, mitigates representation degradation, and accelerates convergence. Our main contributions are summarized as follows: • Representation Degradation: We identify Representation Degradation in diffusion training, showing that model outputs undergo substantial structural distortion and collapse as noise increases, reflecting an instability in the learned representations. • Optimization Analysis: We develop an NTK-based framework showing how allocation, local kernel geometry, and target recoverability jointly induce spectral degeneration and effective low-rank collapse in high-noise regimes. • Elucidated Representation Diffusion: We propose an architecture-agnostic training framework that derives a target-adaptive weighting rule from recoverable signal at each noise level, improving optimization stability and accelerating diffusion training without external priors. 2 Preliminaries In this section, we introduce the diffusion formulation in the log-signal-to-noise ratio (log-SNR) domain and the continuous-time Neural Tangent Kernel framework used in subsequent analysis. 2.1 Diffusion Models in the Log-SNR Domain Let x_{0}\sim q(x_{0}) be a data sample in \mathbb{R}^{d} . The forward diffusion process generates latent variables through the transition kernel q(x_{t}\mid x_{0})=\mathcal{N}(x_{t};\alpha_{t}x_{0},\sigma_{t}^{2}I) . We reparameterize the process by the log-SNR \lambda\triangleq\log\!\left(\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}\right) . We assume that the diffusion schedule is chosen so that the map t\mapsto\lambda_{t} is monotone on the interval of interest, allowing t and \lambda to be used interchangeably. Under this parameterization, the corrupted variable can be written as x_{\lambda}=\alpha(\lambda)x_{0}+\sigma(\lambda)\epsilon , where \epsilon\sim\mathcal{N}(0,I) . We consider a generalized target of the form y_{\lambda}=c_{x}(\lambda)x_{0}+c_{\epsilon}(\lambda)\epsilon , which subsumes standard parameterizations such as x_{\theta} -prediction, \epsilon_{\theta} -prediction, and v_{\theta} -prediction. Following the continuous-time variational formulation of diffusion training [ 27 , 25 ] , we consider the corresponding weighted training objective \mathcal{L}_{\mathrm{ELBO}}(\theta)=\int_{\lambda_{\min}}^{\lambda_{\max}}M(\lambda)\left(\frac{1}{2}\mathbb{E}_{x_{0},\epsilon}\!\left[\|f_{\theta}(x_{\lambda},\lambda)-y_{\lambda}\|_{2}^{2}\right]\right)\mathrm{d}\lambda, (1) where f_{\theta} is parameterized by \theta\in\mathbb{R}^{P} , and M(\lambda)\triangleq w(\lambda)p(\lambda)\left|\frac{\mathrm{d}t}{\mathrm{d}\lambda}\right| denotes the effective allocation measure induced by the loss weighting, the sampling distribution over noise levels, and the schedule Jacobian. Throughout the paper, M(\lambda) serves as a compact description of how optimization effort is distributed across the log-SNR domain. 2.2 Continuous-Time Dynamics and the Neural Tangent Kernel We study Eq. ( 1 ) from a continuous-time optimization perspective. Let \tau\geq 0 denote training time. Under gradient flow, the parameters evolve according to \frac{\mathrm{d}\theta(\tau)}{\mathrm{d}\tau}=-\nabla_{\theta}\mathcal{L}_{\mathrm{ELBO}}(\theta) . Because diffusion models use a shared-parameter architecture across all noise levels, optimization at one noise level generally interacts with optimization at others. This motivates describing the training process on the joint input–noise space rather than treating each noise level independently. For this purpose, we introduce the matrix-valued joint Neural Tangent Kernel (NTK) \Theta\big((x,\lambda),(x^{\prime},\lambda^{\prime})\big)=\big(\nabla_{\theta}f_{\theta}(x,\lambda)\big)\big(\nabla_{\theta}f_{\theta}(x^{\prime},\lambda^{\prime})\big)^{\top}\in\mathbb{R}^{d\times d}, (2) where \nabla_{\theta}f_{\theta}(x,\lambda)\in\mathbb{R}^{d\times P} denotes the Jacobian of the network output with respect to the parameters. This joint kernel acts as a linear operator on \mathbb{R}^{d} and captures both coupling across output dimensions and cross-noise interactions induced by parameter sharing. In particular, it provides a convenient framework for describing how training signals at different noise levels interact along a shared optimization trajectory. In the overparameterized regime [ 19 ] , the NTK perspective offers a useful approximation for analyzing training dynamics. In the following sections, we will build on this viewpoint to study how optimization allocation, input corruption, and target structure jointly influence diffusion training across noise levels. 3 Elucidating Representation Degradation To explain standard diffusion training inefficiency, we study overparameterized continuous-time dynamics. By connecting NTK dynamics with the global ELBO, we formalize target-dependent recoverability mismatch : optimization effort may be over-allocated to weakly recoverable signals. Under our surrogate dynamics,

Read the complete text