Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model

Xiyuan Wang, Muhan Zhang · 2025

Read Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model by Xiyuan Wang, Muhan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14716.txt.

Introduction

Xiyuan Wang and Muhan Zhang unify latent diffusion into a single end to end trainable network. Standard latent diffusion splits work among an encoder, a decoder, and a diffusion network trained in stages. That split wastes compute and blocks single network designs common in vision foundation models. Naive joint training fails through latent collapse.

The authors trace the collapse to interference between diffusion and representation learning. They draw a novel analogy to self distillation in unsupervised learning. Guided by that analogy, they propose Diffusion as Self-Distillation with key changes to the training objective. The changes stabilize joint training. The result is one model that learns representation and generation together.

Catalog record

CREATORXiyuan Wang, Muhan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14716

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model Xiyuan Wang, Muhan Zhang Abstract Standard Latent Diffusion Models rely on a complex, three-part architecture consisting of a separate encoder, decoder, and diffusion network, which are trained in multiple stages. This modular design is computationally inefficient, leads to suboptimal performance, and prevents the unification of diffusion with the single-network architectures common in vision foundation models. Our goal is to unify these three components into a single, end-to-end trainable network. We first demonstrate that a naive joint training approach fails catastrophically due to “latent collapse”, where the diffusion training objective interferes with the network’s ability to learn a good latent representation. We identify the root causes of this instability by drawing a novel analogy between diffusion and self-distillation based unsupervised learning method. Based on this insight, we propose Diffusion as Self-Distillation (DSD), a new framework with key modifications to the training objective that stabilize the latent space. This approach enables, for the first time, the stable end-to-end training of a single network that simultaneously learns to encode, decode, and perform diffusion. DSD achieves outstanding performance on the ImageNet 256 × 256 256\times 256 conditional generation task: FID=13.44/6.38/4.25 with only 42M/118M/205M parameters and 50 training epochs on ImageNet, without using classifier-free-guidance. 1 Introduction Figure 1 : Analogy between Self-Distillation (SD) based unsupervised learning and the standard diffusion model loss, comparing four architectural paradigms. (1) SD: The Online Encoder ( E_{1} ) generates a latent \mathbf{z}_{1} from an augmented image ( \mathbf{x}^{+} , e.g., with a random mask). \mathbf{z}_{1} is then passed through a predictor ( P ) and trained to match the target latent \mathbf{z}_{2} provided by the Target Encoder ( E_{2} ), which is a frozen copy updated via Exponential Moving Average (EMA). SD is a stable method for producing high-quality latent representations. (2) Latent Diffusion: The encoder ( E ) is frozen, and only the diffusion model ( v_{\theta} ) is trained. The clean image latent \mathbf{z} is used in two ways: input \mathbf{z}_{t} for the diffusion model and target \mathbf{z}-\mathbf{\epsilon} in the \text{L}2 loss. (3) Vanilla Joint Training: Unlike LDM, the encoder ( E ) is unfrozen and optimized with the diffusion loss. This setup leads directly to latent collapse and poor image quality. The reasons for this collapse are structurally identified in this figure (indicated by the colored parts). 4. Our Diffusion As Self-Distillation (DSD): By implementing solutions that fix the identified collapse mechanisms, our DSD framework achieves a unified architecture where the encoder, predictor, and diffusion backbone are all integrated into a single Vision Transformer (ViT) with task-specific heads. The design of modern image generation models, particularly Diffusion Models, has heavily centered on the choice of the latent space. Initial models [ 17 , 36 , 37 ] operated directly in the high-dimensional pixel space. The breakthrough of Latent Diffusion Models (LDMs) [ 1 , 6 , 11 , 22 ] shifted the focus to a more compact, learned latent space provided by a pretrained Variational AutoEncoder (VAE [ 20 ] ). While the VAE component has been continuously refined [ 46 , 34 , 42 , 44 ] , the standard LDM pipeline remains modular, requiring three distinct, often large, components: a pre-trained Encoder, a pre-trained Decoder, and the main Diffusion Model. This modular design introduces several significant problems: • Lack of Unification: The three-component architecture prevents LDMs from being unified with other major unsupervised learning paradigms, which typically use a single core model. Methods like contrastive learning [ 5 , 15 ] and self-distillation [ 2 , 3 , 14 ] form the basis of vision foundation models using a single Transformer [ 41 ] or CNN [ 23 ] . The multi-component nature of LDMs makes them an outlier, hindering their adoption as a standard pre-training task for unified foundation models. • Suboptimal Performance: The multi-stage optimization is suboptimal. The VAE is trained with a reconstruction loss, which does not consider the specific properties required by the downstream diffusion model. On the other hand, most current research focuses on optimizing only one component of the LDM pipeline, such as the autoencoder [ 44 , 46 , 42 ] or the diffusion model [ 43 , 27 , 47 ] , rather than the entire system. • Computational Overhead: Encoder and decoder consume a significant portion of resources. For example, in LDM architectures, the VAE can account for up to 20% of the total parameters and introduce substantial latency [ 48 ] . This motivates our central problem: Can we unify the encoder, decoder, and diffusion model into a single, end-to-end trainable network? This unification would drastically simplify the generation pipeline and bridge the gap between latent diffusion and other unsupervised learning methods. A straightforward method is to simply combine the encoder, decoder, and diffusion model into a single network and train the reconstruction and diffusion tasks simultaneously. However, this attempt fails, as we find in our experiments that the reconstruction loss hardly reduces (as shown in the first image of our Fig. 2 ), resulting in poor reconstruction and generation quality. Previous work REPA-E [ 24 ] found a similar problem and addressed it by simply preventing the diffusion loss’s gradient from backpropagating to the VAE, thereby removing the diffusion loss’s impact on the reconstruction task. However, this does not truly solve our problem, as their method detaches the diffusion loss’s gradient on the latent, preventing the diffusion task from guiding the latent representation learning. In this paper, we first reframe our central problem from an unsupervised learning perspective. As shown in Fig. 1 , the latent diffusion model is similar to self-distillation (SD) based unsupervised learning methods [ 14 ] . Self-distillation method matches the representations of different augmentations of the same data. They both use an encoder whose latent output is used in two branches: one is used directly in the loss, while the other is transformed by a predictor (or the diffusion model) before being used in the loss. Both can use an L2 loss as the objective. Moreover, a major concern that plagues optimization in such unsupervised methods can be used to describe our observed problem: the reconstruction loss failing to decrease during joint training is a symptom of collapse —a rapid reduction in the effective dimensionality of the latent space. As shown in the first image of our Fig. 2 , the diffusion loss also leads to a rapid decrease in the latent space’s effective dimension during training. Therefore, based on previous theoretical analyses of collapse in unsupervised learning, we provide a systematic and quantitative analysis by identifying two root causes tied directly to the structure of the standard diffusion loss (L2 loss fitting the diffusion velocity): • Root Cause 1: Latent Variance Suppression. The L2 loss term implicitly contains a term proportional to the variance of the latent representation. Its gradient compels the encoder to minimize latent variance, forcing the latent vectors to cluster around the mean and causing collapse. • Root Cause 2: Failure of Rank Differentiation. Previous work [ 50 ] shows that self-distillation relies on a rank differentiation mechanism to avoid collapse, which requires the predictor to reduce the effective dimension of its output features. However,

Read the complete text