DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space

Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, Han Cai · 2025

Read DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space by Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, Han Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00413.txt.

Introduction

Junyu Chen and six co-authors upgrade the compression stage of image generators to make training converge faster, in a 2025 paper. Wider latent channels improve reconstruction sharpness. They also slow the diffusion model's learning badly. That trade-off caps quality for high-resolution generation. Two fixes attack the slowdown from different sides.

First, the latent space is trained into a set order. Early channels carry object structure, later ones fine detail. Second, extra training goals focus on those structure channels. The diffusion model then locks onto content sooner. On a standard benchmark at high resolution, quality beat the prior version. Generation also ran four times faster in their setup. Code is public for builders of latent diffusion systems. Higher compression ratios become practical with the change.

Catalog record

CREATORJunyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, Han Cai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00413

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, Han Cai Abstract We present DC-AE 1.5, a new family of deep compression autoencoders for high-resolution diffusion models. Increasing the autoencoder’s latent channel number is a highly effective approach for improving its reconstruction quality. However, it results in slow convergence for diffusion models, leading to poorer generation quality despite better reconstruction quality. This issue limits the quality upper bound of latent diffusion models and hinders the employment of autoencoders with higher spatial compression ratios. We introduce two key innovations to address this challenge: i) Structured Latent Space , a training-based approach to impose a desired channel-wise structure on the latent space with front latent channels capturing object structures and latter latent channels capturing image details; ii) Augmented Diffusion Training , an augmented diffusion training strategy with additional diffusion training objectives on object latent channels to accelerate convergence. With these techniques, DC-AE 1.5 delivers faster convergence and better diffusion scaling results than DC-AE. On ImageNet 512 × \times 512, DC-AE-1.5-f64c128 delivers better image generation quality than DC-AE-f32c32 while being 4 × \times faster. 0 0 footnotetext: Correspondence to: Han Cai ( [email protected] ). 1 Introduction Figure 1 : (a) Training Throughput Comparison under Different Autoencoder Spatial Compression Ratios. Increasing the autoencoder’s spatial compression ratio effectively improves diffusion models’ training efficiency by producing a latent space with fewer tokens. However, larger latent channel numbers are required to maintain satisfactory reconstruction quality. (b) rFID and gFID Results under Different Latent Channel Numbers. We use DiT-XL [ 24 ] as the diffusion model. rFID keeps improving with more latent channels, while gFID keeps getting worse. (c) Efficiency-Quality Trade-off Comparison on ImageNet 512 \times 512. Classifer-free guidance [ 13 ] is not used. DC-AE-1.5-f64c128 delivers 4 \times speedup over DC-AE-f32c32 while maintaining a better image generation quality. Latent Diffusion Model (LDM) [ 25 ] has established itself as a dominant paradigm in high-resolution image synthesis [ 19 , 12 , 32 , 21 ] . It leverages an autoencoder to project images of size H\times W\times 3 into compressed latent representations of shape \frac{H}{f}\times\frac{W}{f}\times c 1 1 1 ‘f’ denotes the autoencoder’s spatial compression ratio. ‘c’ represents the latent channel number of the autoencoder. , reducing the computational cost of the diffusion model. The diffusion model’s outputs are fed to the autoencoder at inference time to reconstruct images from the latent representations. In LDM, the autoencoder reconstruction quality is critical, as it sets a quality upper bound for the whole image synthesis pipeline. A common approach to improve the quality of autoencoder reconstruction is to increase its latent channel number [ 10 , 12 , 19 ] . For example, Figure 1 (b) shows the autoencoder’s rFID (reconstruction FID, the lower the better) results with different latent channel numbers. We can see that the rFID consistently improves as the latent channel number increases, decreasing from 1.60 to 0.26 if switching from c16 to c128. Expanding the latent channel number is especially critical for deep compression autoencoders [ 7 ] , which accelerate latent diffusion models by increasing the autoencoder’s spatial compression ratio (Figure 1 a). Under a high spatial compression ratio (e.g., f64), deep compression autoencoders must use a large latent channel number (e.g., c128) to maintain a satisfactory reconstruction quality. However, using a large latent channel number significantly slows the diffusion model’s convergence, leading to worse gFID (generation FID, the lower the better) results. For example, Figure 1 (b) demonstrates DiT-XL’s [ 24 ] gFID results with different latent channel numbers. Although the autoencoder’s rFID keeps improving, the gFID keeps deteriorating. This issue not only limits LDM’s quality upper bound but also bounds its efficiency as it hinders the employment of autoencoders with high spatial compression ratios (e.g., f64). This work presents DC-AE 1.5, which introduces two key innovations to tackle the aforementioned challenge. First, we analyze the autoencoder’s latent space under different latent channel numbers (Section 3.1 ). We find that the latent space has a sparsity issue when using a large number of latent channels (Figure 2 a). It allocates most of the latent channels for capturing image details while the latent channels for capturing object structure become sparse among the whole latent space 2 2 2 We refer to them as detail latent channels and object latent channels , respectively. . This sparsity issue makes it more difficult for diffusion models to learn object structure, leading to the slow convergence issue. As shown in Figure 2 (b), the image details remain good when we increase the latent channel number. However, the object structures suffer from significant distortion. Motivated by the findings, we propose Structured Latent Space (Section 3.2 ) to alleviate the sparsity issue. It introduces a training-based approach (Figure 4 ) to imposing a specific structure on the latent space: the front latent channels focus on capturing object structure, while the latter latent channels focus on capturing image details (Figure 3 ). Second, based on the Structured Latent Space, we propose Augmented Diffusion Training (Section 3.3 ) to address the slow convergence issue. It introduces extra diffusion training objectives on object latent channels to accelerate diffusion models’ learning speed on capturing object structure (Figure 5 ). With these techniques, we significantly accelerate diffusion models’ convergence when using a large latent channel number (Figure 5 b), leading to better gFID results (Table 2 and 3 ). In addition, it also leads to better diffusion model scaling results than previous autoencoders (e.g., DC-AE-f32c32), as shown in Figure 6 . On ImageNet 512 \times 512, DC-AE-1.5-f64c128+USiT-2B achieves 2.18 gFID without classifier-free guidance, surpassing DC-AE-f32c32+USiT-2B while being 4 \times faster (Figure 1 c). We summarize our contributions as follows: • We analyze the latent space of autoencoders under different latent channel numbers. Our study reveals the latent space’s sparsity issue when using a large latent channel number, which leads to the slow convergence issue for diffusion models. • We propose Structured Latent Space and Augmented Diffusion Training, which effectively address the slow convergence issue. • We build DC-AE 1.5 based on our key innovations. It delivers better diffusion scaling results than prior autoencoders, boosting the quality upper bound of LDMs and paving the way for the employment of autoencoders with higher spatial compression ratios. 2 Related Work Autoencoders in Image Generation. Training diffusion models in high-resolution pixel space is computationally prohibitive due to the large token number. To overcome this challenge, [ 25 ] proposes to employ pretrained autoencoders to produce a compressed latent space and operate diffusion models on the compressed space. This strategy significantly reduces diffusion models’ training and inference costs, making it possible to train large-scale diffusion models on web-scale datasets, leading to impressive image generation results [ 19 , 12 , 21 ] . Following that, one line of works focuses on improving the autoencoders’ reconstruction quality by increasing the latent channel number [ 12 , 10 , 19 ] or adding task-specific priors [ 37 ] . Another line of works focuses on improving diffusion models’ efficiency by increasing autoencoders’ spatial compression ratio [ 7 , 5 , 6 ] . A third line of work [

Read the complete text