Causal Diffusion Transformers for Generative Modeling
Chaorui Deng Deyao Zhu Kunchang Li Shi Guang Haoqi Fan · 2024
Read Causal Diffusion Transformers for Generative Modeling by Chaorui Deng Deyao Zhu Kunchang Li Shi Guang Haoqi Fan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12095.txt.
Introduction
Chaorui Deng and four co-authors merge two leading ways of generating content, diffusion models and autoregressive models. Diffusion models build data by removing noise step by step. Autoregressive models predict one piece at a time, each based on what came before, like language models do. Past attempts to blend the two have met mixed success.
The authors show that adding a step-by-step factorization to a diffusion model can lift its performance a lot. Their model, CausalFusion, is a transformer that factorizes data in two ways at once: across the sequence of tokens, and across noise levels. On the ImageNet image generation benchmark it reached the top score, while keeping the freedom to generate any number of tokens. A joint version also made images and captions together. It could edit images from context with no task-specific training.
Catalog record
| CREATOR | Chaorui Deng Deyao Zhu Kunchang Li Shi Guang Haoqi Fan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Diffusion Transformers for Generative Modeling Chaorui Deng Deyao Zhu Kunchang Li Shi Guang Haoqi Fan Abstract We introduce Causal Diffusion as the autoregressive (AR) counterpart of Diffusion models. It is a next-token(s) forecasting framework that is friendly to both discrete and continuous modalities and compatible with existing next-token prediction models like LLaMA and GPT. While recent works attempt to combine diffusion with AR models, we show that introducing sequential factorization to a diffusion model can substantially improve its performance and enables a smooth transition between AR and diffusion generation modes. Hence, we propose CausalFusion - a decoder-only transformer that dual-factorizes data across sequential tokens and diffusion noise levels, leading to state-of-the-art results on the ImageNet generation benchmark while also enjoying the AR advantage of generating an arbitrary number of tokens for in-context reasoning. We further demonstrate CausalFusion’s multimodal capabilities through a joint image generation and captioning model, and showcase CausalFusion’s ability for zero-shot in-context image manipulations. We hope that this work could provide the community with a fresh perspective on training multimodal models over discrete and continuous data. 1 Introduction Autoregressive (AR) and diffusion models are two powerful paradigms for data distribution modeling. AR models, also known as the next token prediction approach, dominate language modeling and are considered central to the success of large language models (LLMs) [ 46 , 47 , 5 , 61 , 62 , 16 ] . On the other hand, diffusion models [ 26 , 44 , 13 , 29 ] , or score-based generative models [ 54 , 37 ] , have emerged as the leading approach for visual generation, driving unprecedented progress in the era of visual content generation [ 4 , 50 , 17 ] . Figure 1 : Illustration of Dual-Factorization . The arrow line indicates CausalFusion’s generation path, moving from one state to the next by jointly generating along the sequential and noise-level dimension at each step. Compared to DiT, our In-context DiT substantially improves results with fewer parameters. CausalFusion further enhances performance without changing the architecture or parameter count. Results were trained on IN1K for 240 epochs. CausalFusion adopts arbitrary AR steps for image generation, but each step only diffuses partial tokens, resulting in similar (or slightly lower) computational complexity. (a) Samples generated by CausalFusion-XL/2, ImageNet 512 \times 512, 800 epoch, DDPM 250 steps, CFG=4.0 (b) Zero-shot image editing results generated by CausalFusion-XL/2, ImageNet 512 \times 512, 800 epoch. We first generate the original image (those on the left), then mask out its centre region, top-half, or bottom-half, and regenerate the image with new class conditions. Details are discussed in Sec 6 . Figure 2 : Visualization results . All samples are generated by models trained only on ImageNet-1K class-conditional generation task, demonstrating CausalFusion’s zero-shot image manipulation ability. See more visualization results in Appendix D . The intrinsic distinction between AR and diffusion models lies in their approach to data distribution factorization. AR models treat data as an ordered sequence, factorizing it along the sequential axis, where the probability of each token is conditioned on all preceding tokens. This factorization enables the AR paradigm to generalize effectively and efficiently across arbitrary number of tokens, making it well-suited for long-sequence reasoning and in-context generation. In contrast, diffusion models factorize data along the noise-level axis, where the tokens at each step are a refined (denoised) version of themselves from the previous step. As a result, the diffusion paradigm is generalizable to arbitrary number of data refinement steps, enabling iterative quality improvement with scaled inference compute. While AR and diffusion models each excel within their respective domains, their distinct factorization approaches reveal complementary potential. Although recent studies [ 75 , 72 , 21 ] have attempted to integrate AR and diffusion within a single model, they typically treat these paradigms as separate modes, missing the potential benefits of jointly exploring them within a 2-D factorization plane. To this end, we introduce CausalFusion , a flexible framework that integrates both sequential and noise-level data factorization to unify their advantages. The degree of factorization along these two axes—namely, the AR step and diffusion step—is adjustable, enabling CausalFusion to revert seamlessly to the traditional AR or diffusion paradigms at either extreme. To enhance its generality, CausalFusion is designed to predict any number of tokens at any AR step, with any pre-defined sequence order and any level of inference compute, thereby minimizing the inductive biases presented in existing generative models. As shown in Figure 1 , this approach provides a broad spectrum between the AR and diffusion paradigms, allowing smooth interpolation within two endpoints during both training and inference. Specifically, we explore CausalFusion in image generation and multimodal generation scenarios, where we observe that the level of training difficulties significantly influences the overall effectiveness of CausalFusion. Difficulties of generative tasks in CausalFusion: Both AR and diffusion paradigms present unique challenges based on difficulties of their specific generative stages. In diffusion models, the effectiveness of training depends heavily on proper loss weighting across noise levels [ 26 , 22 ] , as higher noise levels are more difficult and usually provide more valuable signals than lower noise levels. Similarly, AR models are susceptible to error accumulation [ 3 ] as early-stage predictions are made with limited visible context, making them more error-prone. Optimizing CausalFusion thus requires balancing across these varying task difficulties to optimize training signal impact and ensure sufficient exploration across the entire factorization plane. In this paper, we formally examine the difficulties of generative tasks within CausalFusion. We show that, in addition to the noise levels in diffusion and the amount of visible context in AR, the total number of AR steps, which controls the interpolation between AR and diffusion, also plays a critical role in shaping training difficulties. Driven by these factors, we develop a scalable and versatile model based on the CausalFusion framework. Starting from the DiT architecture [ 44 ] , we gradually convert it into a decoder-only transformer compatible with existing AR models like GPT [ 46 , 47 , 5 ] and LLaMA [ 61 , 62 , 16 ] . We provide insights on how to appropriately choose the number of AR steps during the training of CausalFusion models, and further introduce loss weighing along both the diffusion and AR axis to balance the impact of different generative stages. As shown in Figure 1 and 2 , our model achieves state-of-the-art performance on the ImageNet class-conditional generation benchmark, significantly outperforming DiT [ 44 ] and enabling zero-shot image manipulations due to its AR nature. When pretraining on both text-to-image and image-to-text tasks, our model surpasses forced-fusion frameworks such as TransFusion [ 75 ] , demonstrating the versatility of our CausalFusion framework. We highlight our main contribution below: • We propose CausalFusion as the AR counterpart to DiT, achieving state-of-the-art results and enabling the unlimited token generation for in-context reasoning. • We systematically study CausalFusion on the dual-factorization plane and identify key factors that improve the effectiveness of CausalFusion models. • Compared with recent studies [ 75 ] , CausalFusion enables a smooth, cohesive integration with language modeling for cross-modal generation and reasoning. 2 Related Works Diffusion Models . Diffusion models [ 52 , 53 , 26 ]