xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, Jiannan Wang · 2024
Read xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism by Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, Jiannan Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01738.txt.
Description
A 2024 arXiv research paper by Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, Jiannan Wang. Diffusion models are pivotal for generating high-quality images and videos. Inspired by the success of OpenAI’s Sora ( 32 ) , the backbone of diffusion models is evolving from U-Net to Transformer, known as Diffusion Transformers (DiTs). However, generating high-quality content necessitates longer sequence lengths, exponentially increasing the computation required for the attention mechanism, and escalating DiTs inference latency.
Catalog record
| CREATOR | Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, Jiannan Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01738
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, Jiannan Wang Abstract . Diffusion models are pivotal for generating high-quality images and videos. Inspired by the success of OpenAI’s Sora ( 32 ) , the backbone of diffusion models is evolving from U-Net to Transformer, known as Diffusion Transformers (DiTs). However, generating high-quality content necessitates longer sequence lengths, exponentially increasing the computation required for the attention mechanism, and escalating DiTs inference latency. Parallel inference is essential for real-time DiTs deployments, but relying on a single parallel method is impractical due to poor scalability at large scales. This paper introduces xDiT, a comprehensive parallel inference engine for DiTs. After thoroughly investigating existing DiTs parallel approaches, xDiT chooses Sequence Parallel (SP) and PipeFusion, a novel Patch-level Pipeline Parallel method, as intra-image parallel strategies, alongside CFG parallel for inter-image parallelism. xDiT can flexibly combine these parallel approaches in a hybrid manner, offering a robust and scalable solution. Experimental results on two 8 × \times L40 GPUs (PCIe) nodes interconnected by Ethernet and an 8 × \times A100 (NVLink) node showcase xDiT’s exceptional scalability across five state-of-the-art DiTs. Notably, we are the first to demonstrate DiTs scalability on Ethernet-connected GPU clusters. xDiT is available at https://github.com/xdit-project/xDiT . 1. Introduction In recent years, diffusion models have emerged as a groundbreaking technique in image ( 4 , 10 , 5 , 34 ) and video generation ( 32 , 31 ) . These models create images or videos through a multi-step denoising process, leveraging a neural network model at each step. We are witnessing a transformative shift in the architecture of the denoising networks. While traditionally dominated by U-Net ( 38 ) architectures, these networks are now evolving into Diffusion Transformers (DiTs) ( 34 ) , renowned for their superior model capacity and scalability. Images are encoded into a token sequence as input to DiTs ( 37 ) . The attention mechanisms of DiTs require mutual computation between tokens, leading to computation that scales quadratically with sequence length. Unlike Large Language Models (LLMs), which can generate tokens on the fly, DiTs must complete all steps to produce the final results. This poses significant challenges for real-time deployment. For example, a Sora-like video generation API ( 1 , 20 ) takes over 4 minutes to generate a fewer seconds video. Given the immense computational demands, real-time DiTs deployment inevitably necessitates parallelism across multiple computing devices. However, there is still a lack of effective methodology to scale DiTs to large scales. Although, several sequence parallelism (SP) methods ( 17 , 26 , 12 , 14 ) have been developed to scale long-sequence DiTs inference. Additionally, some approaches leverage input temporal redundancy ( 40 , 30 , 29 , 45 , 46 , 31 ) , indicating a high degree of similarity in both inputs and activations across successive diffusion time steps, to design asynchronous sequence parallelism ( 22 ) and patch-level sequence parallelism ( 11 ) . However, using these methods in isolation fails to adapt to the underlying heterogeneous interconnects of computational devices. For instance, methods using collective communication are only suitable for high-bandwidth interconnects like NVLink, while methods using P2P (Peer-to-Peer) are more suitable for PCIe or Ethernet but do not have an advantage on NVLink. Additionally, the diversity of DiTs model architectures presents challenges for parallel implementation. Unlike LLMs, which generally have a more uniform architecture, DiTs exhibit greater variability. For instance, the method of injecting user input instructions, the connection methods of DiTs blocks, and the operators’ layouts in Transformer blocks can vary significantly. This diversity means that directly applying methods designed for LLMs, such as TP and SP, may not be immediately suitable for DiTs. This paper introduces xDiT, a parallel inference system designed for DiTs. We argue that DiTs inference is analogous to the training of LLMs in that a single parallel is unlikely to scale effectively across various model architectures and network hardware, especially for those heterogeneous low-bandwidth networks. Therefore, a hybrid approach combining multiple parallel methods is necessary to achieve optimal communication efficiency. Our contributions are as follows: • We systematically investigate existing parallel methods for DiTs (including two types of sequence parallelism, DistriFusion, tensor parallelism, and PipeFusion), • We selected PipeFusion and sequence parallelism for intra-image parallelism and CFG parallelism for inter-image parallelism. We designed a method to correctly hybrid these parallel approaches, thereby scaling DiTs inference to a very large scale. • Using the aforementioned methods, we built a system called xDiT. On 16 \times L40 PCIe and 8 \times A100 NVLink devices, we studied the scalability of five image and video generation DiTs (Pixart, Stable-Diffusion3, Flux.1, HunyuanDiT, CogVideoX). xDiT is the first system to successfully scale DiTs inference to 16 GPUs. 2. Background & Related Works Diffusion Models : Diffusion models utilize a noise-prediction deep neural network (DNN) denoted by \epsilon_{\theta} to generate a high-quality image. The process starts from pure Gaussian noise x_{T}\sim\mathcal{N}(0,I) and involves numerous iterative denoising steps to produce the final meaningful image x_{0} , with T representing the total number of diffusion time steps. At each diffusion time step t , given the noisy image x_{t} , the model \epsilon_{\theta} takes x_{t} , t , and an additional condition c (e.g., text, image) as inputs to predict the corresponding noise \epsilon_{t} within x_{t} . At each denoising step, the previous image x_{t-1} can be obtained from the following equation: (1) x_{t-1}=\text{Update}(x_{t},t,\epsilon_{t}),\quad\epsilon_{t}=\epsilon_{\theta}(x_{t},t,c). In this context, Update denotes a function that is specific to the sampler, i.e. DDIM ( 41 ) and DPM ( 27 ) , generally involves operations such as element-wise operations. After multiple steps, we decode x_{0} from the Latent Space to the Pixel Space using a Variational Autoencoder (VAE) ( 18 ) . Consequently, the predominant contributor to diffusion model inference latency is attributed to the forward propagation through the model \epsilon_{\theta} . Diffusion Transformers (DiTs) : The architecture of diffusion model \epsilon_{\theta} is undergoing a pivotal transition from U-Net ( 38 ) to Diffusion Transformers (DiTs) ( 31 , 28 , 6 , 2 , 23 ) , driven by the scaling law demonstrating increased model parameters and training data with enhanced model performance. Unlike U-Nets, which apply convolutional layers capturing spatial hierarchies, DiTs encodes the input into a token sequence in the latent space and leverage the transformer’s self-attention mechanism to model relationships within and across these patches. As shown in the above of Figure 2 , in DiTs, the input noisy latent representation is embedded into tokens and fed into a series of DiT blocks. DiT blocks generally incorporate Multi-Head Self-Attention, Layer Norm, and Pointwise Feedforward Networks. Although the earliest DiTs model architectures ( 34 ) was quite similar to standard transformers ( 42 ) , numerous improvements and variants have been introduced recently, leading to a highly diverse and non-uniform model architecture landscape. For instance, the incorporation of conditioning can be achieved through various methods such as adaptive layer norm ( 34 ) , cross-attention ( 6 ) , and extra input tokens ( 10 ) . As diffusion models tackle higher-resolution images and longer visual sequences, they impose a quadratic computational burden on inference. Input Temporal