Minute-Long Videos with Dual Parallelisms
https://dualparal-project.github.io/dualparal.github.io/, Zeqing Wang, Bowen Zheng, Xingyi Yang, Zhenxiong Tan, Yuecong Xu, Xinchao Wang · 2025
Read Minute-Long Videos with Dual Parallelisms by https://dualparal-project.github.io/dualparal.github.io/, Zeqing Wang, Bowen Zheng, Xingyi Yang, Zhenxiong Tan, Yuecong Xu, Xinchao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21070.txt.
Introduction
Zeqing Wang and co-authors present a way to generate minute-long videos across many graphics cards, in a 2025 paper. Video diffusion models make fine clips but run slowly and eat memory on long pieces. Their method splits the work two ways at once. Frames and model layers are spread over separate cards.
A naive split stalls, since all frames must share the same noise schedule. Their fix processes blocks of frames in a pipeline. Each block moves through the cards with falling noise levels. Cards can then compute and pass data at the same time. Two extra tweaks lift speed further. Long, high-quality video from a card cluster could serve film and simulation work.
Catalog record
| CREATOR | https://dualparal-project.github.io/dualparal.github.io/, Zeqing Wang, Bowen Zheng, Xingyi Yang, Zhenxiong Tan, Yuecong Xu, Xinchao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Minute-Long Videos with Dual Parallelisms https://dualparal-project.github.io/dualparal.github.io/, Zeqing Wang, Bowen Zheng, Xingyi Yang, Zhenxiong Tan, Yuecong Xu, Xinchao Wang Abstract Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos. To address this, we propose a novel distributed inference strategy, termed DualParal . The core idea is that, instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs. However, a naive implementation of this division faces a key limitation: since diffusion models require synchronized noise levels across frames, this implementation leads to the serialization of original parallelisms. We leverage a block-wise denoising scheme to handle this. Namely, we process a sequence of frame blocks through the pipeline with progressively decreasing noise levels. Each GPU handles a specific block and layer subset while passing previous results to the next GPU, enabling asynchronous computation and communication. To further optimize performance, we incorporate two key enhancements. Firstly, a feature cache is implemented on each GPU to store and reuse features from the prior block as context, minimizing inter-GPU communication and redundant computation. Secondly, we employ a coordinated noise initialization strategy, ensuring globally consistent temporal dynamics by sharing initial noise patterns across GPUs without extra resource costs. Together, these enable fast, artifact-free, and infinitely long video generation. Applied to the latest diffusion transformer video generator, our method efficiently produces 1,025-frame videos with up to 6.54 × \times lower latency and 1.48 × \times lower memory cost on 8 × \times RTX 4090 GPUs. 1 Introduction Diffusion Transformer (DiT) [ 18 ] has significantly improved the scalability of video diffusion models [ 13 , 27 , 19 ] , enabling more realistic and higher-resolution video generation. Despite its benefits, large-scale DiT models suffer from their inherent computational inefficiency. This directly results in extended processing durations and memory demands, especially for real-time deployment [ 5 ] . Notably, this inefficiency is further exacerbated when generating long videos. Intuitively, longer videos increase the input sequence length. This has severe implications for latency: the attention mechanism, core to DiT [ 5 , 6 ] , exhibits time complexity that scales quadratically with sequence length. Concurrently, memory consumption escalates substantially due to the combination of a large number of model parameters and the extended video sequences. Therefore, enabling DiT-based models to efficiently generate high-quality long videos remains a formidable and pressing challenge. Recently, parallelization has emerged as a promising solution for efficient long video generation. It uses multiple devices to produce video jointly, which scales memory and boosts processing speed. Among existing strategies, sequence parallelism reduces latency by synchronously processing split hidden [ 11 , 17 ] or input [ 25 , 12 ] sequences using a full model replica on each device. However, they incurs high memory overhead due to the entire model on every device [ 25 , 12 ] . In contrast, pipeline parallelism [ 6 ] mitigates memory usage by partitioning the model across devices as a device pipeline [ 9 , 16 , 24 ] . Therefore, an ideal solution would combine the sequence parallelism with pipeline parallelism to maximize speed and minimize memory usage. However, naively combining sequence and pipeline parallelism is fundamentally conflicting. The core issue stems from the inherent synchronization property of video diffusion models: all input tokens must pass through an entire layer together before any can move on. In pipeline parallelism, this means the full input must finish processing on one device (e.g., Device 1) before passing to the next (e.g., Device 2). This requirement directly contradicts sequence parallelism, which splits the input across devices. As a result, all distributed parts must be gathered back onto a single device for serialized processing on specific model layers. Only then can all parts enter the next pipeline stage, i.e. next device. This repeated gathering serializes computation and negates the benefits of sequence parallelism, reintroducing a serial bottleneck and significant communication overhead. To address this conflict, we propose a novel distributed inference strategy, termed DualParal . At a high level, DualParal divides both the video sequence and model into chunks and applies parallel processing across both. As discussed above, a naive combination presents challenges. Inspired by recent work on interpolating diffusion and autoregressive models [ 1 , 23 ] , we make this feasible by implementing a block-wise denoising scheme for video diffusion models. By the word block-wise , we refer to a strategy where, instead of denoising all frames at a uniform noise level, we divide the video into non-overlapping temporal blocks. Each block is assigned a different noise level according to its position in the video: blocks closer to the end have higher noise levels, while earlier blocks receive lower noise levels. During each inference step, the model processes all blocks asynchronously, incrementally reducing their respective noise levels. Crucially, because noise levels do not need to be synchronized across all frames, block-wise denoising resolves the inherent conflict between the two parallelism strategies. Accordingly, we tailor this inference scheme for multiple devices in conjunction with our DualParal . Specifically, we organize video sequence blocks in a first-in-first-out (FIFO) queue [ 12 , 22 ] , where noise levels decrease from tail to head. In each diffusion step, a new noisy block is appended to the tail, while a clean block is removed from the head. These video blocks are then processed in reverse order, tail to head, through a device pipeline. In this setup, each device handles a specific video block and a model chunk, with denoised outputs passed asynchronously between GPUs. This distributed architecture enhances memory efficiency by distributing the model and achieves near-zero idle time through asynchronous block processing and communication. Even more compelling, DualParal leverages its FIFO queue to enable long video generation. New blocks can be continuously appended to the queue, allowing for producing arbitrarily long videos. Because the number of frames within each block remains fixed, this approach again avoids quadratic increase in processing latency and high memory costs associated with extended video sequences. To further optimize parallel efficiency and maintain video quality, we introduce two key enhancements for DualParal . Firstly , to ensure coherence between adjacent blocks, each block is concatenated with parts of previous and subsequent blocks before processed in the device pipeline, resulting extra resource costs. To reduce this, DualParal employs a feature cache on each GPU that stores and reuses Key-Value (KV) features from the previous block without explicitly concatenating it. This reduce inter-GPU communication and redundant computation in components like Cross-Attention [ 26 , 27 ] and Feed-Forward Networks (FFN) [ 26 ] in the latest video diffusion model Wan2.1 [ 27 ] . Secondly , to maintain global consistency across blocks without extra resource costs for global information, new block is initialized with a coordinated noise space, avoiding performance degradation caused by repetitive noise, all without extra cost. Together, these enable fast, artifact-free, and infinite-length video generation. In summary, our contributions are summarized as follows: (1) We design an efficient distributed inference strategy by parallelizing both the video sequence and model layers, operating under a block-wise denoising scheme, to minimize idle time and optimize both computation and memory