Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation

Xin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, Huan Yang · 2024

Read Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation by Xin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, Huan Yang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01316.txt.

Introduction

Xin Yan and co-authors present Presto, a 2024 video diffusion model built to make fifteen-second clips that stay coherent. Long videos tend to drift or repeat scenes. Their fix splits the hidden states into time segments. Each segment attends to its own sub-caption, guiding that stretch of the story.

The trick adds no new parameters and fits current transformer video models. To train it, the team built a data set of 261,000 rich videos. Each clip holds one full caption plus five step-by-step sub-captions. Tests scored Presto high on meaning and scene dynamism. Richer long clips serve film previsualization and simulation alike.

Catalog record

CREATORXin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, Huan Yang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation Xin Yan, Yuxuan Cai, Qiuyue Wang, Yuan Zhou, Wenhao Huang, Huan Yang Abstract We introduce Presto , a novel video diffusion model designed to generate 15-second videos with long-range coherence and rich content. Extending video generation methods to maintain scenario diversity over long durations presents significant challenges. To address this, we propose a Segmented Cross-Attention (SCA) strategy, which splits hidden states into segments along the temporal dimension, allowing each segment to cross-attend to a corresponding sub-caption. SCA requires no additional parameters, enabling seamless incorporation into current DiT-based architectures. To facilitate high-quality long video generation, we build the LongTake-HD dataset, consisting of 261k content-rich videos with scenario coherence, annotated with an overall video caption and five progressive sub-captions. Experiments show that our Presto achieves 78.5% on the VBench Semantic Score and 100% on the Dynamic Degree, outperforming existing state-of-the-art video generation methods. This demonstrates that our proposed Presto significantly enhances content richness, maintains long-range coherence, and captures intricate textual details. More details are displayed on our project page: presto-video.github.io . 1 Introduction Video diffusion models Ho et al. 2022 ; Singer et al. 2022 ; Voleti et al. 2022 ; Ruan et al. 2023 ; Wang et al. 2024a have shown an impressive ability to generate high-quality videos based on a single text prompt Saharia et al. 2022 ; Nichol et al. 2021 . However, most current approaches primarily focus on generating short video clips ranging from 3 to 8 seconds, limiting the expressiveness and richness of the resulting content. To generate longer videos, early approaches incorporate an additional interpolation or extrapolation phase to extend short clips, using techniques like noise scheduling Wang et al. 2023a ; Qiu et al. 2023 ; Kim et al. 2024 or attention manipulation Lu et al. 2024 ; Tan et al. 2024 . While these methods work well for generating minute-long videos, they struggle to extend beyond the scene content, as they are constrained by the limited capacity of the original short clips. An alternative approach adopts a more direct strategy, typically by adding new modules to extend the video length in an auto-regressive manner Yin et al. 2023 ; Henschel et al. 2024 ; Wang et al. 2024b . However, this introduces the challenge of error propagation. Unlike short clips, long videos require a balance between content diversity and long-range coherence, posing significant challenges for current video generation methods. To generate long videos with rich content, we recognize the importance of expanding the text input by incorporating multiple texts, as demonstrated in previous approaches Oh et al. 2023 ; Bansal et al. 2024 . While combining videos generated from each text can enhance content diversity, it often leads to abrupt transitions between different scenarios. One alternative is to incorporate multiple texts into the video generation model simultaneously. This method provides the model with a broader range of textual inputs, enabling the generation of more content-rich videos, in contrast to the traditional approach of using a single text input, which limits the available information. This approach helps ensure long-term coherence in the generated videos by modeling the texts concurrently, providing a seamless viewing experience. Existing long video generation methods overlook the importance of high-quality data Bansal et al. 2024 ; Henschel et al. 2024 , leading to low consistency and content diversity in generated videos. A large-scale, high-quality video dataset is crucial to obtaining a model that generates content-rich and coherent long videos. This dataset should include long videos with long-range coherence and complex dynamics, along with multiple distinct yet coherent textual descriptions for each video. To this end, we develop a systematic data curation pipeline to collect content-rich video-prompt pairs from public datasets, contributing to creating our LongTake-HD dataset. We filter 261k single-scene video clips from 8.9M publicly accessible videos as the pre-training set. Then, we apply additional meticulous filtering steps to select the finest instances, resulting in a fine-tuning dataset of 47k clips. This refined dataset ensures that our method can generate high-quality, extended-duration, and content-rich videos. To tackle the challenge of long videos, we propose Presto , a novel method capable of generating 15-second videos with rich content and long-range coherence , as shown in . Instead of using a single long caption for each video, we divide the visual content into segments and generate progressive sub-captions that align with the unfolding storyline. Next, we modify the cross-attention mechanism in the Diffusion Transformer (DiT). To adapt the DiT model for long video generation, we refine the text embedding process and the cross-attention mechanism to effectively handle multiple progressive text conditions alongside temporal information. In particular, we introduce Segmented Cross-Attention (SCA) , which divides the hidden states into segments along the temporal dimension and cross-attends each segment with its corresponding sub-caption. SCA introduces no additional parameters or modules and can seamlessly integrate into existing DiT-based methods with minimal fine-tuning. We explore three distinct SCA strategies to manage the interaction between text embeddings and segmented latent features: Isolate Segmented Cross-Attention (ISCA), Sequential Segmented Cross-Attention (SSCA), and Overlap Segmented Cross-Attention (OSCA). Our experiments demonstrate that OSCA enhances the content richness and long-range coherence in the generated long videos. Experimental results demonstrate the effectiveness of our methods. Presto achieves a 78.5% score on the VBench Semantic Score, outperforming both the leading open-source model, Allegro, and the commercial system, Gen-3. Notably, Presto achieves a perfect 100% score on the Dynamic Degree metrics, showcasing its outstanding ability to capture dynamics and transitions. The quantitative results highlight our approach’s strength in capturing intricate textual details and generating videos with rich content. Furthermore, our user study indicates that Presto excels in scenario diversity, scenario coherence, and text-video alignment when compared to various open-source and commercial-level works. Our key contributions are outlined as follows: • We propose a large-scale video dataset LongTake-HD , with 261k high-quality cases curated from publicly sourced videos, characterized by long-duration, content-rich, and long-range coherent videos, and each paired with progressive sub-captions. • We proposed Presto for long video generation with a simple yet effective Segmented Cross-Attention strategy which extends the DiT architecture to accommodate multiple text prompts, enabling the generation of videos with rich content and long-range coherence. 2 Related Work Long-Video Generation is a challenging task in video generation, requiring videos to be both rich and coherent. One paradigm Lu et al. 2024 ; Tan et al. 2024 ; Wang et al. 2023a ; Qiu et al. 2023 ; Kim et al. 2024 is to modify noise scheduling or attention mechanisms during the inference stage, which are highly constrained by the original video clips, often resulting in limited content diversity. Another approach Yin et al. 2023 ; Henschel et al. 2024 ; Wang et al. 2024b directly generates long videos, by introducing additional modules with auto-regressive generation, requiring substantial training resources. Multiple-Text-to-Video Generation (MT2V) Bansal et al. 2024 ; Oh et al. 2023 ; Ramos et al. 2024 ; Villegas et al. 2022 aligns with our work, as we also incorporate multiple text prompts. While MT2V aims to create videos from multiple inputs, our model uniquely adheres to the traditional T2V framework by requiring only a single user prompt. Additionally, MT2V methods such

Read the complete text