Raccoon: Multi-stage Diffusion Training with Coarse-to-Fine Curating Videos
Zhiyu Tan, Junyan Wang, Hao Yang, Luozheng Qin, Hesen Chen [2pt], Qiang Zhou, Hao Li, [6pt] Fudan University The University of Adelaide INF Tech · 2025
Read Raccoon: Multi-stage Diffusion Training with Coarse-to-Fine Curating Videos by Zhiyu Tan, Junyan Wang, Hao Yang, Luozheng Qin, Hesen Chen [2pt], Qiang Zhou, Hao Li, [6pt] Fudan University The University of Adelaide INF Tech (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-21314.txt.
Introduction
Zhiyu Tan and co-authors present Raccoon, a 2025 text-to-video system built as much on data cleaning as on architecture. Video generators are limited by the messiness of their training sets and by heavy compute costs.
Their pipeline first curated a million-clip data set in two passes: broad quality scoring, then vision-language checks for text match and meaning. On that data they trained a transformer with separate attention for space and time, in four progressive stages. The result is appealing, coherent video at lower compute cost. The team plans to release the data set, code, and models for others to build on.
Catalog record
| CREATOR | Zhiyu Tan, Junyan Wang, Hao Yang, Luozheng Qin, Hesen Chen [2pt], Qiang Zhou, Hao Li, [6pt] Fudan University The University of Adelaide INF Tech |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-21314
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Raccoon: Multi-stage Diffusion Training with Coarse-to-Fine Curating Videos Zhiyu Tan, Junyan Wang, Hao Yang, Luozheng Qin, Hesen Chen [2pt], Qiang Zhou, Hao Li, [6pt] Fudan University The University of Adelaide INF Tech Abstract Text-to-video generation has demonstrated promising progress with the advent of diffusion models, yet existing approaches are limited by dataset quality and computational resources. To address these limitations, this paper presents a comprehensive approach that advances both data curation and model design. We introduce CFC-Vids-1M , a high-quality video dataset constructed through a systematic coarse-to-fine curation pipeline. The pipeline first evaluates video quality across multiple dimensions, followed by a fine-grained stage that leverages vision-language models to enhance text-video alignment and semantic richness. Building upon the curated dataset’s emphasis on visual quality and temporal coherence, we develop Raccoon , a transformer-based architecture with decoupled spatial-temporal attention mechanisms. The model is trained through a progressive four-stage strategy designed to efficiently handle the complexities of video generation. Extensive experiments demonstrate that our integrated approach of high-quality data curation and efficient training strategy generates visually appealing and temporally coherent videos while maintaining computational efficiency. We will release our dataset, code, and models. Table 1: Comparison of our dataset and other popular video-text datasets. We’ve compiled statistics on video-text datasets, specifically analyzing metrics such as the domain, the number of video clips after scene detection, average duration, average text length. Dataset Venue & Year Text Domain Clip Num Avg Video Len Avg Text Len Resolution MSVD [ 8 ] ACL 2011 Human Open 2K 9.7s 8.7 words - UCF101 [ 37 ] ICCV 2013 Human Action 13K 7.2s 4.3 words 240p LSMDC [ 31 ] CVPR 2015 Human Movie 118K 4.8s 7.0 words 1080p ActivityNet [ 6 ] CVPR 2015 Human Action 100K 36.0s 13.5 words - MSR-VTT [ 45 ] CVPR 2016 Human Open 10K 15.0s 9.3 words 240p DiDeMo [ 1 ] ICCV 2017 Human Flickr 27K 6.9s 8.0 words - YouCook2 [ 51 ] AAAI 2018 Human Cooking 14K 19.6s 8.8 words - VATEX [ 43 ] ICCV 2019 Human Open 41K 10s 15.2 words - HowTo100M [ 28 ] ICCV 2019 ASR Open 136M 3.6s 4.0 words 240p YT-Temporal-180M [ 48 ] NIPS 2021 ASR Open 180M - - - HD-VILA-100M [ 46 ] CVPR 2022 ASR Open 103M 13.4s 32.5 words 720p Panda-70M [ 10 ] CVPR 2024 Auto Open 70.8M 8.5s 13.2 words 720p CFC-Vids-1M Auto Open 1M 10.6s 89.3 words 720p 1 Introduction Text-to-video generation, which aims to synthesize video sequences from textual descriptions, has emerged as a significant research direction in AIGC. Recent years have witnessed remarkable progress in generative models, particularly in the realm of diffusion-based approaches [ 5 ] . The success of these models in image synthesis has opened new possibilities for video generation, with potential applications spanning creative content creation, educational resources, and visual storytelling [ 16 ] [ 27 ] [ 49 ] . The synthesis of videos that faithfully align with textual descriptions requires both comprehensive training data and sophisticated learning approaches to handle the complexities of spatial-temporal modeling. Recent text-to-video generation models [ 27 , 16 , 3 , 26 , 44 , 11 ] have demonstrated promising results by leveraging large-scale video datasets [ 10 , 46 , 28 ] . Despite these advances, existing datasets present limitations that affect model training effectiveness. These datasets face two primary challenges: 1) video quality issues, including temporal inconsistencies from imprecise scene detection and prevalence of static content lacking motion dynamics; 2) caption quality limitations, manifested in frame-by-frame descriptions rather than coherent narratives, insufficient descriptive details with limited word counts, and inadequate semantic alignment between text and video content. To address these challenges, we introduce CFC-Vids-1M , constructed through a systematic coarse-to-fine curation pipeline that ensures both visual and caption quality. Our proposed coarse-to-fine curation pipeline addresses dataset quality through two systematic stages. At the coarse level, we evaluate video quality across multiple dimensions including aesthetic appeal, temporal consistency, OCR presence, motion dynamics, and category distribution. Each dimension is quantitatively assessed using specialized models. The fine-grained stage focuses on enhancing text-video alignment through a two-step process that leverages vision-language and large language models, ensuring both caption informativeness and semantic accuracy. To fully leverage the curated CFC-Vids-1M dataset for text-to-video synthesis, efficient architectures and training strategies are crucial. We develop a transformer-based model Raccoon with decoupled spatial-temporal attention mechanisms, enabling efficient processing of video sequences. Building upon this architecture, we propose a four-stage training strategy that progressively enhances model capabilities: beginning with semantic learning from pre-trained image models, followed by temporal modeling at low resolution, then scaling to high-resolution video generation, and finally refining visual quality through targeted fine-tuning. This systematic approach effectively addresses the computational challenges of video generation while maintaining generation quality. This paper presents a comprehensive approach to text-to-video generation through both data curation and model design. The proposed coarse-to-fine curation pipeline ensures high-quality training data across multiple dimensions, while the progressive training strategy enables efficient learning of both semantic and temporal aspects. Extensive experiments demonstrate the effectiveness of our approach in generating high-quality and temporally coherent videos. Figure 1 : Overview of data curation. Firstly, we employ a scene splitting algorithm to divide long videos with multiple scenes into single scene shots. We filter and sample videos based on five aspects: video quality, OCR, temporal consistency, category, and motion. Finally, we use a Large Language Model (LLM) to curate video-text pairs for error captions. 2 Related Work Vision-Language Dataset . To advance video understanding and generation, numerous video-text datasets have been developed, differing in aspects such as video length, resolution, domain, and scale. These datasets are compared in Table 1 . Early datasets like MSVD [ 8 ] and MSRVTT [ 45 ] rely on human annotation for video captioning, ensuring high-quality text-video alignment but limiting dataset scale. To address scalability, datasets such as YT-HowTo100M [ 28 ] and HD-VILA-100M [ 46 ] adopt automatic annotation through ASR-generated subtitles, though this approach often leads to caption inaccuracies. Recent work Panda-70M [ 10 ] utilizes a cross-modality teacher model to generate 70 million text-video pairs, demonstrating the potential of model-based caption generation. In this work, we present a two-stage coarse-to-fine curation pipeline that combines multi-dimensional quality filtering and model-based caption enhancement to ensure both visual quality and text-video alignment. Video Generation Methods . Recent advances in video synthesis have explored various architectural paradigms. Autoregressive methods like VQGAN extensions [ 20 , 47 ] predict sequential video tokens but face computational constraints due to their sequential nature. GAN-based approaches [ 22 , 23 ] achieve efficient generation but often struggle with stability and temporal consistency. More recently, transformer-based diffusion models have emerged as a promising direction for video generation [ 27 , 16 , 3 , 26 , 44 , 11 , 19 , 49 ] . These approaches demonstrate superior generation quality and temporal coherence, though at the cost of substantial computational requirements. For instance, W.A.L.T [ 16 ] achieves excellent performance through window-based attention and unified latent space for images and videos. In this work, we present a transformer-based diffusion model trained with a multi-stage strategy for high-quality video generation. Efficient Training Strategies . Recent works in