The Role of Video Generation in Enhancing Data-Limited Action Understanding
Wei Li, Dezhao Luo, Dongbao Yang, Zhenhang Li, Weiping Wang, Yu Zhou · 2025
Read The Role of Video Generation in Enhancing Data-Limited Action Understanding by Wei Li, Dezhao Luo, Dongbao Yang, Zhenhang Li, Weiping Wang, Yu Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19495.txt.
Introduction
Wei Li and co-authors use generated videos to train action models when real data are scarce. Their 2025 paper tackles a common block in video understanding: too few labeled clips. Their method uses a text-to-video model to make labeled training data at scale, with no human effort. Two extra strategies tune how that fake data are used.
First, an information boost enriches the made clips, since real clips tend to carry richer detail. It upgrades both the settings and the characters in a scene. Second, a smoothing step softens labels on low-grade clips, judged by uncertainty, so they do less harm. Tests across four data sets and five tasks gave top results for zero-shot action recognition. The authors frame generation as a bridge over data gaps.
Catalog record
| CREATOR | Wei Li, Dezhao Luo, Dongbao Yang, Zhenhang Li, Weiping Wang, Yu Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Role of Video Generation in Enhancing Data-Limited Action Understanding Wei Li, Dezhao Luo, Dongbao Yang, Zhenhang Li, Weiping Wang, Yu Zhou Abstract Video action understanding tasks in real-world scenarios always suffer data limitations. In this paper, we address the data-limited action understanding problem by bridging data scarcity. We propose a novel method that employs a text-to-video diffusion transformer to generate annotated data for model training. This paradigm enables the generation of realistic annotated data on an infinite scale without human intervention. We proposed the information enhancement strategy and the uncertainty-based label smoothing tailored to generate sample training. Through quantitative and qualitative analysis, we observed that real samples generally contain a richer level of information than generated samples. Based on this observation, the information enhancement strategy is proposed to enhance the informative content of the generated samples from two aspects: the environments and the characters. Furthermore, we observed that some low-quality generated samples might negatively affect model training. To address this, we devised the uncertainty-based label smoothing strategy to increase the smoothing of these samples, thus reducing their impact. We demonstrate the effectiveness of the proposed method on four datasets across five tasks and achieve state-of-the-art performance for zero-shot action recognition. 1 Introduction † † footnotetext: 🖂Corresponding author. Figure 1: The samples (a) and t-SNE visualizations (b) of the synthetic dataset and the real dataset. From left to right, they are: the HMDB-51 dataset, the synthetic HMDB-51 dataset with the basic strategy and the synthetic HMDB-51 dataset with our proposed information enhancement strategy. (c) Unsatisfactory synthetic videos. Over the past decade, deep learning LeCun et al. (2015) ; Miech et al. (2019) has brought remarkable progress benefiting from large-scale annotated data Carreira et al. (2018) ; Sultani et al. (2018) , especially in the computer vision community such as CLIP with 400M image-text pairs Radford et al. (2021) . However, obtaining high-quality datasets is often complex, time-consuming, and costly Carreira et al. (2018) ; Grauman et al. (2022) , requiring a large amount of manual annotation. In particular in the field of video understanding, human annotations can be inaccurate (50% samples are not aligned in the HowTo100M dataset Miech et al. (2019) ). A lack of a well-annotated dataset with sufficient samples will limit the learning of models and result in poor generalization ability, facing challenges in real-world applications such as abnormal action detection Sultani et al. (2018) ; Lv et al. (2023) and long-tail action recognition Perrett et al. (2023) ; Grauman et al. (2022) . In order to solve the data-limited problem in video understanding, previous methods often focus on designing data augmentation strategies Yun et al. (2020) ; Kim et al. (2020) ; Li et al. (2022b) and transfer knowledge Kim et al. (2025) ; Li et al. (2024) ; Rasheed et al. (2023) ; Luo et al. (2023) from other modalities, such as images, where data is easier to collect. However, we argue that these solutions are suboptimal as a) data augmentation does not create novel semantics for general knowledge learning Trabucco et al. (2024) , and b) knowledge learned from other modalities does not provide necessary information for video understanding, specifically, images do not include temporal relations between video frames. With the development of diffusion models Yang et al. (2024) ; Liu et al. (2024) ; Esser et al. (2023) , existing methods propose to synthesis datasets to improve model learning He et al. (2023) ; Tian et al. (2024) ; Luo et al. (2024) ; Feng et al. (2023) . There is a problem in the utilization of generated samples for training video understanding models, manifested by the deficient information within the samples, which leads to suboptimal training efficiency. As illustrated in Figure 1 (a) and (b), our finding is that due to the complex and rich information of real videos, real samples have higher intra-class distances and lower inter-class distances than synthetic samples; while the videos generated by the diffusion transformer tend to include information with simple and similar content, which limits the learnable information contained in the synthetic samples . This also makes the classification difficulty of synthetic samples lower than that of real samples. To solve the problem of deficient information within generated samples, we propose an information enhancement strategy to enhance the effective information of synthetic samples, which refers to context information conducive to action understanding. This strategy injects various character and environment information that conforms to specific human actions into the generated video and alleviates the domain gap between synthetic and real samples. Due to the limited capabilities of the text2video model, some synthetic videos perform unsatisfactorily and contain less effective semantic information, as shown in Figure 1 (c). To alleviate the impact of low-quality synthetic samples on model training, we propose uncertainty-based label smoothing. We calculate the uncertainty of the generated samples with CLIP to measure their quality and adjust the smoothness of label smoothing with uncertainty, which can prevent the model from overfitting low-quality samples and alleviating its impact. We conducted extensive experiments on four datasets (Kinetics-600, UCF-101, HMDB-51, UCF-Crime) across five tasks (few-shot, zero-shot, base-to-novel, long-tail, abnormal action detection) and demonstrated the effectiveness of our method in data-limited action understanding. We achieved state-of-the-art performance for zero-shot action recognition tasks on Kinetics-600, UCF-101, and HMDB-51. In summary, our contributions are as follows: (1) To the best of our knowledge, we are the first to investigate the effectiveness of data generated by video diffusion transformers for enhanced data-limited video understanding that includes action recognition and action detection tasks. (2) To solve the problem of deficient information within generated samples, we propose two strategies: the information enhancement strategy and the uncertainty-based label smoothing. These two strategies significantly improve the training efficiency of the generated samples. (3) We conduct extensive experiments on four datasets across five data-limited action understanding tasks to demonstrate the effectiveness of our proposed method, and our method achieves SOTA performance for zero-shot action recognition. 2 Related Work 2.1 Training with Synthetic Samples The methods of training with synthetic samples can be broadly categorized into two main streams: generative-based and graphics engine-based methods. The graphics engine-based methods use real objects’ 3D models to arrange, reconstruct, or move them according to certain rules and then render them to generate training samples. It has found usage in a myriad of fields, including object detection Gaidon et al. (2016) ; Cabon et al. (2020) , optical flow estimation Dosovitskiy et al. (2015) ; Kim et al. (2022) , auto driving Dosovitskiy et al. (2017) , etc. With the explosion development of AIGC, image generation models have become of sufficient quality to generate training samples. Earlier works on image understanding have concentrated on the generation of samples or data augmentation with GANs Baranchuk et al. (2021) ; Li et al. (2022a) . But with the emergence of the diffusion model, many methods focus on generating samples using diffusion models given its realistic, high-quality and controllable generative ability. He et al. (2023) found that diffusion model synthetic samples can significantly improve the classification results of zero-shot, few-shot and transfer learning, and designed strategies to filter the noise of synthetic samples and enhance their diversity. Feng et al. (2023) proposes a novel test-time