USV: Towards Understanding the User-generated Short-form Videos
Haoyue Cheng, Liwei Jin, Wayne Wu, Chen Qian, Limin Wang · 2026
Read USV: Towards Understanding the User-generated Short-form Videos by Haoyue Cheng, Liwei Jin, Wayne Wu, Chen Qian, Limin Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20838.txt.
Introduction
Haoyue Cheng and five colleagues present USV, a dataset of user-generated short-form videos for high-level semantic video understanding. Such videos are rarely studied, though large-scale video datasets have advanced the field. USV holds around 224,000 videos gathered from user platforms by label queries, without added manual verification or trimming. Most prior work focuses on instance-level recognition, which misses high-level semantic representation, so the authors add topic recognition and video-text retrieval tasks with two unified baseline methods.
Catalog record
| CREATOR | Haoyue Cheng, Liwei Jin, Wayne Wu, Chen Qian, Limin Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53145 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
USV: Towards Understanding the User-generated Short-form Videos Haoyue Cheng, Liwei Jin, Wayne Wu, Chen Qian, Limin Wang Abstract Several large-scale video datasets have been published these years and have advanced the area of video understanding. However, the newly emerged user-generated short-form videos have rarely been studied. This paper presents USV, the User-generated Short-form Video dataset for high-level semantic video understanding. The dataset contains around 224K videos collected from UGC platforms by label queries without extra manual verification and trimming. Although video understanding has achieved plausible improvement these years, most works focus on instance-level recognition, which is not sufficient for learning the representation of the high-level semantic information of videos. Therefore, we further establish two tasks: topic recognition and video-text retrieval on USV. We propose two unified and effective baseline methods Multi-Modality Fusion Network (MMF-Net) and Video-Text Contrastive Learning (VTCL), to tackle the topic recognition task and video-text retrieval respectively, and carry out comprehensive benchmarks to facilitate future research. Our project page is https://usvdataset.github.io . 1 1 1 † {\dagger} : Equal contribution, ✉: Corresponding author. 1 Introduction Recently, user-generated short-form videos from platforms such as TikTok 66 , Kwai 33 , and Reels 55 have drawn much attention 65 . Understanding user-generated short-form videos is of great importance for practical usages such as video recommendation 8 , venue analysis 75 and automated video summarization 24 . Take the video recommendation as an example. Zhu et al . 80 recommend videos by recalling videos that most fit the interested topic distribution of users. Deng et al . 9 take a real-time hot topic detection as the first step of recommendation. However, neither a dataset nor a benchmark exists in previous literature for understanding user-generated short-form videos from a multi-modality and high-level semantic perspective. In these years, many video datasets have been proposed and pushed the boundary of video understanding. For example, some 28 ; 32 ; 39 ; 47 ; 51 ; 59 ; 62 are built for action recognition, others 19 ; 22 ; 25 are built for action localization. They mainly focus on recognizing instance-level actions and entities in long-term videos. None of the aforementioned works collect data purely from user-generated short-form videos, nor for leveraging multi-modality cues to facilitate understanding of high-level semantic information of videos. User-generated short-form videos have four main features that distinguish them from other video forms. 1) Topic Concentration : User-generated short-form videos are more topic concentrated 49 compared to professional-generated ones such as movies and TV series. Because short-form video platforms often have a duration constraint, short-form videos are forced to convey a single main topic in a few seconds. 2) Text Richness : User-generated short-form videos usually include much text information, such as titles, subtitles, dialogue, and comments. These texts are all user-generated and have rich semantic information related to the videos. 3) High Activity : User-generated video platforms are highly active, with millions of videos uploaded every day, together with all kinds of new topics of videos. This difference makes manually filtering the noise for a clean dataset trivial compared with scaling up with the noise, which makes the traditional time-consuming data process with manual annotating impractical. 4) Large Diversity : User-generated short-form videos have more unique genres ( e.g . , slides, lectures, podcasts) and narratives ( e.g . , selfies, portraits) to demonstrate their topics. With the large diversity, the multi-modality cues are rather important for understanding. In this study, we aim to move towards understanding the user-generated short-form videos based on the aforementioned features. First, we introduce a new dataset named User-generated Short-form Video (USV-1.0). Specifically, USV-1.0 (the first version of USV) contains around 224K videos of 212 topic categories. Considering the first feature of Topic Concentration , we propose a new task named topic recognition. Regarding the second feature of Text Richness , we define the second task as video-text retrieval. Detailed definition and motivation of these two tasks can be found in Sec. 4.1 . For the third feature of High Activity , we use no human labor for verification and trimming but directly assign the queried words as the topics and the user-generated titles as the text to retrieve. This web-supervised scheme is good at leveraging the large-scale user-generated video stream to facilitate video understanding in the real world. The fourth feature of Large Diversity identifies the main challenge of our tasks: how to integrate sufficient cues from various modalities to boost high-level semantic video understanding? For topic recognition, we propose a simple yet effective baseline method Multi-Modality Fusion Network (MMF-Net) as a baseline. Specifically, MMF-Net is a three-branch network that fuses models’ predictions for three modalities to form a consensus on the topic. For video-text retrieval, we adopt a video-text contrastive learning (VTCL) framework, which has proven effective in self-supervised learning. For both tasks, we build a comprehensive benchmark to facilitate future research. Benchmark experiments are conducted with our implementation based on a common protocol and evaluated under a unified setting without bells and whistles. To be brief, our contributions are three-fold: 1. New Data: we collect a new dataset: USV-1.0, the first large-scale dataset that aims to push the boundary of real-world short-form video understanding to our knowledge. 2. New Tasks: we define a new task called topic recognition, and we first try to perform video-text retrieval based on user-generated titles. Both tasks focus on understanding high-level semantic information. 3. New Methods: we propose MMF-Net and VTCL, the first trials to utilize both audio and subtitles to tackle the topic recognition task and the video-text retrieval respectively, and build a comprehensive benchmark to facilitate future research. 2 Related Work Video Recognition Datasets. Several video datasets have been published these years. From initial trials 32 ; 62 to large-scale benchmarks 28 ; 47 ; 48 . Then datasets for specific fields emerged, such as fine-grained gym datasets 23 ; 59 , human gestures 42 , surveillance footage 51 and RGB-D camera 58 . Also, datasets of a tremendous scale 1 ; 12 ; 26 with the help of automatic annotation systems and web data have been derived. Most of these listed restrict their label space to be instance-level visual entities, and most videos are generated by professionals. A similar idea of topical understanding is observed in YouTube8M 1 . However, they also restrict their topic entities to be visual and collected from YouTube with mostly long-term videos that are hard to infer a single topic or retrieve by a single title. This difficulty forces them to label a video with several topics and eventually degrades into visual-instance recognition. A few works have been done towards understanding short-form videos 40 ; 49 ; 50 ; 71 ; 78 . However, most of their works are based on Vine, a video platform that has been shut down for years. Besides, the task designed on those datasets is scene/venue classification, which is still an instance-level recognition task. Ours is not restricted to visual entities and is labeled video-wisely and purely from user-generated content. Video-Text Retrieval. Learning videos with language has been a trend towards understanding videos, and video-text retrieval is one of the fundamental tasks. Common datasets including 57 ; 73 ; 79 are relatively small, and