VISTA : Enhancing Long-Duration and High-Resolution Video Understanding by VI deo S patio T emporal A ugmentation
Weiming Ren, Huan Yang, Jie Min, Cong Wei, Wenhu Chen · 2024
Read VISTA : Enhancing Long-Duration and High-Resolution Video Understanding by VI deo S patio T emporal A ugmentation by Weiming Ren, Huan Yang, Jie Min, Cong Wei, Wenhu Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00927.txt.
Introduction
Weiming Ren and co-authors present VISTA, a way to build training data for long and sharp video, in a 2024 paper. Large multimodal models struggle with long or high resolution video, mostly for lack of good training sets. Their frame joins existing clips in space and time to fake longer, sharper videos, then writes question and answer pairs about them.
Seven join methods produced a dataset of four hundred thousand pairs. Fine tuning several video models on it lifted scores by a few points on four hard long video tests. The team also built the first full benchmark for high resolution video understanding, where their tuned models gained more. Data crafting, not new model design, drove the gains.
Catalog record
| CREATOR | Weiming Ren, Huan Yang, Jie Min, Cong Wei, Wenhu Chen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00927
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VISTA : Enhancing Long-Duration and High-Resolution Video Understanding by VI deo S patio T emporal A ugmentation Weiming Ren, Huan Yang, Jie Min, Cong Wei, Wenhu Chen Abstract Current large multimodal models (LMMs) face significant challenges in processing and comprehending long-duration or high-resolution videos, which is mainly due to the lack of high-quality datasets. To address this issue from a data-centric perspective, we propose VISTA , a simple yet effective VI deo S patio T emporal A ugmentation framework that synthesizes long-duration and high-resolution video instruction-following pairs from existing video-caption datasets. VISTA spatially and temporally combines videos to create new synthetic videos with extended durations and enhanced resolutions, and subsequently produces question-answer pairs pertaining to these newly synthesized videos. Based on this paradigm, we develop seven video augmentation methods and curate VISTA-400K, a video instruction-following dataset aimed at enhancing long-duration and high-resolution video understanding. Finetuning various video LMMs on our data resulted in an average improvement of 3.3% across four challenging benchmarks for long-video understanding. Furthermore, we introduce the first comprehensive high-resolution video understanding benchmark HRVideoBench, on which our finetuned models achieve a 6.5% performance gain. These results highlight the effectiveness of our framework. 1 Introduction Recent advancements in large language models (LLMs) and large multimodal models (LMMs) have brought transformative changes to video understanding tasks. Traditionally, video understanding relied on training task-specific models using domain-specific datasets (e.g. action recognition on Kinetics [ 3 ] , video retrieval on MSR-VTT [ 54 ] , and video captioning on YouCook2 [ 67 ] ). In contrast, it is now possible to process video inputs and address diverse tasks using a single video LMM [ 26 , 6 , 32 ] through instruction following. However, most current (open-sourced) video LMMs are optimized for understanding and reasoning over short, low-resolution videos. Processing long-sequence video input, such as long or high-resolution videos, remains a significant challenge for video LMMs. Figure 1 : VISTA is a simple but effective framework that generates high-quality video instruction data from existing video-caption pairs. Our VISTA-400K dataset enhances model performances on various long and high-resolution video benchmarks. Most efforts to enhance long-sequence video understanding have centered around designing better model architectures. Conversely, the creation of higher-quality (long/high-resolution) video instruction-following datasets remains highly under-explored. The primary challenges stem from the scarcity of publicly available (license-friendly) high-quality video-instruction data. Consequently, open-source video instruction-following datasets often face limitations such as low resolution or short duration. For example, VideoChat2 [ 23 ] collects a video instruction-following dataset by combining videos from multiple domain-specific datasets and rewriting the instructions using ChatGPT [ 34 ] . However, the video sources used in this dataset primarily contain short videos. ShareGPT4Video [ 4 ] curates a video-instruction dataset containing 40K videos with detailed video captions. Nevertheless, the source videos are collected at a sparse sampling rate of 0.15 fps, and the video contents often lack motion and can appear nearly static. FineVideo [ 8 ] is a long video dataset featuring diverse video contents and high-quality metadata such as detailed captions and narrative progressions. While it offers satisfactory video durations, the dataset is limited by its resolution, as it predominantly comprises 360p videos. Though some concurrent studies have improved LMMs’ long-sequence video understanding capabilities, they often release only the model weights, hiding their video training data. For instance, Kangaroo [ 28 ] ’s training data contains 700K long video data, but specifics about the curation of the videos and the construction of the instructions are not provided. Similarly, Qwen2-VL [ 46 ] , MiniCPM-V-2.6 [ 57 ] and Aria [ 19 ] all claim that their instruction-following dataset contains long video training data, yet without providing the sources and statistics of the video data. This lack of transparency hinders a clear understanding of what types of video instruction data truly benefit long-sequence video understanding tasks and impedes further advancements in improving existing models. In this study, we propose VISTA, a simple yet effective video augmentation pipeline to synthesize long and high-resolution video instruction data from existing video datasets . VISTA leverages insights from image and video classification data augmentation techniques such as CutMix [ 59 ] , MixUp [ 61 ] and VideoMix [ 60 ] , which demonstrate that training on synthetic data created by overlaying or mixing multiple images or videos results in more robust classifiers. Similarly, our method spatially and temporally combines videos to create (artificial) augmented video samples with longer durations and higher resolutions, followed by synthesizing instruction data based on these new videos. Our data synthesis pipeline utilizes existing public video-caption datasets, making it fully open-sourced and scalable. This allows us to construct VISTA-400K, a high-quality video instruction-following dataset aimed at improving the long and high-resolution video understanding capabilities of video LMMs. By finetuning various video LMMs on our dataset, we observe an average of 3.3% improvement across multiple long video understanding benchmarks. Additionally, we compile a new benchmark HRVideoBench that focuses on high-resolution videos. Results on HRVideoBench (+6.5% after finetuning) demonstrate that our method produces models well-suited for high-resolution video understanding. Our contributions are summarized below: 1. We present VISTA-400K, a high-quality synthetic video instruction-following dataset. VISTA-400K includes challenging QA pairs designed to enhance video LMMs’ ability to understand long and high-res video inputs. 2. We collect HRVideoBench, a comprehensive benchmark dedicated to evaluating video LMMs’ capability in understanding fine object details and subtle, localized actions within high-resolution videos. 3. Our VISTA-finetuned models achieve an average gain of 3.3% on four challenging long-video benchmarks and 6.5% on HRVideoBench compared to the vanilla models. Our ablation study indicates that disabling our proposed video augmentations significantly reduces performance. Figure 2 : Our proposed video augmentation and instruction-following data synthesis schemes for VISTA-400K. Given input videos, We perform spatiotemporal video combinations to produce augmented video samples with longer duration and higher resolution. 2 VISTA-400K Dataset Our goal is to create high-quality video instruction-following data from existing video-caption datasets. Specifically, we aim to (1) generate extended or higher-resolution videos by spatially or temporally combining existing videos, and (2) produce high-quality question-answer pairs by leveraging existing video captions. Formally, given a set of candidate videos \mathbf{V}=\{V_{1},V_{2},...,V_{N}\} with captions \mathbf{C}=\{C_{1},C_{2},...,C_{N}\} , we seek to create an augmented video V^{*} and synthesize a QA pair (q,a) : \displaystyle V^{*}=\Phi(\mathbf{V}),\quad\quad(q,a)=\Theta(\mathbf{C}). (1) For the video augmentation operator \Phi , we draw inspiration from data augmentation techniques used in image and video classification, such as CutMix [ 59 ] and VideoMix [ 60 ] , to perform video mixing and combination. We use a highly capable language model Gemini-1.5-Pro [ 44 ] as \Theta to generate synthetic QA pairs. As shown in Figure 2 , our dataset contains a total of seven subsets, each featuring different video augmentation methods. In this section, we detail the dataset construction process for each subset. Table 1: Statistics of our synthetic video instruction-following dataset. “(N)” and “(H)” corresponds to the “needle” (short or low-res videos) and the “haystack” (long or high-res videos) in NIAH subsets. Subset Instruction Type Video Source #Videos Avg. Duration Avg. Resolution Long Video Captioning Video Captioning Panda-70M [ 5 ] 58,617 33.2s 1277 \times 720 Event Relationship QA Freeform QA/MCQ Panda-70M [ 5 ] 56,854 33.4s 1278 \times 720 Temporal NIAH