OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng, Yang Liu · 2026
Read OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding by Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng, Yang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25276.txt.
Introduction
Zheng and four colleagues introduce OmniVTG, a large data set for open-world video temporal grounding plus a Self-Correction Chain-of-Thought training paradigm. Open-world grounding struggles because data sets are small and narrow, leaving gaps between common and rare concepts. OmniVTG is built by finding vocabulary gaps and collecting videos likely to contain target concepts. The paradigm trains models to predict, then reflect on and refine predictions. Experiments show strong open-world grounding on OmniVTG and state-of-the-art zero-shot results on four benchmarks.
Catalog record
| CREATOR | Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng, Yang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35487 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng, Yang Liu Abstract Video Temporal Grounding (VTG), the task of localizing video segments from text queries, struggles in open-world settings due to limited dataset scale and semantic diversity, causing performance gaps between common and rare concepts. To overcome these limitations, we introduce OmniVTG, a new large-scale dataset for open-world VTG, coupled with a Self-Correction Chain-of-Thought (CoT) training paradigm designed to enhance the grounding capabilities of Multimodal Large Language Models (MLLMs). Our OmniVTG is constructed via a novel Semantic Coverage Iterative Expansion pipeline, which first identifies gaps in the vocabulary of existing datasets and collects videos that are highly likely to contain these target concepts. For high-quality annotation, we leverage the insight that modern MLLMs excel at dense captioning more than direct grounding and design a caption-centric data engine to prompt MLLMs to generate dense, timestamped descriptions. Beyond the dataset, we observe that simple supervised finetuning (SFT) is insufficient, as a performance gap between rare and common concepts still persists. We find that MLLMs’ video understanding ability significantly surpasses their direct grounding ability. Based on this, we propose a Self-Correction Chain-of-Thought (CoT) training paradigm. We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions. This capability is instilled via a three-stage pipeline of SFT, CoT finetuning, and reinforcement learning. Extensive experiments show our approach not only excels at open-world grounding in our OmniVTG dataset but also achieves state-of-the-art zero-shot performance on four existing VTG benchmarks. Code is available at https://github.com/oceanflowlab/OmniVTG . 1 Introduction Figure 1 : (a) Open-world video temporal grounding performance on our OmniVTG dataset. (b) The MLLM’s (Qwen2.5-VL-7B 2 ) ability in video understanding and temporal grounding tasks. (c) The accuracy of timestamps in grounding and dense caption tasks performed by Gemini-2.5-Pro 5 . Figure 2 : The visualizations and comparisons of our OmniVTG dataset. Video Temporal Grounding (VTG) is the task of identifying the precise start and end timestamps of events described by a natural language query within an untrimmed video. With the development of deep learning 2 ; 17 ; 30 ; 47 ; 3 and big data 12 ; 7 ; 36 ; 20 , this task is being increasing important for real-world applications, such as video search and automated content analysis. In open-world scenarios, the challenge of VTG becomes significantly more complex. Real-world videos often contain a broad spectrum of events and concepts, ranging from common and everyday occurrences to rare, abstract, or even highly domain-specific phenomena. To effectively handle these diverse events, models must be able to understand a wide range of semantic concepts, including not only frequent and well-defined categories but also rare and ambiguous ones. As shown in Fig. 1 (a), we find that the performance of existing Video Temporal Grounding MLLMs 17 ; 30 ; 2 is significantly lower when handling these open-world scenarios, especially when they contain rare concepts. This reflects the limitations of existing datasets and models. In terms of datasets, existing public VTG datasets fall short of the open-world requirements, suffering from two primary limitations. First, limited semantic coverage. Mainstream datasets are often restricted to specific domains (e.g., indoor activities in Charades-STA 7 ) or possess a narrow vocabulary as shown in Fig. 2 (c). Second, limited scale and quality. Datasets relying on manual annotation 12 ; 7 ; 13 are expensive and laborious to create, making them difficult to scale. Automated pipelines 29 ; 23 often depend on sources like Automatic Speech Recognition (ASR), which cannot guarantee a precise match between the query (speech) and the visual content. To address these data limitations, we introduce OmniVTG, a large-scale, semantic-rich open-world dataset designed to expand semantic coverage. Our collection process employs a Semantic Coverage Iterative Expansion strategy, actively identifying underrepresented concepts and collecting relevant videos from the web. We begin by analyzing the vocabulary coverage of existing datasets to identify underrepresented rare concepts. We then perform targeted expansion by retrieving videos that are more likely to contain these rare concepts. To efficiently gather relevant videos, we leverage LLMs to translate target concepts into effective search keywords for video collection from the internet. To achieve automated annotation, we observed that the accuracy of timestamps generated during dense captioning is significantly higher than that obtained through direct grounding for modern MLLM 5 as shown in Fig. 1 (c) 1 1 1 We use Gemini-2.5-Pro to generate dense captions and manually check the timestamps to report accuracy. Then, we use the same model to ground this caption, and the grounding performance is much lower. . Based on this insight, we designed a caption-centric data engine. We prompt MLLM 5 to generate dense, timestamped captions that explicitly cover our target rare concepts, ensuring high-quality and fully automated temporal annotations. As shown in Fig. 2 (a) and (c), in this way, our dataset successfully covers rare vocabulary and enriches the query semantics. Finally, we manually inspect and refine a subset of these annotations to create a robust test set for evaluating open-world grounding performance. In terms of models, while Supervised Fine-Tuning (SFT) on OmniVTG improves overall performance, we still observe that the gap between rare and common concepts persists, as shown in Fig 1 (b). This suggests the model needs a more robust reasoning mechanism to handle unfamiliar, rare concepts. Furthermore, we find that the model’s capabilities in video understanding are significantly stronger than its direct grounding ability, and the performance gap between rare and common concepts is much smaller. For example, as shown in Fig. 1 (b), the model is better at judging whether a given video segment matches a text query and judging the state of the event described by the query at a specific timestamp (i.e., not started, ongoing, or ended) 2 2 2 We finetune Qwen2.5-VL on all tasks and test the performance. . Based on this observation, we propose a Self-Correction Chain-of-Thought (CoT) . We require the model to first make predictions and then reflect on them using its video understanding capabilities. For example, it should determine whether the predicted segment indeed matches the query. Additionally, it should assess the state of the predicted start and end time events, such as inferring that the start time needs to be moved backward if the event has not yet begun at the predicted start time. To achieve this, we construct CoT fine-tuning data using our OmniVTG dataset that embodies this self-reflection mechanism to train the model and then further enhance the model’s reasoning abilities through reinforcement learning. Our contributions are summarized as follows: (1) We introduce OmniVTG, a large-scale, semantic-rich dataset constructed via a novel Semantic Coverage Iterative Expansion strategy. (2) We propose a Self-Correction Chain-of-Thought mechanism by leveraging the model’s video understanding performance to reflect and refine its temporal grounding predictions. (3) Comprehensive experiments show our approach not only excels at our manually annotated OmniVTG test set but also achieves state-of-the-art zero-shot performance on four public VTG benchmarks. 2 Related Work 2.1 Datasets for Video Temporal Grounding Data collection methods. Prominent datasets 12 ; 7 ; 13 , rely on intensive manual annotation. While this process yields high-quality labels,