TimeLoc : A Unified End-to-End Framework for Precise Timestamp Localization in Long Videos

Chen-Lin Zhang, Lin Sui, Shuming Liu, Fangzhou Mu, Zhangcheng Wang, Bernard Ghanem · 2025

Read TimeLoc : A Unified End-to-End Framework for Precise Timestamp Localization in Long Videos by Chen-Lin Zhang, Lin Sui, Shuming Liu, Fangzhou Mu, Zhangcheng Wang, Bernard Ghanem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06526.txt.

Introduction

Chen-Lin Zhang and co-authors present a single system in 2025 that finds the exact moments of events in long, untrimmed videos. Pinpointing times is a core skill for video software, but past methods were built for one task each. The task splits into finding actions, matching a text query to a span, pulling a named moment, and spotting event edges.

Their system takes text queries as input and can give several actions as output. It uses a one stage design, which keeps it simple and fast. The video encoder is trained jointly across tasks, so skill from one task aids the others. A unified finder could serve search, sports analysis, and safety review with one tool. The team reports tests across the task family to show the breadth of the design.

Catalog record

CREATORChen-Lin Zhang, Lin Sui, Shuming Liu, Fangzhou Mu, Zhangcheng Wang, Bernard Ghanem
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06526

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TimeLoc : A Unified End-to-End Framework for Precise Timestamp Localization in Long Videos Chen-Lin Zhang, Lin Sui, Shuming Liu, Fangzhou Mu, Zhangcheng Wang, Bernard Ghanem Abstract Temporal localization in untrimmed videos, which aims to identify specific timestamps, is crucial for video understanding but remains challenging. This task encompasses several subtasks, including temporal action localization, temporal video grounding, moment retrieval, and generic event boundary detection. Existing methods in each subfield are typically designed for specific tasks and lack generalizability across domains. In this paper, we propose TimeLoc , a unified end-to-end framework for timestamp localization that can handle multiple tasks. First, our approach employs a simple yet effective one-stage localization model that supports text queries as input and multiple actions as output. Second, we jointly train the video encoder and localization model in an end-to-end manner. To efficiently process long videos, we introduce temporal chunking, enabling the handling of videos with over 30k frames. Third, we find that fine-tuning pre-trained text encoders with a multi-stage training strategy further enhances text-conditioned localization. TimeLoc achieves state-of-the-art results across multiple benchmarks: +1.3% and +1.9% mAP over previous best methods on THUMOS14 and EPIC-Kitchens-100, +1.1% on Kinetics-GEBD, +2.94% mAP on QVHighlights, and significant improvements in temporal video grounding (+11.5% on TACoS and +6.7% on Charades-STA under [email protected]). Our code and checkpoints will be released at https://github.com/sming256/TimeLoc . 2 2 footnotetext: Corresponding author: Chen-Lin Zhang < [email protected] > 1 Introduction In video understanding, many tasks involve localizing actions ( i.e. , temporal action localization [ 26 , 59 , 28 ] ), text descriptions ( i.e. , temporal sentence grounding [ 43 , 38 ] ), and event boundaries ( i.e. , generic event boundary detection [ 22 , 42 , 65 ] ) by identifying timestamps within an untrimmed video. These timestamp localization tasks have attracted growing interest for their many applications in intelligent personal assistants [ 16 ] , healthcare [ 18 ] , and human-robot interaction [ 12 , 67 ] , yet present a significant challenge as they require “finding needles in a haystack” through joint reasoning over spatial, temporal, and textual information. Although recent advances have led to methods that excel in individual tasks ( e.g. , [ 59 , 35 ] ), a unified modeling approach that generalizes across the spectrum of timestamp localization tasks is currently lacking, despite their shared problem structure. Moreover, existing methods often train only the localization head(s) on pre-extracted, frozen video features, as naively forwarding a full-sized video through a typically large video encoder results in impractical GPU memory usage during gradient updates. However, this prevents the video encoder from adapting to the downstream task, potentially harming model accuracy. Figure 1 : TimeLoc achieves state-of-the-art performance across various temporal localization tasks , including temporal action localization, temporal sentence grounding, moment retrieval, and generic event boundary detection. To overcome these challenges, we present TimeLoc , a unified framework for Time stamp Loc alization that supports efficient end-to-end training. TimeLoc follows a minimalist design that is both simple and effective. First , compared to multi-stage approaches like UnLoc [ 55 ] and UniVTG [ 20 ] , TimeLoc features a lightweight one-stage model design that can tackle a broader range of timestamp localization tasks, including generic event boundary detection, while achieving stronger performance. Second , given this universal model architecture, we develop a temporal chunking strategy to facilitate the joint training of video encoder and localization head(s) on long-form videos, for the first time enabling the processing of over 30K frames on a single GPU. Third , on selected tasks with textual inputs, we demonstrate that fine-tuning the pre-trained text encoder following a multi-stage training strategy yields significant performance improvements. Notwithstanding its simplicity, TimeLoc achieves state-of-the-art (SoTA) accuracy across numerous timestamp localization tasks, as summarized in Figure 1 . On temporal action localization , it surpasses the previous SoTA method AdaTAD [ 28 ] by 1.3% mAP on THUMOS14 [ 10 ] and 1.9% mAP on EPIC-Kitchens-100 [ 5 ] , using the same VideoMAE-L backbone. On generic event boundary detection , TimeLoc outperforms EfficientGEBD [ 65 ] by 1.1% on Kinetics-GEBD [ 42 ] . On temporal sentence grounding , TimeLoc demonstrates a significant gain of 8.89% [email protected] and 11.5% [email protected] on TACoS [ 38 ] , and an edge of 6.7% [email protected] and 6.7% [email protected] on Charades-STA [ 43 ] , when compared to SnAG [ 35 ] . On text-based moment retrieval , TimeLoc enables an average mAP improvement of 2.94% on QVHighlights [ 14 ] over CG-DETR [ 33 ] . Our contributions are summarized as follows: \bullet We propose a unified, end-to-end framework for timestamp localization tasks including temporal action localization, temporal grounding, moment retrieval, and generic event boundary detection. Our framework can be easily extended to include other localization tasks. \bullet We introduce temporal chunking to facilitate end-to-end training on long videos and find that fine-tuning the text encoder with a multi-stage training strategy further improves the accuracy of text-based timestamp prediction. \bullet Our framework achieves state-of-the-art performance across a myriad of timestamp localization benchmarks, significantly outperforming previous methods. 2 Related work 2.1 Temporal Video Understanding Temporal Action Localization aims to identify action instances in untrimmed videos and classify their categories. Methods for this task can be divided into three types: one-stage, two-stage, and DETR-based approaches. One-stage methods, such as ActionFormer [ 59 ] , TriDet [ 40 ] , and DyFADet [ 56 ] , integrate action classification and boundary regression directly using multi-scale feature pyramids. Two-stage methods incorporate a proposal feature extraction step, exemplified by VSGN [ 62 ] ’s boundary sampling technique. Recently, end-to-end approaches have emerged [ 27 , 28 ] . AFSD [ 19 ] processes videos at a small spatial resolution for efficient end-to-end learning, while E2E-TAD [ 29 ] extensively studies end-to-end training strategies. Generic Event Boundary Detection (GEBD) , introduced by [ 42 ] , focuses on detecting taxonomy-free event boundaries and segmenting videos into meaningful temporal chunks. Most existing works [ 46 , 65 ] formulate GEBD as a binary classification problem. DDM-Net [ 46 ] constructs a feature bank to store temporal features and captures motion patterns using dense difference maps. EfficientGEBD [ 65 ] establishes a strong baseline and utilizes video-domain backbones. A DETR-based detection model was proposed in [ 45 ] to address GEBD; however, it was trained offline. Existing GEBD methods are task-specific and lack generalizability across domains. Temporal Video Grounding localizes moments in untrimmed videos based on text descriptions. Similar to temporal action localization, grounding methods are divided into two-stage and single-stage approaches, focusing on proposal generation and cross-modal fusion. Two-stage methods generate and score temporal segments, with some models conditioning on sentence queries to reduce dense sampling [ 54 , 2 , 53 , 23 , 52 ] . In contrast, single-stage methods localize moments efficiently without explicit proposals, using global features or learnable queries [ 15 , 51 , 24 , 64 , 60 ] . Despite their efficiency, single-stage methods typically underperform compared to two-stage models on benchmarks. Cross-modal fusion integrates video and text information for improved grounding, evolving from simple late fusion to sophisticated early fusion strategies incorporating LSTMs, GCNs, and memory banks.

Read the complete text