DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos
Zijia Lu † \dagger, A S M Iftekhar † \dagger, Gaurav Mittal, Tianjian Meng, Xiawei Wang, Cheng Zhao, Rohith Kukkala, Ehsan Elhamifar, Mei Chen, Microsoft, Northeastern University [email protected], {asmiftekhar, gamit, tianjianmeng, xiaweiwang}@microsoft.com{chengzhao, rokukkal, meic}@microsof · 2025
Read DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos by Zijia Lu † \dagger, A S M Iftekhar † \dagger, Gaurav Mittal, Tianjian Meng, Xiawei Wang, Cheng Zhao, Rohith Kukkala, Ehsan Elhamifar, Mei Chen, Microsoft, Northeastern University [email protected], {asmiftekhar, gamit, tianjianmeng, xiaweiwang}@microsoft.com{chengzhao, rokukkal, meic}@microsof (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16376.txt.
Introduction
Zijia Lu, Gaurav Mittal, and their colleagues find moments in long videos from text queries without paying full price for every second, in a paper published in two thousand twenty-five. The task, temporal grounding, asks a system to point to the exact span a description refers to, and long videos make the naive approach ruinously expensive.
The naive approach cuts the video into clips and runs a heavyweight encoder on each one. DeCafNet instead delegates the first pass to a light sidekick encoder. The sidekick scans all clips cheaply and draws a saliency map of likely matches. Only the most promising clips go to the full expert encoder for careful processing. Features from the two passes are then combined for the final answer. The delegate-and-conquer split aims to keep accuracy while slashing compute. It targets search and retrieval over the long videos that dominate real collections.
Catalog record
| CREATOR | Zijia Lu † \dagger, A S M Iftekhar † \dagger, Gaurav Mittal, Tianjian Meng, Xiawei Wang, Cheng Zhao, Rohith Kukkala, Ehsan Elhamifar, Mei Chen, Microsoft, Northeastern University [email protected], {asmiftekhar, gamit, tianjianmeng, xiaweiwang}@microsoft.com{chengzhao, rokukkal, meic}@microsof |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos Zijia Lu † \dagger, A S M Iftekhar † \dagger, Gaurav Mittal, Tianjian Meng, Xiawei Wang, Cheng Zhao, Rohith Kukkala, Ehsan Elhamifar, Mei Chen, Microsoft, Northeastern University [email protected], {asmiftekhar, gamit, tianjianmeng, xiaweiwang}@microsoft.com{chengzhao, rokukkal, meic}@microsoft.com, [email protected] Abstract Long Video Temporal Grounding (LVTG) aims at identifying specific moments within lengthy videos based on user-provided text queries for effective content retrieval. The approach taken by existing methods of dividing video into clips and processing each clip via a full-scale expert encoder is challenging to scale due to prohibitive computational costs of processing a large number of clips in long videos. To address this issue, we introduce DeCafNet, an approach employing “delegate-and-conquer” strategy to achieve computation efficiency without sacrificing grounding performance. DeCafNet introduces a sidekick encoder that performs dense feature extraction over all video clips in a resource-efficient manner, while generating a saliency map to identify the most relevant clips for full processing by the expert encoder. To effectively leverage features from sidekick and expert encoders that exist at different temporal resolutions, we introduce DeCaf-Grounder, which unifies and refines them via query-aware temporal aggregation and multi-scale temporal refinement for accurate grounding. Experiments on two LTVG benchmark datasets demonstrate that DeCafNet reduces computation by up to 47% while still outperforming existing methods, establishing a new state-of-the-art for LTVG in terms of both efficiency and performance. Our code is available at https://github.com/ZijiaLewisLu/CVPR2025-DeCafNet . $\dagger$ $\dagger$ footnotetext: Equal contribution Figure 1 : Model inference time and grounding performance on Ego4d-NLQ [ 13 ] dataset using one A100 gpu. The circle sizes indicate the TFLOPs for methods. Numbers beside the green circles indicate the amount of salient clips processed by expert encoder. 1 Introduction Long Video Temporal Grounding (LVTG) is the task of identifying specific moments or events within long videos (spanning from several minutes to a few hours [ 15 ] ) based on a user-provided text query. LVTG allows effective retrieval of relevant content from lengthy videos with a range of applications, such as video summarization [ 12 , 34 , 22 ] , content recommendation [ 18 , 32 ] , and surveillance [ 55 , 19 ] , where rapid detection of pertinent segments is critical. State-of-the-art (SOTA) LVTG methods [ 15 , 17 , 33 , 59 ] build on techniques originally developed for temporal video grounding in short videos, following a common two-step paradigm. First, they divide a long video into fixed-duration clips, processing each clip through an expert encoder–a large, pre-trained model trained on extensive, multi-domain video data–to extract video clip features. Second, a grounding model employs temporal and cross-modal reasoning to perform the grounding task. Although effective for short videos, such an approach struggles to scale with longer video lengths due to the high computational cost of processing each clip through the expert encoder. As video duration increases, the number of clips increases, leading to a significant surge in computational demands. Long temporal sequences, whether visual [ 5 ] , textual [ 3 ] , or multimodal [ 43 ] , are computationally heterogeneous, meaning that not all locations in the sequence are equally complex or contribute equally to the final prediction. Consequently, we find that, for a long video, the temporal moment associated with the input query mostly constitutes only a small portion of the whole video. Many clips in a long video are not relevant to the query in LVTG. Therefore, we can process them in a significantly more resource-efficient manner compared to the expert encoder, reducing overall computation required for LVTG and improving scalability while maintaining grounding performance. Leveraging the mentioned observation, we introduce DeCafNet to overcome the limitations of existing methods. DeCafNet employs a delegate-and-conquer strategy by delegating a significant portion of the computation to an efficient sidekick encoder to conquer the computational bottleneck. The sidekick encoder serves two key functions. First, it computes features for each video clip in a resource-efficient manner to reduce the overall computational cost. Second, it generates a saliency map over video clips by comparing the extracted features with the text query. This allows DeCafNet to identify the top-c% of query-relevant clips that require full-scale processing by the expert encoder. This delegate-and-conquer approach ensures that only the most salient clips undergo intensive computation to achieve both high performance and efficiency. With dense video features extracted by the sidekick encoder for every clip and the top-c% salient features from the expert encoder, directly using a standard LVTG grounding module like [ 31 , 15 ] gives suboptimal performance due to the diverse temporal resolutions of these feature sets. To optimize performance, we introduce DeCaf-Grounder, a novel grounding module designed specifically for our delegate-and-conquer encoder structure. DeCaf-Grounder unifies the features of sidekick and expert encoders via query-aware temporal aggregation and refines them over multiple temporal scales using multi-scale temporal refinement. We evaluate DeCafNet on two LVTG benchmark datasets and successfully achieve stronger grounding performance compared to existing works with 47\% less computation on average. Figure 1 visualizes DeCafNet’s computational efficiency and accuracy in Ego4D-NLQ [ 13 ] dataset. We outperform all existing methods when only processing top-50% of salient clips using the expert encoder, with negligible additional cost from the sidekick encoder. Overall, DeCafNet has 47% less TFLOPs and 51% less inference time than the existing works. Our contributions are, • We introduce DeCafNet, a novel approach that enhances computational efficiency while improving grounding performance on LVTG. • DeCafNet introduces a novel delegate-and-conquer approach, employing a combination of a sidekick encoder and an expert encoder to compute a set of dense and top-c% salient features. Our DeCaf-Grounder then aggregates and refines these features across multiple temporal scales to perform LTVG optimally. • DeCafNet significantly outperforms existing methods in terms of computational efficiency on two benchmark datasets. Even at much-reduced computation, DeCafNet achieves SOTA performance on temporal grounding in long videos, validating both efficiency and effectiveness. Figure 2 : DeCafNet Overview. The sidekick encoder efficiently extracts features from input video clips, which, combined with text features, generate a saliency map to select the most salient clips. The expert encoder then processes these salient clips. DeCaf-Grounder uses the extracted features from both the sidekick and the expert encoders to predict the moment associated with the input text query. 2 Related Works SVTG methods aim to locate specific events within short videos, typically lasting from a few seconds to a few minutes [ 36 , 41 , 16 , 21 ] . There is extensive research in this area, which generally falls into proposal-based and proposal-free methods. Earlier proposal-based approaches have used techniques ranging from sliding windows [ 10 , 52 , 11 , 14 ] to ranking mechanisms [ 47 , 48 , 56 ] , to identify candidate segments. Proposal-free methods [ 23 , 51 , 33 ] , on the other hand, leverage Transformer-based algorithms to directly predict start and end points of events. Initial efforts in SVTG detection focused on localizing predefined action categories [ 49 , 63 ] , meanwhile recent approaches [ 10 , 21 , 33 ] have broadened their scope by using free-form text queries, such as captions, to locate specific moments. These newer methods explore diverse