STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding
Aaryan Garg, Akash Kumar, Yogesh S Rawat · 2025
Read STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding by Aaryan Garg, Akash Kumar, Yogesh S Rawat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20678.txt.
Introduction
Aaryan Garg, Akash Kumar, and Yogesh S Rawat tackle video grounding without box labels in a twenty twenty-five computer science paper. Their task locates a described subject in space and time inside a video. Only a text query is given for training. Recent vision-language models offer a starting point, but simple use of them falls short.
The team first links text queries to short video segments called tubelets. Then their STPro system learns in stages. One module builds up complex actions step by step. Another module raises scene difficulty in a guided way. Tests on three benchmark datasets show gains over earlier methods, including on crowded scenes.
Catalog record
| CREATOR | Aaryan Garg, Akash Kumar, Yogesh S Rawat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20678
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding Aaryan Garg, Akash Kumar, Yogesh S Rawat Abstract In this work we study Weakly Supervised Spatio-Temporal Video Grounding (WSTVG), a challenging task of localizing subjects spatio-temporally in videos using only textual queries and no bounding box supervision. Inspired by recent advances in vision-language foundation models, we investigate their utility for WSTVG, leveraging their zero-shot grounding capabilities. However, we find that a simple adaptation lacks essential spatio-temporal grounding abilities. To bridge this gap, we introduce Tubelet Referral Grounding (TRG) , which connects textual queries to tubelets to enable spatio-temporal predictions. Despite its promise, TRG struggles with compositional action understanding and dense scene scenarios. To address these limitations, we propose STPro , a novel progressive learning framework with two key modules: (1) Sub-Action Temporal Curriculum Learning (SA-TCL) , which incrementally builds compositional action understanding, and (2) Congestion-Guided Spatial Curriculum Learning (CG-SCL) , which adapts the model to complex scenes by spatially increasing task difficulty. STPro achieves state-of-the-art results on three benchmark datasets, with improvements of 1.0% on VidSTG-Declarative and 3.0% on HCSTVG-v1. 1 Introduction Spatio-temporal video grounding (STVG) seeks to detect and localize objects within video sequences across both space and time, guided solely by textual descriptions. This capability is crucial for applications like surveillance, autonomous navigation, and scene understanding. However, STVG presents significant challenges, requiring models to distinguish target objects from distractors over time and to identify precise temporal boundaries of actions. While recent methods tackle STVG in fully-supervised settings ( Yang et al. 2022 ; Jin et al. 2022 ; Wasim et al. 2024 ) , they rely on costly, labor-intensive frame-level annotations. In this work, we address these challenges through weakly supervised STVG (WSTVG), training models with high-level video descriptions and eliminating the need for detailed spatio-temporal labels. Existing research in weakly supervised learning for grounding has focused primarily on image-based phrase Datta et al. 2019 ; Wang et al. 2020 ; Liu et al. 2021a and referral grounding Liu et al. 2019 ; Liu et al. 2022 ; Jin et al. 2023 . More recently, studies have extended weak supervision to the STVG task Shi et al. 2019 ; Chen et al. 2019a ; Li et al. 2023 ; Jin and Mu 2024 . However, these approaches are often complex, involving multiple hierarchical algorithms ( Li et al. 2023 ) , additional modality information ( Chen et al. 2019a ) , and restricted object categories ( Shi et al. 2019 ; Chen et al. 2019a ) . To address these limitations, we propose a streamlined approach that relies solely on the vision modality and supports free-form text queries. Recent advancements in vision-language foundation models (VFMs) Madan et al. 2024 have shown remarkable zero-shot grounding abilities in image-based referral tasks, driven by large-scale pretraining on diverse multimodal datasets Lin et al. 2023a ; Zhang et al. 2025 ; Liu et al. 2024 . These VFMs excel at associating visual entities with complex, free-form textual descriptions, making them highly effective in scenarios where image-based object localization must generalize across a variety of contexts. Grounding DINO (G-DINO) Liu et al. 2024 , one of the leading VFMs, has demonstrated robust zero-shot performance on numerous image-level grounding benchmarks. However, while recent work has adapted G-DINO for fully supervised video grounding Wasim et al. 2024 , the potential of VFMs remains unexplored in weakly supervised spatio-temporal video grounding (WSTVG), where precise annotations are not available. Applying image-based VFMs directly to videos introduces significant challenges. Unlike static images, videos often contain target subjects that are only visible in certain frames, with dynamic interactions that require an understanding of both spatial and temporal continuity. Simply extending G-DINO to videos results in noisy training due to irrelevant frames and complex action sequences that it cannot accurately localize. To address these limitations, we introduce the Tubelet Referral Grounding (TRG) module, which adapts image-based VFMs to spatio-temporal contexts by linking textual queries with tubelets (spatio-temporal object proposals). While TRG provides a strong baseline for adapting VFMs to WSTVG, it has two main shortcomings: it struggles with complex free-form queries involving actor-action relationships and lacks an ability to handle dense scenes with ordered action compositions, limiting its effectiveness in complex scenarios. To overcome these challenges, we propose Spatial and Temporal Progressive Learning (STPro) , a framework specifically designed to enhance VFMs for WSTVG tasks. STPro introduces two progressive learning strategies: (1) Sub-Action Temporal Curriculum Learning (SA-TCL) , which incrementally develops the model’s understanding of complex action sequences by gradually increasing task difficulty over time, and (2) Congestion-Guided Spatial Curriculum Learning (CG-SCL) , which focuses the model on progressively sparse scenes. Together, these strategies help the model adapt to intricate actor-action relationships and noisy, free-form textual descriptions, significantly enhancing the VFM’s capability to handle spatio-temporal grounding in weakly supervised settings. In summary, we make the following contributions: • We introduce Temporal Referral Grounding (TRG) module, which adapts foundation models for the weakly supervised spatio-temporal video grounding task, providing a strong baseline for handling temporal and spatial contexts. • We propose the Sub-Action Temporal Curriculum Learning (SA-TCL) paradigm, which improves the model’s temporal grounding capability by incrementally increasing the complexity of action sequences, enabling better handling of temporal dependencies in videos. • We introduce Congestion-Guided Spatial Curriculum Learning (CG-SCL) , a novel approach that progressively adapts the model to more complex and dense scene structures, improving its ability to understand spatial relationships in challenging video scenarios. We perform our experiments on three different benchmark datasets, ViDSTG and HCSTVG-v1 and HCSTVG-v2 demonstrating effectiveness of our proposed approach. STPro outperform previous state-of-the-art methods on WSTVG task by an absolute margin of 1.0% on VidSTG-Declarative and 3.0% on HCSTVG-v1. 2 Related Work Spatio-Temporal Video Grounding: It requires grounding video tubelets (detections across time) to textual queries, aligning both spatial and temporal dimensions. Traditional methods use a two-stage pipeline, separating spatial ( Rohrbach et al. 2015 ; Yamaguchi et al. 2017 ) and temporal grounding ( Gao et al. 2017 ) but rely on fixed object categories, limiting adaptability for free-form queries. Recent multimodal approaches ( Su et al. 2021 ; Yang et al. 2022 ; Jin et al. 2022 ) combine image detectors, video encoders, and spatio-temporal decoders to address feature alignment and leverage both static and motion cues ( Lin et al. 2023c ; Gu et al. 2024 ) . These methods typically require extensive frame-level annotations. In contrast , we remove the need for spatio-temporal labels, greatly reducing annotation cost. Weakly Supervised Learning : Existing works on dense tasks ( Kumar et al. 2025b ; Singh et al. 2024 ; Kumar et al. 2023 ; Rana and Rawat 2023 ; Kumar and Rawat 2022 ; Rana and Rawat 2022 ; Dave et al. 2022 ; Modi et al. 2022 ) are unimodal, rely on semi-supervised or active learning, and cannot be extended to WSSTVG. Grounding techniques fall into three categories: phrase grounding ( Rohrbach et al. 2016b ; Datta et al. 2019 ) (identifying objects from text using margin-based losses ( Datta et al. 2019 ) , contrastive learning ( Gupta et al. 2020 ) , and reconstruction ( Rohrbach et al. 2016b ) ); referral grounding ( Liu et al.