Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

Tianshu Zhang, Yan Wang, Ji Qi, Lijie Wen · 2026

Read Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification by Tianshu Zhang, Yan Wang, Ji Qi, Lijie Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29023.txt.

Introduction

Zhang, Wang, Qi and Wen present a pipeline for finding where and when an object described in words appears in a long video. This task, spatio-temporal grounding, demands precise timing plus robust tracking, and modern vision-language models can reason about it but are costly and unstable when applied frame by frame. Their pipeline instead tracks at the level of seconds, smoothing across seconds to keep continuity while shortening the sequence the model must handle.

To teach reasoning, the team synthesise step-by-step trajectories with advanced multimodal models, then swap generated coordinates for ground-truth ones. A reinforcement learning stage tunes the policy, scored by a verifier combining time and video overlap. Across frame-rate settings, the method achieved a strong trade-off between speed and localisation quality.

Catalog record

CREATORTianshu Zhang, Yan Wang, Ji Qi, Lijie Wen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification Tianshu Zhang, Yan Wang, Ji Qi, Lijie Wen Abstract Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries. While recent vision-language models (VLMs) show strong reasoning ability, directly applying frame-by-frame inference to long sequences is computationally expensive and unstable. We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second smoothing to preserve continuity while reducing sequence length. To improve reasoning supervision, we synthesize chain-of-thought style trajectories using advanced multimodal models for temporal localization and target selection, and replace generated spatio-temporal coordinates with ground-truth annotations to avoid noisy supervision. We further optimize the policy with reinforcement learning using a verifier based on t ​ _ ​ IoU + m ​ v ​ _ ​ IoU t\_\mathrm{IoU}+mv\_\mathrm{IoU} . Experiments across multiple FPS settings show that our method achieves a strong trade-off between efficiency and localization quality. 1 Introduction Spatio-temporal video grounding (STVG) asks a model to localize a natural-language query in both time and space ( 30 ) . Given a video and a sentence such as “the person in red picks up the bag”, the model must identify the event interval and track the referred target throughout it. The task combines temporal moment localization, referring expression comprehension, object tracking, and video-language reasoning, making it a natural benchmark for language-controllable video understanding. STVG provides a more complete grounding interface than temporal localization, frame-level referring localization, or tracking alone. Temporal-only methods cannot identify where the described target is, while tracking methods do not solve language-conditioned search by themselves. STVG combines these requirements into a single task, making it useful for video retrieval, interactive annotation, human activity understanding, robot perception, and video agents. Existing systems make progress by coupling proposal generation, temporal boundary prediction, detection, tracking, cross-modal fusion, and target-aware reasoning ( 47 ; 14 ) , but many still rely on dense frame processing or specialized modules whose cost grows quickly with video length. Figure 1: Comparison between our method and traditional VLMs. Left: existing VLMs performing per-frame tracking lead to token explosion. Right: our proposed per-second tracking mode with a post-smoothing strategy and the spatio-temporal reasoning paradigm. The difficulty becomes more pronounced in long videos. A short query may describe a brief action inside long background context, and the referred object may remain visible before and after the event with different relevance. Dense sampling preserves boundary cues but increases visual tokens and tracking states; sparse sampling is cheaper but may miss decisive transitions. Even after the correct segment is found, appearance change, occlusion, fast motion, and distractors can cause identity switches or jitter. Recent MLLMs offer a promising alternative because they provide stronger language understanding, broader visual knowledge, and structured intermediate reasoning. For STVG, this is appealing: complex queries often require parsing attributes, relations, actions, and event order before deciding when and where the target appears. MLLMs can connect such linguistic constraints to visual evidence and express decisions in a flexible output format. However, there is still a clear gap between this promise and a robust STVG system. Direct frame-level MLLM grounding requires long visual contexts, fine temporal reasoning, and precise text-generated trajectories. One workaround is to input frames iteratively and track step by step ( 10 ) , but this turns long-video grounding into many sequential MLLM calls and makes the wall-clock cost too high. MLLM-based STVG therefore needs a formulation that preserves semantic reasoning while controlling sequence length and grounding noise. Our work focuses on this practical missing piece: how should an MLLM-based STVG system handle long videos while producing usable temporal and spatial grounding? We argue that the basic reasoning unit should be the second rather than the frame. Many STVG queries describe actions and interactions that persist across short temporal neighborhoods; second-level units therefore shorten the sequence, preserve semantic event structure, and yield compact trajectories. A lightweight inter-second smoothing step restores continuity and reduces jitter. The second-level formulation also changes the training problem. We use three stages: CPT for broad spatio-temporal grounding, supervised fine-tuning for structured target and event reasoning, and reinforcement learning for metric-aligned localization. During SFT, generated traces expose temporal and identity decisions, but final coordinates and segments are replaced with annotations; RL then scores complete predictions with temporal and motion-aware spatial overlap, aligning optimization with evaluation more directly than token likelihood alone. Taken together, our formulation couples second-level tracking, ground-truth-corrected reasoning supervision, and verifier-driven RL. These components address long visual inputs, drift-prone traces, text-token coordinates, and metric-misaligned supervision. We evaluate on VidSTG subsets and HC-STVG, with FPS sensitivity and ablation studies analyzing the efficiency-quality trade-off. Our contributions are summarized as follows: • We formulate MLLM-based STVG around second-level tracking, reducing long-video sequence cost while maintaining track continuity through inter-second smoothing. • We develop a three-stage training pipeline that combines CPT, corrected reasoning SFT, and verifier-driven RL for spatio-temporal localization. • We construct reasoning supervision with generated decision traces and ground-truth coordinate replacement, separating useful semantic reasoning from unreliable generated geometry. • We optimize with a verifier based on t_{\mathrm{IoU}}+mv_{\mathrm{IoU}} and evaluate across VidSTG subsets, HC-STVG, multiple FPS settings, and targeted ablations. 2 Related Work Video grounding and spatio-temporal localization. Spatio-temporal video grounding (STVG) localizes both the temporal interval and frame-level trajectory of a target described by language. Early non-MLLM studies approached this through language-guided video retrieval, object referring, and sentence-guided video segmentation 42 ; 37 ; 12 . Later conventional methods rely on task-specific graph reasoning, two-stage localization, visual-linguistic Transformers, DETR-style tube prediction, consistency modeling, context-guided decoding, and open-vocabulary grounding 32 ; 43 ; 17 ; 20 ; 13 ; 40 . These systems provide strong task-specific baselines, but they remain specialized cross-modal matching or tube regression pipelines with limited high-level reasoning and open-world knowledge. Grounding in Multimodal Large Language Models. Recent MLLMs show promising grounding ability, but most work studies either spatial grounding in images or temporal grounding in videos. Spatial grounding models such as Qwen-VL ( 1 ) connect language to regions through location tokens, natural-language coordinates, image-caption-box alignment, referring mechanisms, or masks, while video MLLMs such as TimeChat ( 28 ) and query-enrichment methods ( 26 ) localize relevant moments from language. Later MLLMs ( 35 ; 2 ; 34 ) increasingly treat spatial and temporal grounding as core capabilities, yet they are still often evaluated separately ( 5 ; 22 ) ; methods that jointly predict temporal boundaries and spatial trajectories remain less common ( 36 ; 19 ; 6 ) . Recent systems such as LLaVA-ST ( 18 ) and SpaceVLLM ( 39 ) explore fine-grained spatial-temporal outputs, and other work studies zero-shot STVG or reinforcement fine-tuning ( 10 ; 21 ; 46 ; 15 ) . However, many approaches depend on architecture changes ( 3 ; 44 ; 38 ) or external-tool pipelines ( 11 ; 41 ; 25 ; 41 ) , limiting generality ( 45 ) . Our work instead studies joint spatial-temporal localization and RL enhancement without changing architecture or inference pipelines. 3 Method 3.1 Problem Definition We study spatio-temporal video grounding (STVG) under a language-conditioned setting. Given an input video V=\{f_{1},\dots,f_{T}\} with frame rate F and a natural-language query

Read the complete text