Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos

Tuyen Tran, Thao Minh Le, Quang-Hung Le, Truyen Tran · 2025

Read Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos by Tuyen Tran, Thao Minh Le, Quang-Hung Le, Truyen Tran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07330.txt.

Introduction

Pointing at the right thing in a video from a spoken description means tracking who does what, and when. Tuyen Tran, Thao Le, Quang-Hung Le, and Truyen Tran built a two-part system in 2025 that takes the task in steps. A planner chops a long description into a chain of short noun-and-verb phrases; a refiner then sharpens the visual tokens phrase by phrase, first across the frame and then across time.

The loop keeps the refined picture features from step to step, so later phrases build on earlier ones. On tests for finding described objects and moments, including a new benchmark with long queries, the system beat the best prior methods, with the clearest gains on hard prompts.

Catalog record

CREATORTuyen Tran, Thao Minh Le, Quang-Hung Le, Truyen Tran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos Tuyen Tran, Thao Minh Le, Quang-Hung Le, Truyen Tran Abstract Vision-language alignment in video must address the complexity of language, evolving interacting entities, their action chains, and semantic gaps between language and vision. This work introduces Planner-Refiner , \text{Planner-Refiner}, a framework to overcome these challenges. Planner-Refiner bridges the semantic gap by iteratively refining visual elements’ space-time representation, guided by language until semantic gaps are minimal. A Planner module schedules language guidance by decomposing complex linguistic prompts into short sentence chains. The Refiner processes each short sentence—a noun-phrase and verb-phrase pair—to direct visual tokens’ self-attention across space then time, achieving efficient single-step refinement. A recurrent system chains these steps, maintaining refined visual token representations. The final representation feeds into task-specific heads for alignment generation. We demonstrate Planner-Refiner ’s effectiveness on two video-language alignment tasks: Referring Video Object Segmentation and Temporal Grounding with varying language complexity. We further introduce a new MeViS-X benchmark to assess models’ capability with long queries. Superior performance versus state-of-the-art methods on these benchmarks shows the approach’s potential, especially for complex prompts. Our code is available at: https://github.com/tranxuantuyen/Planner-Refiner † † orcid: 0009-0002-8161-7637 † † orcid: 0000-0002-8089-9962 † † orcid: 0000-0003-4727-6859 † † orcid: 0000-0001-6531-8907 1 Introduction Figure 1: Our Planner-Refiner (bottom-right), compared with the current single-pass approach (top-right). Ours is a recurrent system that iteratively refines the visual representation across space-time conditioned on the language. The linguistic input of each step is planned by a Recurrent Step Planner, which decomposes the input prompt into a chain of short sentences. This mechanism enables more fine-grained refinement of visual tokens, resulting in improved accuracy in predictions. Video-language alignment has emerged as a critical challenge in multimodal understanding, particularly in bridging the semantic gaps between low-level visual content and linguistic concepts [ Rizve et al.(2024)Rizve, Fei, Unnikrishnan, Tran, Yao, Zeng, Shah, and Chilimbi , Singh et al.(2022)Singh, Hu, Goswami, Couairon, Galuba, Rohrbach, and Kiela , Tran(2024) , Ding et al.(2024)Ding, Hong, Liu, Xu, Yang, Fan, Miao, Gu, Li, He, et al. ] . While recent large video-language models have demonstrated impressive capabilities, their effectiveness remains largely limited to matching simple textual concepts to visual regions [ Liu et al.(2023)Liu, Li, Wu, and Lee , Li et al.(2023)Li, Li, Savarese, and Hoi ] . This falls short to meet practical requirements that demand a sophisticated understanding of complex concepts featuring multiple components within a single query. Consider a video surveillance scenario [ Yuan et al.(2024)Yuan, Zhang, Liu, Liu, Chen, Jin, and Jiao ] where machines are required to process a complex query like “Find in video a person wearing a red jacket enters the room, picks up a black bag, and exits through the left door”. Such queries require simultaneous comprehension of a series of interrelated components including cross-modality subject and attribute identification, modeling object dynamics in space-time and understanding cross-object dependencies. The challenge lies in developing computer systems that are capable of modeling the interactions of multiple visual entities in space and time in reflection the interpretation of a given query [ Xue et al.(2022)Xue, Sun, Liu, Fu, Song, Li, and Luo , Munasinghe et al.(2024)Munasinghe, Gani, Zhu, Cao, Xing, Khan, and Khan , Tran et al.(2024)Tran, Le, Tran, and Tran ] . This is not a trivial task especially in complex scenes where the involved entities perform intricate sequences of actions and interactions. Fig. 1 provides such an example. Here, we consider the task of identifying a video segment featuring a horse with a sequence of actions, including faces right , turns to the left , swings its tail and strikes the other horse’s head . Current state-of-the-art approaches, such as DsHmp [ He and Ding(2024) ] , can successfully recognize the presence of objects (e.g., horses), but struggle with comprehending complex action sequences associated with specific instances. There are two fundamental limitations in the current state-of-the-art approaches. First, base vision-language systems are typically pretrained on datasets that focus on isolated, single-concept scenarios [ Li et al.(2023)Li, Li, Savarese, and Hoi , Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al. , Smaira et al.(2020)Smaira, Carreira, Noland, Clancy, Wu, and Zisserman ] , thus incapable of reasoning about compositional scenarios involving complex relationships between elements. Second, current methods often overly focus on visual modeling while employing oversimplified approaches to handle complex queries. Works such as [ Heo et al.(2022)Heo, Hwang, Oh, Lee, and Kim , Ding et al.(2023a)Ding, Liu, He, Jiang, and Loy , Miao et al.(2024)Miao, Bennamoun, Gao, Shah, and Mian , Cheng et al.(2022)Cheng, Misra, Schwing, Kirillov, and Girdhar ] compress entire input sequences into single embeddings, neglecting crucial linguistic structural information about objects interactions that goes beyond the ordinary autoregressive modeling of sequence of words. While DsHmp [ He and Ding(2024) ] partially addresses this limitation by breaking a textual query into static and motion cues, its rigid two-component architecture proves inadequate for compositional sentences with varying complexity levels. To address these limitations, we introduce Planner-Refiner , a novel framework that minimizes visual-linguistic semantic gaps through iterative refinement of visual representations in space-time guided by language. At its core, Planner-Refiner employs two key components: a language-based step Planner module that decomposes a complex textual query, also referred to as the referring prompt, into atomic action sequences, and a following Refiner recurrent system that progressively refines visual representations as the action sequences revealed. More concretely, the Planner dynamically creates a variable-length sequence of sub-prompts, each comprising a noun phrase (object) and a verb phrase (action), with the sequence’s length determined by the syntactic structure of the input query. These sub-prompts are ordered in temporal dimension, reflecting the specific action chain performed by the target instance. They then serve as guide to the Refiner to iteratively refine spatial and temporal visual tokens. The refinements of these visual tokens are performed separately following the English grammar structure where noun typically comes before verb in a simple sentence. This architecture enables progressive improvement in understanding both objects and their actions over time, facilitating more accurate vision-language alignments. We demonstrate the effectiveness of our approach through comprehensive experiments on two video-language alignment tasks: Referring Video Object Segmentation (RVOS) and Video Temporal Grounding (VTG). Our results show that Planner-Refiner achieves a 20% relative improvement in segmentation accuracy over the baseline on the RVOS task. Additionally, we introduce a novel MeViS-X benchmark, specifically designed to evaluate model performance on complex, compositional descriptions. Planner-Refiner demonstrates significant improvements over state-of-the-art methods on this benchmark, underscoring its effectiveness in handling complex video-language tasks, particularly those involving dynamic, temporally extended descriptions. In summary, our contributions include: i. A novel two-component system for vision-language alignment in videos, consisting of a Language-based Step Planner that dynamically determines a sequence of reasoning steps based on query complexity, followed by a Language-guided Visual Token Refiner for language-guided refinement of visual tokens. ii. The MeViS-X benchmark for evaluating reasoning capability with complex sentences in the RVOS task. iii. Extensive experiments and analysis across video-language alignment tasks, providing insights into the effectiveness of the proposed approach towards solving such a challenging task. 2 Related Work Recent large-scale vision-language models [ Li et al.(2023)Li, Li, Savarese, and Hoi , Munasinghe et

Read the complete text