QTrack: Query-Driven Reasoning for Multi-modal MOT

Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav †, Abrar Ul Riyaz, Wasif Tak, Moloud Abdar, Janibul Bashir · 2026

Read QTrack: Query-Driven Reasoning for Multi-modal MOT by Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav †, Abrar Ul Riyaz, Wasif Tak, Moloud Abdar, Janibul Bashir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13759.txt.

Introduction

Tajamul Ashraf and co-authors introduce QTrack, a query-driven method for multi-modal multi-object tracking. Some applications need only user-specified targets tracked, not every detected instance in a clip. Their offline task gives a video clip, a natural-language query, and reference-frame boxes for the targets. QTrack is a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization. That training contrasts outputs under the observed clip versus a repeated reference frame to force use of visual evidence.

Catalog record

CREATORTajamul Ashraf, Tavaheed Tariq, Sonia Yadav †, Abrar Ul Riyaz, Wasif Tak, Moloud Abdar, Janibul Bashir
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69457
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13759

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QTrack: Query-Driven Reasoning for Multi-modal MOT Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav †, Abrar Ul Riyaz, Wasif Tak, Moloud Abdar, Janibul Bashir Abstract Multi-object tracking ( MOT ) has traditionally focused on estimating trajectories of all objects in a video, without selectively reasoning about user-specified targets under semantic instructions. In this work, we introduce a query-driven tracking paradigm that formulates tracking as a spatiotemporal reasoning problem conditioned on natural language queries. Given a reference frame, a video sequence, and a textual query, the goal is to localize and track only the target(s) specified in the query while maintaining temporal coherence and identity consistency. To support this setting, we construct RMOT26 , a large-scale benchmark with grounded queries and sequence-level splits to prevent identity leakage and enable robust generalization evaluation. We further present QTrack , an end-to-end vision language model that integrates multi-modal reasoning with tracking-oriented localization. Additionally, we introduce a Temporal Perception-Aware Policy Optimization strategy with structured rewards to encourage motion-aware reasoning. Extensive experiments demonstrate the effectiveness of our approach for reasoning-centric, language-guided tracking. Code and data are available at https://github.com/gaash-lab/QTrack Project Page: https://gaashlab.github.io/QTrack/ 1 Why Query-Driven Tracking? Multi-object tracking ( MOT ) is a fundamental computer vision task that aims to detect and consistently associate multiple objects across frames temporally [ 49 ] . Given a video, traditional MOT methods [ 7 , 28 , 37 ] primarily aim to estimate the trajectories of all moving objects, i.e., answering “what are the locations and identities of objects over time?” . These approaches operate at the category level with predefined classes and are not designed for query-specific, language-conditioned tracking of a particular target. Despite considerable advancements in the deep learning era [ 39 , 43 , 12 ] , focusing solely on spatial localization in existing MOT frameworks is insufficient for comprehensive video understanding in many real-world applications. For example, beyond estimating trajectories, understanding trajectory-associated semantic 1 1 1 Here, by “semantic”, we refer to high-level, trajectory-based activity understanding in videos within the context of tracking, rather than category-level labeling as in semantic segmentation. attributes such as the identity, role, or contextual relevance of a queried target is crucial for real-world applications like surveillance and robotics. For instance, consider a crowded street scene where multiple individuals wear similar clothing. A query such as “track the person who picks up the red backpack and later enters the taxi” cannot be solved through appearance similarity alone. The model must reason about temporally evolving actions and maintain identity consistency even when the target undergoes occlusion or appearance changes. This motivates extending conventional MOT (addressing “where”) toward query-conditioned tracking that jointly reasons about “which” target to follow based on semantic instructions. Figure 1 : Comparison of tracking paradigms. (a) Traditional MOT follows a tracking-by-detection paradigm, tracking all objects from predefined categories regardless of user intent. (b) QTrack enables reasoning-aware, query-conditioned tracking: given a video and natural language query, it selectively identifies and tracks only the specified targets, shifting from all-object tracking to semantic, user-driven tracking. In parallel, vision-language models ( VLMs ) have demonstrated remarkable capabilities in multi-modal grounding [ 59 , 72 ] , visual reasoning [ 54 , 64 , 3 ] , and following [ 17 , 76 ] . By aligning visual representations with natural language semantics, VLMs enable fine-grained instance understanding beyond category-level perception. While prior works on referring video object segmentation and language-guided tracking focus on grounding a static description to a target instance, they often treat tracking and reasoning as loosely coupled modules. These systems do not explicitly leverage semantic queries to guide identity persistence, motion prediction, or occlusion recovery. Motivated by these limitations, we introduce QTrack , a query-centric tracking paradigm that extends conventional MOT beyond trajectory estimation toward language-conditioned, target-specific reasoning. Unlike traditional MOT methods that focus solely on predicting object trajectories (“where”), this paradigm enables tracking of objects specified through natural language queries, integrating spatial localization with semantic grounding. Given a video, a reference grounding, and a query (e.g., “track the person wearing a red jacket”), the model identifies and persistently tracks only the specified instance across frames. This requires joint reasoning over appearance, motion dynamics, contextual cues, and temporal continuity, emphasizing which object to follow and why , rather than indiscriminately tracking all category-level instances. By unifying “where” (trajectory estimation) with “which” (query-grounded selection), the framework advances tracking from passive perception to reasoning-aware, language-guided video understanding. To realize this paradigm, we develop a reinforcement learning-optimized vision–language framework for temporally consistent, query-conditioned object localization. Given a video and a query, the model generates structured reasoning and directly predicts bounding boxes for the queried targets in an end-to-end manner. Unlike detect-then-track pipelines or loosely coupled vision–language systems, our approach jointly optimizes grounding and temporal consistency. Specifically, we introduce Temporal Perception-Aware Policy Optimization (TAPO), which incorporates temporal corruption and KL-based regularization to encourage motion-aware reasoning and identity persistence. While standard supervised training optimizes frame-level localization losses and may suffer from identity drift under long-term dependencies, our reinforcement learning formulation directly optimizes sequence-level objectives, promoting coherent motion dynamics and robust identity preservation across frames. To facilitate research on query-driven tracking with reasoning, we introduce RMOT26 , which consists of a diverse collection of real-world video sequences spanning crowded scenes, sports, complex interactions, non-linear motion, and severe occlusions. Each sample is paired with (i) a reference frame with explicit spatial grounding of the target instance and (ii) a natural-language query specifying the tracking objective (e.g., identity, attributes, role, or context). Unlike conventional MOT benchmarks that evaluate category-level tracking of all objects, RMOT26 explicitly tests a model’s ability to reason about which instance to follow and to maintain identity persistence under appearance ambiguity, long-term temporal dependencies, and occlusion recovery. We design multiple query types to probe single-target tracking, multi-target reasoning, and occlusion-aware re-identification, with strong spatial and temporal supervision derived from high-quality verified annotations. All splits are performed at the sequence level to prevent identity leakage and ensure robust generalization. Despite its streamlined design, QTrack demonstrates strong performance in language-guided tracking scenarios and significantly outperforms baselines that combine tracking and VLM reasoning in an offline manner, highlighting the effectiveness of reinforcement learning-driven spatiotemporal reasoning. In summary, our contributions are: • We introduce QTrack , a novel query-driven, reasoning-centric tracking paradigm that advances MOT from passive trajectory prediction to language-conditioned spatiotemporal reasoning, enabled by our Temporal Perception-Aware Policy Optimization ( TAPO ) and structured reward design. • We construct RMOT26 , a large-scale benchmark with grounded natural language queries and temporally consistent supervision to systematically evaluate reasoning-aware tracking. • We demonstrate strong performance and provide in-depth analysis on 10+ VLM baselines to guide future research on reasoning-aware, multimodal tracking systems. 2 Related Work 2.1 Tracking and Language-Guided Grounding Benchmarks have played a critical role in advancing MOT , from early datasets such as PETS2009 [ 19 ] to the MOT Challenge series [ 40 , 29 , 16 , 15 ] . Domain-specific benchmarks include KITTI [ 60 ] and BDD100K [ 71 ] for autonomous driving, ImageNet-VID [ 24 ] and TAO [ 14 ] for large-scale and long-tail tracking, DanceTrack [ 52 ] and SportsMOT [ 13 ] for articulated human motion, AnimalTrack [ 73 ] for wildlife

Read the complete text