ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking
Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao Huazhong University of Science and Technology {sijiachen, yanqiuyu, yuen, wenbingtao}@hust.edu.cn https://github.com/chen-si-jia/ReaMOT · 2025
Read ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking by Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao Huazhong University of Science and Technology {sijiachen, yanqiuyu, yuen, wenbingtao}@hust.edu.cn https://github.com/chen-si-jia/ReaMOT (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20381.txt.
Introduction
Sijia Chen and three co-authors set a harder task for video tracking in 2025: follow objects picked out by instructions that demand reasoning, not just matching words to looks. They built the ReaMOT benchmark with 1,156 instructions, 869 video clips and six scenario types. Over three quarters of the instructions need high-level inference about hidden conditions. A tailored set of scores measures success.
Their tracker, ReaTrack, splits the job in two. A large vision-language model handles the thinking and locates the right targets, while a strong segmenter keeps motion smooth frame to frame. The design needs no training and avoids the flicker that direct model use produces. On the reasoning-heavy subset it more than tripled the leading score. The team released data and code for others to build on.
Catalog record
| CREATOR | Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao Huazhong University of Science and Technology {sijiachen, yanqiuyu, yuen, wenbingtao}@hust.edu.cn https://github.com/chen-si-jia/ReaMOT |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20381
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao Huazhong University of Science and Technology {sijiachen, yanqiuyu, yuen, wenbingtao}@hust.edu.cn https://github.com/chen-si-jia/ReaMOT Abstract Referring Multi-object tracking (RMOT) is an important research field in computer vision. Its task form is to guide the models to track the objects that conform to the language instruction. However, the RMOT task commonly requires clear language instructions, such methods often fail to work when complex language instructions with reasoning characteristics appear. In this work, we propose a new task, called Rea soning-based M ulti- O bject T racking ( ReaMOT ). ReaMOT is a more challenging task that requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects’ trajectories. To advance the ReaMOT task and evaluate the reasoning capabilities of tracking models, we construct ReaMOT Challenge , a reasoning-based multi-object tracking benchmark built upon 12 datasets. Specifically, it comprises 1,156 language instructions with reasoning characteristic, 423,359 image-language pairs, and 869 diverse scenes, which is divided into three levels of reasoning difficulty. In addition, we propose a set of evaluation metrics tailored for the ReaMOT task. Furthermore, we propose ReaTrack , a training-free framework for reasoning-based multi-object tracking based on large vision-language models (LVLM) and SAM2, as a baseline for the ReaMOT task. Extensive experiments on the ReaMOT Challenge benchmark demonstrate the effectiveness of our ReaTrack framework. 1 Introduction Multi-object tracking (MOT) is one of the most challenging tasks in computer vision, widely used in video surveillance [ 44 ] , autonomous driving [ 26 ] , intelligent transportation [ 6 ] and other fields. Existing MOT methods have demonstrated effectiveness in handling most visual generic scenarios. However, in visual-language scenarios, traditional MOT tasks face huge challenges and limitations. To solve this problem, the referring multi-object tracking (RMOT) task [ 43 ] was recently proposed, which requires the models to track the objects that conform to the language instruction. Since the RMOT task generally relies on clear language description, such methods often fail to work when faced with complex language instructions that require reasoning. However, such language instructions are common in daily life, making it crucial for models to effectively understand them. To bridge this gap, it is helpful to examine how humans handle such instructions. Human reasoning abilities can be demonstrated in various aspects: (1) when the instruction is ambiguous, people can infer the actual targets behind this instruction; and (2) when the instruction describes a goal that is not directly observable, people can leverage prior knowledge to deduce the relevant entities. Motivated by these observations, we propose a new task called Rea soning-based M ulti- O bject T racking ( ReaMOT ), which is designed to handle the challenge of multi-object tracking under language instructions with complex reasoning. This task requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects’ trajectories. As shown in Figure 1 (a), for the language instruction "The cars on the right.", the models of the RMOT task only need to track the cars on the right, as referred to by the language instruction, without going through a series of complex reasoning processes. In contrast, as shown in Figure 1 (b), for the language instruction "The players of that team with clear tactics and high organization.", the ReaMOT task requires the models to engage in a deep reasoning process, analyzing aspects such as offensive strategies, tactical execution, defensive positioning, and reaction speed of both teams, to accurately deduce the players of the target team that match the language instruction and track their trajectories. Figure 1 : The difference between RMOT and ReaMOT tasks. Unlike RMOT, the ReaMOT task requires the models to engage in a deep reasoning process for deducing and tracking the targets. To advance the research of the ReaMOT task and evaluate the reasoning capabilities of tracking models, we propose a benchmark, called ReaMOT Challenge . Specifically, the ReaMOT Challenge includes 1,156 language instructions with reasoning characteristic, 423,359 image-language pairs, and 869 different scenes. In addition, we adopt a new annotation method, fully combining the capabilities of GPT-assisted annotation with the accuracy of manual annotation. Moreover, we design a set of reasoning difficulty scoring rules to quantify the reasoning difficulty of language instructions. Based on these rules, we divide the ReaMOT Challenge into three levels of reasoning difficulty: Easy, Medium, and Hard. Finally, we propose a set of evaluation metrics tailored for the ReaMOT task. With the emergence of large vision-language models (LVLM) [ 5 ] , it is no longer difficult to understand complex language instructions that require reasoning, making it possible to accurately reason about intentions of language instructions and perform corresponding multi-object tracking. Therefore, we propose ReaTrack , a training-free reasoning-based multi-object tracking framework, as a baseline for the ReaMOT task. Specifically, ReaTrack combines the powerful reasoning ability of large vision-language models (LVLM) [ 5 ] and the accurate tracking ability of SAM2 [ 36 ] . Finally, we evaluate the ReaTrack framework on the ReaMOT Challenge benchmark under zero-shot settings and compare it against several competing methods, as well as different versions of ReaTrack using various LVLMs, including LLaVA-1.5 [ 29 ] , InternVL2.5 [ 10 ] , LLaVA-NEXT [ 28 ] , Qwen-VL-Chat [ 4 ] , and Qwen2.5-VL [ 5 ] . The variant of ReaTrack equipped with Qwen2.5-VL [ 5 ] achieves the best overall performance, and we report our main results based on this configuration. ReaTrack consistently outperforms other methods across three reasoning difficulty levels: Easy, Medium, and Hard, achieving top scores on four metrics: RIDF1, RMOTA, RRcll, and RPrcn. Compared to other methods, it improves RIDF1 by up to 37.50%, RMOTA by 12.53%, RRcll by 55.62%, and RPrcn by 27.87%, demonstrating its superior generalization and robustness under complex reasoning scenarios. In summary, our main contributions are as follows: 1. We propose a new task, called Rea soning-based M ulti- O bject T racking ( ReaMOT ). It is a challenging task that requires accurate reasoning about objects that match the language instruction with reasoning characteristic and tracking the objects’ trajectories. 2. We construct a benchmark, named ReaMOT Challenge , to advance the research on the ReaMOT task and evaluate the models’ reasoning capabilities. The benchmark includes 1,156 language instructions with reasoning characteristic, 423,359 image-language pairs and 869 diverse scenes, divided into three levels of reasoning difficulty: Easy, Medium, and Hard. 3. We propose a training-free framework for reasoning-based multi-object tracking, named ReaTrack , which is based on large vision-language models (LVLM) and SAM2. We evaluate the ReaTrack framework and other methods on the ReaMOT Challenge benchmark. The evaluation results show that our ReaTrack framework achieves state-of-the-art (SOTA) performance. 2 Related Work Multiple Object Tracking. We review three main frameworks for multiple object tracking: Tracking-by-Detection, Joint Detection and Tracking, and Transformer-based Tracking. Tracking-by-Detection [ 7 , 42 , 49 , 14 , 15 , 20 ] typically involves two core components. During the detection phase, an object detector is employed to identify and locate objects of interest within each frame. In the subsequent association phase, earlier approaches relied on motion prediction models to estimate the positions of objects in the