MV-TAP: Tracking Any Point in Multi-View Videos
Jahyeok Koo, Inès Hyeonsu KimMungyeom KimJunghyun ParkSeohyeon ParkJaeyeong KimJung YiSeokju ChoSeungryong Kim[10pt] KAIST AI https://cvlab-kaist.github.io/MV-TAP · 2025
Read MV-TAP: Tracking Any Point in Multi-View Videos by Jahyeok Koo, Inès Hyeonsu KimMungyeom KimJunghyun ParkSeohyeon ParkJaeyeong KimJung YiSeokju ChoSeungryong Kim[10pt] KAIST AI https://cvlab-kaist.github.io/MV-TAP (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02006.txt.
Introduction
Tracking a single point as it moves through a scene is a basic vision skill, and multiple cameras make it both richer and harder. Jahyeok Koo and co-authors present MV-TAP in 2025, a point tracker that follows points across multi-view videos of dynamic scenes by sharing information between views.
Multi-camera rigs observe complex real-world action from several angles at once, and understanding moving objects in that footage is central to many applications. A point hidden in one view may be plainly visible in another, and cross-view information lets the tracker exploit that. The method aims at steadier, longer tracks through crowded, changing scenes.
Catalog record
| CREATOR | Jahyeok Koo, Inès Hyeonsu KimMungyeom KimJunghyun ParkSeohyeon ParkJaeyeong KimJung YiSeokju ChoSeungryong Kim[10pt] KAIST AI https://cvlab-kaist.github.io/MV-TAP |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MV-TAP: Tracking Any Point in Multi-View Videos Jahyeok Koo, Inès Hyeonsu KimMungyeom KimJunghyun ParkSeohyeon ParkJaeyeong KimJung YiSeokju ChoSeungryong Kim[10pt] KAIST AI https://cvlab-kaist.github.io/MV-TAP Abstract Multi-view camera systems enable rich observations of complex real-world scenes, and understanding dynamic objects in multi-view settings has become central to various applications. In this work, we present MV-TAP, a novel point tracker that tracks points across multi-view videos of dynamic scenes by leveraging cross-view information. MV-TAP utilizes camera geometry and a cross-view attention mechanism to aggregate spatio-temporal information across views, enabling more complete and reliable trajectory estimation in multi-view videos. To support this task, we construct a large-scale synthetic training dataset and real-world evaluation sets tailored for multi-view tracking. Extensive experiments demonstrate that MV-TAP outperforms existing point-tracking methods on challenging benchmarks, establishing an effective baseline for advancing research in multi-view point tracking. Figure 1 : We present MV-TAP ( T racking A ny P oint in M ulti-view V ideos), a model designed to effectively integrate information across multiple viewpoints for robust and high-quality point tracking. (a) We visualize the results of MV-TAP on Harmony4D 23 . (b) MV-TAP achieves noticeable gains over other baselines 9 ; 21 ; 20 ; 6 ; 48 ; 44 ; 45 , demonstrating its ability to leverage multi-view information. † † footnotetext: ∗ Equal contribution 1 Introduction Multi-view camera systems have advanced significantly in recent years, enabling the understanding of complex real-world scenes. Building upon these advances, understanding dynamic objects in multi-view systems has become crucial for a wide range of applications, including motion capture 19 ; 5 ; 12 ; 23 , robot manipulation 47 ; 17 ; 35 , and autonomous driving 26 ; 3 . Point tracking has been widely used to understand dynamic scenes by providing fine-grained spatio-temporal correspondences across video frames and indicating point-wise occlusions 14 ; 8 ; 20 ; 6 . Hence, it supports a wide range of downstream tasks, including embodied AI 39 ; 2 , autonomous driving 1 , 4D reconstruction 41 ; 10 , and video editing 11 ; 18 . Despite its strong capability, existing point tracking methods have only been explored in single-view videos. Since the 2D projection of a 3D scene inherently incurs geometric ambiguities, such as frequent occlusion, erratic motion, and depth uncertainty, single-view methods 14 ; 8 struggle with these challenges. Consequently, a direct application of these trackers independently to each viewpoint fails to leverage the multi-view cues required to construct reliable multi-view point trajectories. Figure 2 : Motivation. We conceptually contrast our (c) multi-view point tracking with (a) single-view point tracking 9 ; 21 ; 20 ; 6 ; 48 ; 44 ; 45 and (b) multi-view matching 29 ; 32 ; 7 ; 33 ; 37 . Our approach simultaneously models both view- and frame-wise interactions to ensure cross-view and temporal consistency. Motivated by these observations, we formulate the novel task of multi-view point tracking , which aims to track a set of query points throughout multiple videos of a dynamic scene. Our key intuition is that jointly processing multiple video streams of a dynamic scene provides useful spatio-temporal constraints that can help resolve the ambiguities from monocular setting. For instance, a point that is occluded or motion-blurred in one view may be clearly visible in other views. By reasoning across these viewpoints simultaneously, such ambiguities can be resolved, enabling more robust and accurate point tracking. While multi-view correspondence has been studied as a promising way 29 ; 32 ; 7 ; 33 ; 37 to capture geometric relationships, these methods are often ill-suited for dynamic point tracking. They typically focus on static scenes 34 , assume rigid geometry, or require geometric priors 45 that are unavailable in casual, in-the-wild videos. Although a prior approach 31 targets multi-view 3D point tracking in the 3D world coordinate system, it relies on external depth inputs which may introduce reprojection errors when reprojecting 3D points to the 2D pixel space. This leaves a critical gap in methodology: there is no established paradigm for leveraging only multi-view videos to track points in pixel space through dynamic scenes. To address these challenges, we present MV-TAP ( T racking A ny P oint in M ulti-view V ideos), which builds a holistic understanding of dynamic scenes in multi-view videos. Our framework leverages camera encoding and a cross-view attention mechanism to effectively aggregate information across all views and timesteps. Furthermore, to facilitate research in this direction, we construct a large-scale synthetic dataset specifically designed for training multi-view point-tracking models and propose an evaluation benchmark to assess generalization and robustness across diverse multi-scene settings. Experiments on multi-view benchmarks demonstrate that our approach significantly improves upon existing tracking methods, yielding more complete and accurate predictions. Our code and dataset will be made publicly available. Our contributions are summarized as follows: • We define, for the first time, the task of multi-view point tracking in pixel space for establishing robust spatio-temporal correspondences in multi-view dynamic videos. • We propose MV-TAP, a framework that leverages camera geometry and cross-view information to address inherent limitations of single-view point tracking, such as occlusion and motion ambiguity. • We demonstrate through extensive experiments that our method achieves competitive performance, providing an effective baseline for this new task. Figure 3 : Overall architecture of MV-TAP. Given synchronized multi-view videos, per-view correlation volumes are extracted from a CNN encoder feature for each query point. These correlations are then tokenized and added with camera embedding for relative geometric context across views and temporal embedding. Trajectories and occlusion states are iteratively updated by a Transformer architecture, comprising temporal, spatial, and view attention modules. 2 Related Work Point tracking. The goal of point tracking is to predict the trajectories and visibilities of given query points within a video. Recent methods 14 ; 9 ; 44 ; 25 ; 6 ; 21 ; 20 ; 15 ; 24 ; 48 focus on estimating tracks by finding correspondences across frames while capturing temporal context. Specifically, PIPs 14 proposes iterative refinement on initial query points using local correlation volumes. TAPIR 9 combines this idea with a global correlation approach, which was introduced by TAP-Net 25 . LocoTrack 6 further introduces bidirectional local 4D correlation volumes for enhanced local correspondence. Notably, CoTracker 21 ; 20 learns the correlation between tracking points by interleaving frame- and track-wise attention modules. This approach achieves superior performance by explicitly learning the correlations between tracking points as well as temporal consistency. Unlike previous approaches, SpatialTracker 44 performs 3D point tracking by combining 2D tracker with depth estimation. TAPNext 48 reframes the point tracking problem as an autoregressive next-token prediction, enabling causal online tracking without specialized inductive biases. Despite these advances, existing methods are primarily designed for monocular videos, and their application to multi-view scenarios remains underexplored. The most relevant work to ours is MVTracker 31 , which tracks query points across multi-view synchronized videos in 3D space. However, this approach highly relies on the quality of precomputed depth for lifting to 3D space. Unlike MVTracker, MV-TAP performs directly in the 2D pixel space with a view-wise attention module, showing improved robustness. Multi-view point matching. Multi-view point matching aims to identify correspondences for a set of query points across images captured