Multi-View 3D Point Tracking
Frano Rajič, Haofei Xu, Marko Mihajlovic, Siyuan Li, Irem Demir, Emircan Gündoğdu Lei Ke Sergey Prokudin Marc Pollefeys Siyu Tang, [0.75ex] ETH Zürich Carnegie Mellon University Balgrist University Hospital Microsoft · 2025
Read Multi-View 3D Point Tracking by Frano Rajič, Haofei Xu, Marko Mihajlovic, Siyuan Li, Irem Demir, Emircan Gündoğdu Lei Ke Sergey Prokudin Marc Pollefeys Siyu Tang, [0.75ex] ETH Zürich Carnegie Mellon University Balgrist University Hospital Microsoft (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21060.txt.
Introduction
Frano Rajič and colleagues tracked points through space and time from several cameras at once in a 2025 paper. Single-camera trackers lose points behind occluders and cannot judge depth, while older multi-camera systems demanded dozens of rigs and per-clip optimization. Their feed-forward tracker takes a practical handful of views with known poses and depth, merges their features into one point cloud, and updates point positions with nearest-neighbor correlation plus a transformer step.
Trained on thousands of synthetic multi-view clips, it reached centimeter-level median errors on two real benchmarks covering studios and hand-object scenes. It handles anywhere from one to eight views and clips of varied length without retraining. Robotics and motion capture gain an online three-dimensional tracker.
Catalog record
| CREATOR | Frano Rajič, Haofei Xu, Marko Mihajlovic, Siyuan Li, Irem Demir, Emircan Gündoğdu Lei Ke Sergey Prokudin Marc Pollefeys Siyu Tang, [0.75ex] ETH Zürich Carnegie Mellon University Balgrist University Hospital Microsoft |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-View 3D Point Tracking Frano Rajič, Haofei Xu, Marko Mihajlovic, Siyuan Li, Irem Demir, Emircan Gündoğdu Lei Ke Sergey Prokudin Marc Pollefeys Siyu Tang, [0.75ex] ETH Zürich Carnegie Mellon University Balgrist University Hospital Microsoft Abstract We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras ( e.g . , four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks—Panoptic Studio and DexYCB—achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1–8 views with varying vantage points and video lengths of 24–150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications. Project page: https://ethz-vlg.github.io/mvtracker . Figure 1 : We introduce MVTracker , the first data-driven multi-view 3D point tracker for tracking arbitrary 3D points across multiple cameras. Our method fuses multi-view features into a unified 3D feature point cloud, within which it leverages kNN-based correlation to capture spatiotemporal relationships across views. A transformer then iteratively refines the point tracks, handling occlusions and adapting to varying camera setups without per-sequence optimization. This figure shows results on SelfCap [ 40 ] using DUSt3R-based [ 37 ] depth. 1 Introduction Tracking arbitrary points in 3D [ 30 ] is a fundamental problem in computer vision, with numerous applications in dynamic scene reconstruction [ 14 ] , robotics [ 10 ] , and augmented reality [ 25 ] . While remarkable progress has been made with the recent advancement of 2D point tracking methods [ 17 , 16 , 12 , 7 , 8 , 9 , 5 , 45 ] , they remain inherently limited in modeling 3D-consistent motion due to the fundamental ambiguity of the 3D-to-2D projection process. Thus, high-quality 3D point tracking remains a challenging task. Scene flow approaches [ 30 , 26 , 31 , 28 ] can estimate a dense 3D flow field for every 3D point, but they are usually limited to two consecutive video frames, whereas 3D point tracking operates on long sequences ( e.g . , tens or even hundreds of video frames). Recently, several methods [ 38 , 18 , 24 ] have been proposed to tackle the 3D point tracking task from monocular videos. However, their performance remains far from satisfactory for real-world applications that demand high quality and robustness, due to the difficulty of estimating 3D from single viewpoints in challenging scenarios such as occlusion and complex motion. To address these challenges, we adopt a multi-camera setup and develop the first feed-forward model that can efficiently and robustly predict long-range 3D point trajectories from multi-view videos. Unlike previous multi-camera methods [ 30 , 23 , 44 ] that require more than 20 cameras and tedious per-sequence optimization, our approach enables feed-forward 3D point tracking with a practical and flexible number of cameras, such as a set of four camera views with arbitrary viewpoints. Thus, our method strikes a promising balance between accuracy and practicality, making it particularly suitable for real-world applications. Key to our model is a dynamic fused 3D feature point cloud, constructed by combining the unprojected per-view depth maps, which not only effectively aggregates multi-view information to a global scene representation, but also facilitates reliable and efficient correspondence search with the k-nearest-neighbors (kNN) operation. This is different from the triplane representation used in the previous state-of-the-art method SpatialTracker [ 38 ] , which inevitably suffers from information loss during the triplane splatting process and is therefore less effective in processing varying numbers of input cameras. Moreover, our model performs reasonably well with different sources of depth input, whether from accurate depth sensors or noisy estimates from methods such as DUSt3R [ 37 ] and VGGT [ 33 ] . Using the fused dynamic point cloud, we retrieve local neighbors for each tracked point using a kNN search and compute multi-scale correlation features that capture both appearance similarity and 3D offset information. A spatiotemporal transformer then iteratively refines each point’s 3D position and appearance over a sliding temporal window. Finally, the outputs from overlapping windows are merged to produce globally consistent 3D point trajectories. To train our model, we construct a synthetic multi-view dataset using Kubric [ 11 ] and simulate 5K sequences. We evaluate the model on two real-world datasets, DexYCB [ 3 ] and Panoptic Studio [ 15 ] , for which we construct 3D point trajectories by leveraging ground-truth object and hand pose estimation labels in DexYCB and merging existing monocular trajectory labels [ 18 ] in Panoptic Studio. We conduct extensive experiments on these multi-view video datasets, which suggest that our model outperforms baseline methods by a significant margin. Our model also performs well with different camera setups, different numbers of cameras, and different sources of depth maps, indicating the robustness of our proposed approach. We release our source code and models alongside the training and evaluation datasets to facilitate future research on multi-view 3D point tracking. 2 Related Work Scene Flow. Scene flow methods [ 30 , 26 , 31 , 28 ] are usually designed to estimate the dense 3D motion between two consecutive video frames. Traditional methods [ 30 , 26 , 31 ] try to solve this task with an optimization framework, which is typically slow and requires many cameras ( e.g . , 31 in [ 30 ] ) to well-regularize the optimization process. Modern approaches [ 41 , 28 , 21 ] explore data-driven models to directly predict scene flow in a feed-forward manner. Despite recent progress, existing models are limited to two frames and are not able to track long-range correspondences in 3D. 2D Point Tracking. Recent years have witnessed significant progress in long-term 2D point tracking [ 7 , 12 , 45 , 34 , 8 , 16 , 9 , 17 , 5 , 13 ] . For instance, CoTracker2 [ 16 ] leverages self-attention to aggregate spatial context from supporting tracks, while LocoTrack [ 5 ] extends traditional 2D correlation into bidirectional 4D correlation for more robust local matching. These methods typically estimate long-range point trajectories over an entire video, handling occlusions effectively. In this work, we investigate their extension to 3D, but with a sparse multi-view setup and known camera parameters, rather than monocular input. Moreover, unlike recent trackers [ 17 , 9 ] that reduce the synthetic-to-real domain gap via self-supervision or pseudolabeling, our approach focuses on direct supervision from synthetic data. 3D Point Tracking. 3D point tracking has seen innovative contributions from recent methods [ 42 , 23 , 38 , 24 , 39 , 43 ,