Generative Point Tracking with Flow Matching

Mattie Tesfaldet, Adam W. Harley, Konstantinos G. Derpanis, Derek Nowrouzezahrai, Christopher Pal · 2025

Read Generative Point Tracking with Flow Matching by Mattie Tesfaldet, Adam W. Harley, Konstantinos G. Derpanis, Derek Nowrouzezahrai, Christopher Pal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20951.txt.

Introduction

Mattie Tesfaldet leads four co-authors on GenPT, a generative tracker that follows a point through video while keeping several plausible paths alive. Occlusion and appearance change make tracking uncertain, yet standard trackers output one average path and lose the alternatives.

Trained with a flow matching formulation that refines trajectories step by step, GenPT models multi modal futures instead of collapsing to a mean. Through occlusions it can hold competing hypotheses and settle as evidence returns. Generative tracking suits editing, sports analysis, and robotics, where a wrong single guess is worse than a short list of honest options.

Catalog record

CREATORMattie Tesfaldet, Adam W. Harley, Konstantinos G. Derpanis, Derek Nowrouzezahrai, Christopher Pal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20951

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Point Tracking with Flow Matching Mattie Tesfaldet, Adam W. Harley, Konstantinos G. Derpanis, Derek Nowrouzezahrai, Christopher Pal Abstract Tracking a point through a video can be a challenging task due to uncertainty arising from visual obfuscations, such as appearance changes and occlusions. Although current state-of-the-art discriminative models excel in regressing long-term point trajectory estimates—even through occlusions—they are limited to regressing to a mean (or mode) in the presence of uncertainty, and fail to capture multi-modality. To overcome this limitation, we introduce Generative Point Tracker (GenPT), a generative framework for modelling multi-modal trajectories. GenPT is trained with a novel flow matching formulation that combines the iterative refinement of discriminative trackers, a window-dependent prior for cross-window consistency, and a variance schedule tuned specifically for point coordinates. We show how our model’s generative capabilities can be leveraged to improve point trajectory estimates by utilizing a best-first search strategy on generated samples during inference, guided by the model’s own confidence of its predictions. Empirically, we evaluate GenPT against the current state of the art on the standard PointOdyssey, Dynamic Replica, and TAP-Vid benchmarks. Further, we introduce a TAP-Vid variant with additional occlusions to assess occluded point tracking performance and highlight our model’s ability to capture multi-modality. GenPT is capable of capturing the multi-modality in point trajectories, which translates to state-of-the-art tracking accuracy on occluded points, while maintaining competitive tracking accuracy on visible points compared to extant discriminative point trackers. 1 Introduction Point tracking has emerged as a popular and important approach for motion analysis in computer vision ( Sand & Teller 2008 ; Harley et al. 2022 ; Cho et al. 2024 ; Karaev et al. 2024b ; Karaev et al. 2024a ) . Sitting between optical flow estimation ( Luo et al. 2024 ; Saxena et al. 2024 ; Teed & Deng 2020 ) and feature tracking ( Qin et al. 2024 ; Qin et al. 2023 ) , point tracking aims to estimate the long-range trajectories of a set of points across a video. Its utility is integral to various downstream tasks, such as structure-from-motion ( He et al. 2024 ; Wang et al. 2021 ) , video stabilization ( Yu & Ramamoorthi 2020 ; Liu et al. 2014 ) , and 4D reconstruction ( Jiang et al. 2025 ) . Figure 1: Multi-modal prediction of point trajectories . (Top) 100 randomly sampled trajectories of a single tracked point from our model, Generative Point Tracker (GenPT). The first image shows the full path of each sampled trajectory. Our model captures the multi-modality present in uncertain regions of the video. When tracking uncertainty is high, the majority of predictions are clustered around regions where the point is most likely to be. Occasionally, spurious (but valid) predictions can occur due to feature similarity elsewhere in the frame. Video is from the TAP-Vid RGB-S dataset ( Doersch et al. 2022 ) . (Middle) Ground truth for top row. (Bottom) Point trajectory estimates can be improved by utilizing a best-first search strategy on generated samples during inference, guided by the model’s own confidence of its predictions in each window of observation. Current state-of-the-art point trackers are based on a discriminative, regression-based approach that aims to improve tracking accuracy. However, this paradigm is limited to regressing to a single mean or mode, and fails to capture the multi-modality and uncertainty inherent to point tracking, particularly during occlusions or appearance changes. This ill-posed nature of the problem motivates a generative approach capable of producing a variety of plausible solutions. To address these limitations, we introduce GenPT , a generative point tracker. Our primary contributions are: 1. We introduce GenPT, the first generative point tracker trained in a modified flow matching setup ( Tong et al. 2024 ; Lipman et al. 2023 ) to capture the uncertainty inherent in point tracking. 2. We introduce three changes to the vanilla flow matching setup that are crucial to obtaining a competitive model: we incorporate the benefits of the iterative refinement used during training and inference of recent discriminative point trackers ( Harley et al. 2022 ; Karaev et al. 2024a ) ; we use a window-dependent prior to link samples across windows; and we use a specialized variance schedule in the conditional probability path that is tuned for point coordinates. 3. We show how our model’s generative capabilities can be leveraged to improve point trajectory estimates by utilizing a best-first search strategy on generated samples during inference, guided by the model’s own confidence of its predictions. We also provide architectural improvements that make our model more efficient than the state of the art, using fewer parameters and achieving faster inference, detailed in the appendix. Finally, to demonstrate the efficacy of generative tracking, we introduce a simple benchmark designed to evaluate occluded point tracking accuracy. 2 Related work Point tracking is the task of estimating the motion of a set of point-wise features across a long temporal window. It lies between the short-range density of optical flow and the long-range sparsity of feature tracking. Sand & Teller 2008 introduced this problem through the particle video representation, and Harley et al. 2022 later adapted it to deep learning with their PIPs model, which leveraged RAFT’s iterative refinement guided by dense feature correlation maps ( Teed & Deng 2020 ) . Recent models have built upon this foundation, introducing innovations like spatial attention between trajectories in CoTracker ( Karaev et al. 2024b ) to improve tracking through occlusions. LocoTrack ( Cho et al. 2024 ) proposed an efficient and lightweight model alongside an improved approach to producing correlations by exchanging 2D correlation maps with 4D correlation volumes, resulting in improved tracking. CoTracker3 ( Karaev et al. 2024a ) improved upon CoTracker by using a simplified version of the 4D correlations from LocoTrack while simplifying other components of the CoTracker model, resulting in one of the current state-of-the-art point trackers and our main competing baseline. A key limitation shared by these models, however, is their discriminative nature; they can only regress to a single mean or mode, failing to capture the multi-modality and uncertainty inherent in point tracking. In this work, we integrate the key mechanisms that drive current state-of-the-art discriminative point trackers ( e.g . iterative refinement guided by dense feature correlations) into a generative model, maintaining high tracking performance while enabling the modelling of multi-modality in point trajectories. Diffusion-based models ( Tong et al. 2024 ; Lipman et al. 2023 ; Luo 2022 ; Song et al. 2021 ; Song & Ermon 2019 ) have become a popular approach to generative modeling, achieving strong performance across various computer vision tasks ( Esser et al. 2024 ; Kim et al. 2024 ; Karras et al. 2022 ; Song et al. 2021 ; Song & Ermon 2019 ; Shafir et al. 2024 ; Tan et al. 2022 ; Liu et al. 2023 ; Zeng et al. 2022 ) . Most relevant to our work are recent approaches that formulate optical flow within a diffusion-based framework ( Saxena et al. 2024 ; Luo et al. 2024 ) , where their ability to capture multi-modality in ill-posed cases—such as occlusions—has shown to be effective. As point tracking faces similar challenges, this motivates a generative approach to

Read the complete text