TAPNext++: What’s Next for Tracking Any Point (TAP)?

Sebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch, Ross Goroshin, David Joseph Tan, Sarath Chandar, Rudolph Triebel, Federico Tombari, [-0.5ex] Mila - Quebec AI Institute Université de Montréal Chandar Research Lab, [-0.5ex] Polytechnique Montréal Canada CIFAR AI Chair Technical University Munich (TUM) · 2026

Read TAPNext++: What’s Next for Tracking Any Point (TAP)? by Sebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch, Ross Goroshin, David Joseph Tan, Sarath Chandar, Rudolph Triebel, Federico Tombari, [-0.5ex] Mila - Quebec AI Institute Université de Montréal Chandar Research Lab, [-0.5ex] Polytechnique Montréal Canada CIFAR AI Chair Technical University Munich (TUM) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10582.txt.

Introduction

Jung and eight colleagues stretch point tracking to far longer videos. Old TAPNext fades on long clips and misses points that return after hiding. Their TAPNext++ trains on 1024 frame clips with sequence parallel methods. A new re detection score grades points that reappear after gaps. Tailored shape tricks plus watched hidden points lift re detection to new best marks.

Catalog record

CREATORSebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch, Ross Goroshin, David Joseph Tan, Sarath Chandar, Rudolph Triebel, Federico Tombari, [-0.5ex] Mila - Quebec AI Institute Université de Montréal Chandar Research Lab, [-0.5ex] Polytechnique Montréal Canada CIFAR AI Chair Technical University Munich (TUM)
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50095
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10582

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TAPNext++: What’s Next for Tracking Any Point (TAP)? Sebastian Jung, Artem Zholus, Martin Sundermeyer, Carl Doersch, Ross Goroshin, David Joseph Tan, Sarath Chandar, Rudolph Triebel, Federico Tombari, [-0.5ex] Mila - Quebec AI Institute Université de Montréal Chandar Research Lab, [-0.5ex] Polytechnique Montréal Canada CIFAR AI Chair Technical University Munich (TUM) Abstract Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes an end-to-end, recurrent transformer architecture to track points frame-by-frame in a purely online fashion – demonstrating competitive performance at minimal latency. However, we show that TAPNext struggles with longer video sequences and also frequently fails to re-detect query points that reappear after being occluded or leaving the frame. In this work, we present TAPNext++, a model that tracks points in sequences that are orders of magnitude longer while preserving the low memory and compute footprint of the architecture. We train the recurrent video transformer using several data-driven solutions, including training on long 1024-frame sequences enabled by sequence parallelism techniques. We highlight that re-detection performance is a blind spot in the current literature and introduce a new metric, Re-Detection Average Jaccard ( AJ RD \text{AJ}_{\mathrm{RD}} ), to explicitly evaluate tracking on re-appearing points. To improve re-detection of points, we introduce tailored geometric augmentations, such as periodic roll that simulates point re-entries, and supervising occluded points. We demonstrate that recurrent transformers can be substantially improved for point tracking and set a new state-of-the-art on multiple benchmarks. Model and code can be found at https://tap-next-plus-plus.github.io . Figure 1 : Overview. TAPNext++ sets a new state-of-the-art for online point tracking by simultaneously improving accuracy, speed, and long-term robustness. Left: Our method reaches the best speed-accuracy trade-off, achieving higher mean \delta^{avg} than competing online methods while being substantially faster. Middle: Despite running frame-by-frame without explicit memory, TAPNext++ outperforms previous methods on long-sequence benchmarks. Right: TAPNext++ also demonstrates superior robustness to occlusion and re-appearences, achieving state-of-the-art \text{AJ}_{\mathrm{RD}} scores, a new metric evaluating point tracking after re-detection, on both RoboTAP and PointOdyssey. * * footnotetext: Equal contribution. † † footnotetext: Work done during an internship at Google. 1 Introduction Tracking Any Point (TAP) in a video is a fundamental computer vision problem with many downstream applications in robotics, augmented reality, video editing, and 3D/4D reconstruction. Depending on the application, point tracking is performed online or offline, in 2D or 3D and under varying computational constraints. In this work, we focus on highly efficient, frame-by-frame 2D point tracking which is very relevant for AR applications such as anchoring digital content in the real world. In contrast to optical flow estimation, which predicts dense pixel-level correspondences between adjacent frames, TAP methods attempt to consistently track points over multiple frames under strong appearance changes and occlusions. However, despite recent progress 30 ; 2 ; 13 , state-of-the-art methods struggle to achieve robust performance in long sequences as shown in PointOdyssey 29 . Common strategies to counteract this issue are: (1) tracking methods that operate over a sliding window of frames instead of one frame at a time 13 ; 12 , (2) hard coding an explicit memory representation of past frames 1 ; 2 , (3) re-detecting points with appearance-based foundation models 2 and (4) keeping the first frame as a persistent input at every time step 12 . However, as our experiments demonstrate, these remedies fail to solve the root issue of temporally deteriorating representations and typically increase runtime and memory usage. Figure 2 : Qualitative Comparison on Challenging Re-entry Scenarios. We compare TAPNext++ with prior methods on two sequences featuring point re-entries at varying locations. Left: An object leaves the frame on the right and re-enters from the left. BootsTAPNext and Track-On fail to re-detect the object upon re-entry. In contrast, CoTracker3 and TAPNext++ successfully relocate the points. However, CoTracker3 fails to track static points on the texture-less table, which TAPNext++ tracks correctly. The variant TAPNext++ without Roll also struggles with immediate re-detection, highlighting the importance of roll augmentation for re-entry robustness. Right: Three objects are juggled, repeatedly leaving and re-entering the frame. Point trajectories for the last 8 frames are visualized as traces for clarity. TAPNext++ with roll is the only method that robustly tracks all objects through these challenging dynamics, all competing methods fail. TAPNext 30 reformulates point tracking as next token prediction and deploys linear recurrent SSM layers 22 to track scene dynamics, particularly of query points, over time. Its constant-sized, linear-recurrent memory allows TAPNext to jointly track 1024 points at 348 FPS on an H100 GPU, demonstrating the efficiency required for mobile applications in robotics and augmented reality. However, as also noted in concurrent work 2 , the BootsTAPNext model trained on a large number of short videos at a fixed 256\times 256 resolution 7 fails to generalize to longer sequences 29 . In this work, we present TAPNext++, a model demonstrating that these tracking failures are not inherent to the underlying recurrent video transformer architecture and can be effectively tackled through improved training strategies and datasets. Tracking points over long time horizons reveals another severe limitation of current TAP methods: when points reappear after being occluded or leaving the view, their re-detection frequently fails. This limits applications, e.g. in AR/VR, where both the camera and the scene are dynamic and reappearances are a frequent occurrence. Recent approaches resort to hard-coded appearance-based re-detection without relying on motion priors required to disambiguate, e.g. , multiple instances of visually similar points. Some approaches 13 track occluded points explicitly within the image, which improves re-detection after short-term occlusions, but usually fail for long-term occlusions, particularly when tracked points venture outside the image boundaries. Other methods 27 attempt to build an explicit 3D map using monocular video depth foundation models 18 , assuming a mostly static scene and generous computational resources. We find that the re-detection limitation is a blind spot in current point tracking metrics and evaluations. The closest related metric is the survival rate 29 , which measures the average number of frames until point tracking failure. However, the survival rate does not directly measure re-detection performance because it includes many points that are always visible. Therefore, we propose a variation of the Average Jaccard metric specifically targeting the performance of point tracking after reappearance and evaluate recent approaches on it. Our main contributions are as follows: • We introduce methods that allow scaling the training of TAPNext architectures to long sequences and multiple resolutions – resulting in a lightweight, fully online model that sets a new state-of-the-art on multiple point tracking benchmarks. • We introduce Kubric-1024, which extends the original synthetic Kubric dataset 11 to 1024 frames and analyze the effects of long-sequence training on memory. • We propose novel re-detection metrics that directly measure the accuracy of point tracks after re-appearance and improve on these metrics through tailored geometric augmentations. • We demonstrate how our contributions effectively enhance the memory of the TAPNext model, outperforming all existing methods in long-context and re-detection scenarios. We highlight that we do not require any new model architectures to surpass existing methods while further enhancing the compute and memory efficiency of the original TAPNext. 2 Related Work Point Tracking Tracking

Read the complete text