TrackMAE: Video Representation Learning via Track Mask and Predict

Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem · 2026

Read TrackMAE: Video Representation Learning via Track Mask and Predict by Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27268.txt.

Introduction

Vandeghen and co-authors present TrackMAE, a masked video modeling method that trains on motion. Standard masked video models hide motion only implicitly in pixel reconstruction. TrackMAE instead tracks sparse points with an off-the-shelf tracker to build trajectories. Those trajectories become an explicit reconstruction target for the model. They also guide a motion-aware masking strategy that replaces random tube masks. Training covers both pixel and feature reconstruction spaces. Across six datasets, TrackMAE beat video self-supervised baselines. Its representations proved more discriminative and more generalizable.

Catalog record

CREATORRenaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TrackMAE: Video Representation Learning via Track Mask and Predict Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem Abstract Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve the random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction space by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms the state-of-the-art video SSL baselines, therefore learning more discriminative and generalizable representations. Code available at https://github.com/rvandeghen/TrackMAE Figure 1 : TrackMAE improves masked video modeling by jointly predicting spatial features and motion trajectories in a mask-and-predict fashion. 1 Introduction Self-supervised learning (SSL) has become the default pretraining recipe, allowing models to learn diverse yet powerful representations for different modalities, including text ( Devlin et al. 2019 ; Brown et al. 2020 ) image ( Chen et al. 2020 ; He et al. 2020 ; Oquab et al. 2024 ; Caron et al. 2018 ; Caron et al. 2020 ; Caron et al. 2021 ; Bao et al. 2022 ; Zhou et al. 2022 ; He et al. 2022 ) and video ( Tong et al. 2022 ; Wang et al. 2023b ; Huang et al. 2023 ) replacing manual labels with pretext objectives that exploit the structure of the raw data. Among video methods, masked video modeling (MVM) stands out for its simplicity and scalability: a high fraction (often 80–95%) of spatiotemporal tokens is hidden, and a vision transformer is trained to reconstruct them from the visible context ( Tong et al. 2022 ; Sun et al. 2023 ; Huang et al. 2023 ; Fan et al. 2023 ) . In practice, videos are decomposed into tubelets ( e.g . , 2{\times}16{\times}16 ) and only visible tokens are encoded, which makes computation efficient and enables training on large corpora. However, the most common instantiation of MVM is pixel reconstruction with random tube masking, which tends to emphasize low-level appearance statistics (color/texture continuity, local smoothness) and under-utilize temporal structure. Real-world videos also exhibit strong temporal redundancy and sparse foreground motion, so masked pixel reconstruction can often be solved via spatial correlations or short-horizon consistency without learning fine-grained dynamics. This mismatch is observed in the downstream performance of such models, as shown in ( Thoker et al. 2022 ; Thoker et al. 2025a ) . They perform well on appearance-dominated datasets like Kinetics-400 ( Kay et al. 2017 ) or UCF101 ( Soomro et al. 2012 ) but lag on motion-centric tasks such as Something-Something V2 ( Goyal et al. 2017 ) and FineGym ( Shao et al. 2020 ) , where accurate temporal modeling is crucial. Recent efforts inject more structure into MVM along two orthogonal axes. One line alters where to learn by biasing masks toward informative regions, e.g . , selecting high-motion tokens via flow ( Huang et al. 2023 ) or motion vectors ( Fan et al. 2023 ) , or adaptively sampling tokens with learned heuristics ( Bandara et al. 2023 ) . Another line changes what to learn by replacing pixels with feature-space targets ( e.g . , HOG, DINO, CLIP), which mitigates pixel-level shortcuts and encourages learning of high-level semantics like object/part structure, attributes, and object–scene relations ( Sun et al. 2023 ; Thoker et al. 2025b ) . While both directions help, they supervise motion only implicitly . The model is never asked to predict how things move or to maintain identity over time. As a result, improvements in motion sensitivity are limited, especially under high masking ratios where temporal cues must be inferred from few visible tokens. We suggest that temporal correspondence should be a first-hand signal during pretraining, complementing pixel/feature targets rather than competing with them. In this work, we propose to directly use motion as a training signal by complementing the spatial reconstruction with a motion prediction target. By leveraging the motion prediction produced by a point tracker, our method TrackMAE ( Fig. 1 ) aims to jointly reconstruct both the spatial and motion targets. In particular, we use CoTracker3 ( Karaev et al. 2025 ) , a robust and high-quality point tracker, to extract motion trajectories. TrackMAE adds two components to masked video modeling: (i) a lightweight trajectory decoder that predicts point-track displacements, and (ii) motion-aware masking that preferentially samples visible tokens from both high- and low-motion regions. Finally, we show that our motion target complements both pixel- and feature-based spatial targets, leading to on par with or state-of-the-art results on several benchmarks. Our contributions: 1. We propose TrackMAE, a new masked video pretraining scheme with explicit motion awareness based on prediction of tracked point trajectories. 2. We improve the random tube masking strategy with a motion-aware masking that equally samples visible tokens from high- and low-motion regions. 3. Extensive evaluation on multiple datasets demonstrating state-of-the-art performance on motion-centric benchmarks as well as strong generalization capabilities. 2 Related work Masked video modeling. Masked modeling has emerged as a powerful learning paradigm for representation learning for both visual and non-visual modalities like text ( Brown et al. 2020 ; Devlin et al. 2019 ) , audio ( Huang et al. 2022 ; Chong et al. 2023 ; Gong et al. 2022 ) , images ( Bao et al. 2022 ; Xie et al. 2022 ; He et al. 2022 ; Zhou et al. 2022 ; Assran et al. 2023 ; Gupta et al. 2023 ; Eymaël et al. 2024 ) , and video ( Tong et al. 2022 ; Huang et al. 2023 ; Fan et al. 2023 ; Sun et al. 2023 ; Wang et al. 2023a ; Feichtenhofer et al. 2022 ) . The goal is to hide a portion of the input from the model and aim to predict the hidden part from the visible context only. In vanilla masked video modeling ( Tong et al. 2022 ) , a considerable portion of the input pixels are typically kept hidden at random, and the model is trained to predict those hidden pixels from the visible portion, thereby encoding useful video representations. Many follow-up works improve this mask-and-predict task by either leveraging what to mask ( Qing et al. 2024 ; Fan et al. 2023 ; Huang et al. 2023 ; Bandara et al. 2023 ) or what to predict ( Li et al. 2023 ; Sun et al. 2023 ; Wei et al. 2022 ; Yang et al. 2024 ; Thoker et al. 2025b ; Salehi et al. 2024 ) . In the masking paradigm, MAR ( Qing et al. 2024 ) designed a new masking strategy tailored for action recognition, based on the running cell masking. MGM

Read the complete text