Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection
Ayush K. Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan F. Smeaton, Noel E. O’Connor · 2025
Read Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection by Ayush K. Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan F. Smeaton, Noel E. O’Connor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08561.txt.
Introduction
Ayush Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan Smeaton, and Noel O'Connor improve how video models learn by hiding parts of a clip. Their paper is from two thousand twenty-five. In this training style, the model must rebuild the hidden parts from what it can see. Which parts get hidden matters a lot, and fixed hiding rules can waste the model's effort.
Their sampler learns to hide the parts with the most motion, where the action is. A reinforcement learning method trains the sampler and the main model together from scratch. This allows very heavy hiding without hurting later performance on action recognition. Tests on four video benchmarks showed gains in accuracy and memory efficiency. The method transfers well across data sets.
Catalog record
| CREATOR | Ayush K. Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan F. Smeaton, Noel E. O’Connor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection Ayush K. Rai, Kyle Min, Tarun Krishna, Feiyan Hu, Alan F. Smeaton, Noel E. O’Connor Abstract Masked video modeling (MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable T rajectory-Aware A daptive T oken S ampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods. * * footnotetext: Equal contribution Figure 1: A depicts our overall architecture with MAE ( f_{\phi} ) and TATS ( g_{\theta} ). B illustrates the joint training (Algorithm 1 ) of f_{\phi} and g_{\theta} using PPO. Until epoch m_{o} , standard random space-time masking is applied. Afterward, every k steps, Phase 1 ( g_{\theta} frozen, f_{\phi} unfrozen) stores old state of g_{\theta} in memory buffer \mathcal{M}_{b} as episodes, followed by Phase 2 ( g_{\theta} unfrozen, f_{\phi} frozen), where g_{\theta} is optimized via \mathcal{L}_{s}(\theta) . The optimization process then alternates between Phase 1 and Phase 2. 1 Introduction Self-supervised video representation learning has recently emerged as a prominent area of research due to the generalization capabilities of the learned embeddings. Such representations can be applied to several downstream tasks such as action recognition [ 60 , 22 ] , object detection [ 2 ] , and segmentation [ 3 ] in videos. Due to the scarcity of labeled data, a standard approach in self-supervised learning (SSL) methods for video understanding involves defining a pretext task. A pretext task can be interpreted as a self-supervised pseudo-objective for pre-training a model. Intuitively, if a model learns to solve a complex task that requires a high-level understanding of its input, then the features learned as a result should generalize well to other tasks. Inspired by BERT [ 30 ] in language modeling, masked modeling in the form of masked autoencoders (MAE) was adopted for images [ 64 , 23 , 33 ] and for videos [ 14 , 52 , 59 ] as a self-supervised pretext task. Masked modeling involves masking a large fraction (between 75-95%) of the input data and then learning to reconstruct or predict the removed content based on the visible information. Although this concept is simple, it has been shown to improve performance [ 14 , 23 ] , generalization [ 14 , 23 ] , data efficiency [ 52 ] , memory efficiency [ 14 , 4 ] , scalability [ 59 , 23 ] , robustness [ 24 ] and to reduce overfitting [ 17 ] on downstream tasks. Several studies have explored different formulations of MAE, focusing on masking portions of input, features, augmenting the masked modeling objective [ 6 , 10 , 33 , 70 , 72 , 69 , 64 ] . However, less emphasis has been given to adaptive masking mechanisms that adaptively select space-time patches based on the input. The masking mechanism forms a crucial component of the family of MAE methods, as it is responsible for selecting which information is to be exploited by the encoder and predicted by the decoder. [ 23 , 70 ] explored random masking approaches for image patches, blocks, and grids. Though such approaches have shown promise and performance gains on downstream tasks, there still exists a research gap in terms of the masking mechanism being able to adapt to the input. With fixed masking mechanisms, MAEs are unable to exploit the expressivity of transformer-based encoders. In this direction, other contemporary works have investigated different masking strategies for images such as semantically guided masking [ 33 ] , uniform sampling for pyramid-based vision transformer (ViT) [ 34 ] and utilizing mask generators based on object priors [ 7 ] and learning easy-to-hard masking through curriculum learning [ 36 ] . The challenging aspect of MVM is the extra time dimension and high spatiotemporal inductive biases from adjacent frames carrying highly redundant information. This introduces potential information leakage as masked space-time patches can be trivially inferred from spatiotemporal neighborhoods, enabling learning of the shortcuts and less generalizable representations during pre-training. Hence, a substantial amount of compute and memory resources are inefficiently utilized in the prediction of uninformative tokens. On the contrary, such a high level of redundancy can be exploited to aggressively mask space-time tokens with a high mask ratio without compromising the prediction quality of the masked space-time patches. Several approaches in MVM have utilized frame, tube, and patch-based masking [ 14 , 52 , 59 ] , and there is no single universal masking strategy that works for all datasets. VideoMAE [ 52 ] achieves the best action classification results on Something-Something v2 (SSv2) [ 20 ] with random tube masking while STMAE [ 14 ] achieves its best performance on Kinetics-400 (K400) [ 29 ] with random space-time patch masking. An explanation for this observation is that not all space-time tokens carry meaningful information, and a fixed masking strategy steers the model’s optimization towards a particular task. Thus, it is crucial to incorporate adaptive computation in MAEs to dynamically select informative tokens based on the given input and the mask ratio. Previous works such as MGMAE [ 25 ] proposed motion-guided masking by extracting the optical flow from pre-trained models, and AdaMAE [ 4 ] introduced a token sampler module to select high-activity regions using REINFORCE [ 67 ] . In order to exploit unequal information density among patches, we introduce TATS module that learns a video-specific masking strategy from scratch to select space-time patches based on their spatio-temporal motion and trajectory information using Trajectory Attention (TA) [ 42 ] . TATS does not rely on any computationally expensive dense optical flow features or semantic cues obtained from external pretrained models like RAFT [ 51 ] , CLIP [ 44 ] , or DINOv2 [ 40 ] . TATS can be interpreted as a policy agent that models a categorical distribution over the set of input space-time tokens by leveraging their trajectory information and then samples the most relevant tokens based on a predefined mask ratio. However, since training MAE in conjunction with TATS is unstable due to the non-differentiability of the sampling operation, we additionally propose a unified training recipe to train MAE and TATS modules simultaneously using PPO [ 46 ] method used in reinforcement learning (RL). Our goal is