Is Temporal Prompting All We Need For Limited Labeled Action Recognition?
Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin · 2025
Read Is Temporal Prompting All We Need For Limited Labeled Action Recognition? by Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01890.txt.
Introduction
Shreyank Gowda and co-authors ask in 2025 whether a few well placed prompts are all a video model needs to learn actions from almost no labels. Video recognition usually eats giant labeled sets. Vision language models pretrained on images and text generalize well, but bending them toward video is heavy work, and time, the essence of an action, is exactly what image models ignore.
Their adaptation, TP-CLIP, leaves the core model untouched and instead adds temporal visual prompts, small learned signals that teach the frozen network to read change across frames. The recipe keeps the original's broad generalization while costing a fraction of the compute and tunable parameters of rival adaptations. Across data sets it won in both zero shot and few shot tests, sometimes by wide margins. The title's question leans toward yes: for action recognition on a label budget, prompt the clock, not the machine.
Catalog record
| CREATOR | Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01890
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is Temporal Prompting All We Need For Limited Labeled Action Recognition? Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin Abstract Video understanding has shown remarkable improvements in recent years, largely dependent on the availability of large scaled labeled datasets. Recent advancements in visual-language models, especially based on contrastive pretraining, have shown remarkable generalization in zero-shot tasks, helping to overcome this dependence on labeled datasets. Adaptations of such models for videos, typically involve modifying the architecture of vision-language models to cater to video data. However, this is not trivial, since such adaptations are mostly computationally intensive and struggle with temporal modeling. We present TP-CLIP, an adaptation of CLIP that leverages temporal visual prompting for temporal adaptation without modifying the core CLIP architecture. This preserves its generalization abilities. TP-CLIP efficiently integrates into the CLIP architecture, leveraging its pre-trained capabilities for video data. Extensive experiments across various datasets demonstrate its efficacy in zero-shot and few-shot learning, outperforming existing approaches with fewer parameters and computational efficiency. In particular, we use just 1/3 the GFLOPs and 1/28 the number of tuneable parameters in comparison to recent state-of-the-art and still outperform it by up to 15.8% depending on the task and dataset. 1 Introduction Video understanding has significantly evolved recently Gowda et al. 2021a ; Yang et al. 2020 ; Wang et al. 2021a ; Mazzia et al. 2022 , fueled by computer vision and machine learning breakthroughs. Central to this progress are large language models (LLMs) Kenton and Toutanova 2019 and visual-language models Radford et al. 2021 ; Li et al. 2022a ; Singh et al. 2022 ; Alayrac et al. 2022 , showcasing potential across image-language tasks. Models like CLIP Radford et al. 2021 and its video adaptations Rao et al. 2022 ; Wang et al. 2021b ; Wasim et al. 2023 have been pivotal in advancing this field. Historically, video understanding heavily relied on extensive labeled datasets Carreira and Zisserman 2017 ; Carreira et al. 2018 . However, this reliance shifted with the emergence of LLMs and visual-language models, showcasing strong zero-shot generalization capabilities, reducing the need for labeled data. Early efforts like VideoBERT Sun et al. 2019 employed self-supervised learning from video-text data, validating pre-training and fine-tuning effectiveness across multiple video tasks. This approach evolved with diverse architectures and strategies Rasheed et al. 2023 ; Wasim et al. 2023 ; Ju et al. 2022a . Figure 1 : Comparing CLIP-based models on UCF101: Our TP-CLIP (highlighted) excels with fewer parameters, surpassing recent SOTA models. Bubble size = model GFLOPS. Such advancements have been further accelerated by the emergence of LLMs, such as ChatGPT, by integrating extensive knowledge and underpinning contextual comprehension. This integration facilitates sophisticated multimodal models, adept at processing intricate interactions between visual and textual data, akin to human-like understanding. Recent research explores methodologies to enhance these models’ video understanding capabilities, leveraging their extensive training on multimodal data to handle diverse tasks effectively. Nevertheless, temporal modeling remains a key challenge in video understanding, especially for adaptations of image-language models seeking to leverage existing pre-trained capabilities. Various adaptations for video, like adding self-attention layers or employing dedicated video models, have been proposed Luo et al. 2022 ; Xu et al. 2021 ; Pan et al. 2022 ; Wasim et al. 2023 . Prompt learning has emerged as a recent approach Wasim et al. 2023 ; Ju et al. 2022b , yet it requires full fine-tuning, potentially compromising model generalization. Prompted by this and the surge in prompt learning research, we inquire: Can we maintain CLIP’s generalization more efficiently? Specifically, can temporal prompting alone suffice to adapt CLIP for videos? Introducing TP-CLIP, we propose a method that seamlessly integrates temporal visual prompting into the CLIP architecture, thereby maintaining its generalization prowess. This approach efficiently handles video data, mitigating the computational intensity and complexities associated with temporal modeling. In Figure 1 , TP-CLIP is compared with other CLIP-based video adaptations, showcasing significant savings in tunable parameters and GFLOPs, alongside accuracy improvements. This method upholds CLIP’s core strengths while expanding its capabilities to tackle video data. Across various datasets, our experiments demonstrate TP-CLIP’s effectiveness in zero-shot and few-shot learning, surpassing existing approaches in terms of parameter efficiency and computational effectiveness. 2 Related Work Action Recognition. Action recognition has evolved significantly, starting from early works that primarily focused on handcrafted features and traditional classifiers Dalal and Triggs 2005 ; Wang and Schmid 2013 ; Wang et al. 2013 ; Jain et al. 2013 . Progressively, the field shifted towards deep learning methods, with Convolutional Neural Networks, Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks becoming a staple for extracting spatial and temporal features from video frames Simonyan and Zisserman 2014 ; Tran et al. 2018 ; Wang et al. 2016 ; Carreira and Zisserman 2017 . With the advancements in Transformers Vaswani et al. 2017a , video-based transformer models have become increasingly popular Bertasius et al. 2021 ; Arnab et al. 2021 ; Liu et al. 2022 ; Li et al. 2022b . However, these models are extremely expensive to train and whilst efficient variants exist, there is much scope for improvement. Vision-Language Models. The domain of vision-language models represents a fascinating intersection of computer vision and natural language processing. Early works in this area primarily focused on image captioning Xu et al. 2015 , where the goal was to generate descriptive text for images. This evolved into more complex tasks like visual question answering (VQA), requiring the model to understand both visual content and language queries. Recent breakthroughs include multimodal transformers Radford et al. 2021 ; Li et al. 2022a ; Alayrac et al. 2022 ; Singh et al. 2022 , which integrate visual and textual inputs in a unified framework, enabling more sophisticated interactions between visual and textual data excelling in tasks ranging from zero-shot image classification to creative image generation based on textual descriptions. These advancements underscore the growing capability of AI models to understand and generate content across visual and linguistic domains. Whilst these models excel in generalization abilities, they are trained on millions and sometimes billions of data points to obtain their high level performance. This is not something feasible for small research labs and companies. Efficiently adapting these models to tasks with minimal parameter costs is an important research direction. Adapting Vision-Language Models for Video. Adapting vision-language models (VLMs) for video tasks is a dynamic research domain, merging visual and linguistic advancements for video analysis. ”Tem-Adapter” Chen et al. 2023a focuses on video question answering, emphasizing temporal dynamics and complex semantics through novel aligner components. LAVILA Zhao et al. 2023 repurposes Large Language Models for video-language representation learning, enhancing temporal synchronization and alignment diversity. CLIP4Clip Luo et al. 2022 tailors CLIP for video-language retrieval tasks by encoding frames for feature sequences. VideoCLIP Xu et al. 2021 utilizes contrastive pre-training to unify video and text understanding. Other methods include spatio-temporal adapters Pan et al. 2022 , cross-frame communication layers Ju et al. 2022b , dynamic dilated convolutions Gowda et al. 2024a and dedicated video encoder modules Ni et al. 2022 . ViFi-CLIP Rasheed et al. 2023 fine-tunes CLIP for enhanced temporal understanding in various video learning scenarios. While promising, these methods introduce computational overhead and require full fine-tuning, potentially