Video Understanding: Through A Temporal Lens NGUYEN THANH THONG Abstract The fundamental difference between an image and a video is that a video introduces a temporal dimension. Such dimension triggers our awareness of temporal relations present among video elements, such as object positions, frames, and events. For example, when an object changes position, we perceive its positional changes as a movement. When we watch a video, we experience motion in the flow of frames. Additionally, one can recall the way we connect the events in a movie, seeking to construct a coherent narrative. These examples demonstrate that a video is indicative of temporal relations among its elements, requiring any video-understanding model to possess an awareness of temporality. Thus, in this thesis we aim to address the following question: “ How to utilize the existence of temporal relations among video elements to advance video understanding? ” First, video understanding performance has been constrained by training data of limited scale and quality, a bottleneck caused by the labor-intensiveness and expensive cost of obtaining high-quality annotated video data. To overcome this, we introduce an automatic annotation approach that harnesses the large vision-language model’s capacity to form a temporal progression among video frames to generate language annotations for videos. Because such automatically generated annotations may introduce noise to training, we propose a contrastive learning objective with a subtractive angular margin to regularize a model against perfect similarity between representations of a video and its language description. Our approach is theoretically guaranteed based on the intuition that a video and its caption often convey overlapping but not identical information. Second, in resource-constrained scenarios, both manual and automatic approaches are unable to generate sufficient supervised training data for video understanding models. In these cases, fully updating model parameters can lead to overfitting and degraded performance. To address this issue, we employ a parameter-efficient training approach to finetune only a finite subset of modules, which we call adapters. We equip our adapters with recurrent computation layers to construct recurrent adapters that specifically adapt to modeling temporal relations among video frames. Furthermore, we integrate a partial video-language alignment objective into training, ensuring that our model effectively captures temporal semantics relevant to the task. Our approach demonstrates improved performance in multi-task scenarios with limited training data. Third, existing works focus on short videos with independent or simple events. Most of them seek to infer temporal relations among coarsely sampled video elements, such as visual patches and video frames. However, for long videos, coarsely sampled elements hardly fully capture video content, while densely sampled elements will result in excessive computational cost. To cope with this issue, we introduce a state space layer (SSL) to our video understanding model to efficiently model temporal relations among densely sampled video elements for long-term video understanding. Moreover, recognizing that there is a lack of benchmarks that authentically evaluate long-term video understanding, we introduce two new long-term video understanding benchmarks: one based on egocentric videos averaging 18 minutes and another based on movies spanning up to 2 hours. We carefully construct detailed prompts to instruct GPT-4 to generate highly difficult questions that require a model to simultaneously summarize, compare, and compress information across a long video. Experiments show that our method demonstrates superiority over self-attention and recurrent layers for long-term video understanding. Fourth, despite the existence of temporal relations among video elements, the attention to such relations still remains scarce compared to the attention to cross-modal relations between a video and its language description. To address this gap, we first propose a novel contrastive learning framework to drive a model to focus on relations between two motions in videos. Then, we extend our framework to capture relations between video moments. Extensive experiments indicate that our explicit temporal modeling approach significantly improves video understanding by making representations more aware of fine-grained temporal relations. Finally, with the advent of large language models (LLMs) featuring outstanding reasoning capabilities, several works have sought to employ LLMs by constructing large vision-language models (LVLM) for video understanding. Unfortunately, rather than explicitly modeling temporal relationships in a video, they often rely on spatial inductive biases, assuming that spatial knowledge can smoothly extend to temporal comprehension. Such reliance has prevented modern video understanding models from fully utilizing their understanding capability. To overcome this limitation, we conduct a thorough empirical study to demystify crucial components that influence temporal understanding in LVLMs. Our empirical study reveals that the crucial component lies in the intermediate interface between the visual encoder and the LLM. Building on such insight, we propose a temporal-oriented recipe that encompasses temporal-oriented training scheme and an upscaled interface. Our final model developed using our recipe significantly enhances previous LVLMs across standard video understanding benchmarks. \dsp Doctor of Philosophy \field Institute of Data Science \degreeyear 2025 \supervisor Professor See-Kiong Ng \examiners Associate Professor Yao Yingjie Angela Assistant Professor Bryan Hooi Kuen-Yew \declaredate 02 June 2025 \declaresign pic/signature.png \declarationpage Acknowledgements. First and foremost, I am deeply indebted to my supervisor, Prof. See-Kiong Ng, for his guidance, advice and encouragement, as well as giving me extraordinary experiences throughout my doctoral program. Above all and the most needed, he has provided me with unflinching encouragement and support in a wide variety of ways. Without his guidance, this thesis would not have been completed or written. I am deeply grateful to Prof. Luu Anh Tuan, who is my external supervisor and collaborator ever since my undergraduate years. He has inspired me to take on this PhD journey, and I have continued to benefit from his advice during my PhD years. Thanks to his detailed comments and advice, my research journey has become extremely fruitful and enjoyable. I also wish to express my gratitude to Prof. Angela Yao, Prof. Min-Yen Kan, and Prof. Bryan Hooi for their priceless advice. I thank them for their thorough comments and suggestions towards my research works and research direction. Their critical comments contribute to the backbone of this research. I also want to thank Prof. Bryan Low for his valuable advice towards my research career. I would like to thank AI Singapore and Google Inc. for providing me with the scholarship to pursue this research, as well as supporting the finance for me to present papers in several overseas conferences. Besides, I am thankful to my friends, colleagues, and collaborators. I am grateful for their friendship throughout my study, and I really enjoy my time with these brilliant people. Finally, I would like to sincerely and deeply thank my parents, who have taken care of me with great love in these years, and my sister for all of her jokes. Most importantly, my heartfelt thanks go to my spouse. It is her constant support and encouragement that gives me strength to persevere throughout my research journey. Authorship Attribution Statement This thesis contains material from the following paper(s) published in the respective peer-reviewed conferences in which I am listed as an author. Chapter 2 is published with material from the following paper(s): 1. Thong Nguyen , Yi Bin, Junbin Xiao, Leigang Qu, Yicong Li, Jay Zhangjie Wu, Cong-Duy T Nguyen, See-Kiong Ng, Luu Anh Tuan. Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives . Annual Meeting of the Association for Computational Linguistics ( ACL ), 2024 (Findings) Chapter 3 is published with material from: 1. Thong Nguyen , Yi Bin, Xiaobao Wu, Xinshuai Dong, Zhiyuan Hu, Khoi Le, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. MAMA: A Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning . The European Conference on Computer Vision ( ECCV ), 2024 Chapter 4 is published with material from: 1. Thong Nguyen , Xiaobao Wu, Xinshuai Dong, Khoi Le, Zhiyuan Hu, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling . Proceedings of AAAI Conference on Artificial Intelligence ( AAAI ), 2024 Chapter 5 is published with material from: 1. Thong Nguyen , Zhiyuan Hu, Xiaobao Wu, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. Encoding and Controlling Global Semantics for Long-form Video Question Answering . Proceedings of Empirical Methods in Natural Language Processing ( EMNLP ), 2024 Chapter 6 is published with material from: 1. Thong Nguyen , Xiaobao Wu, Yi Bin, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation . Proceedings of AAAI Conference on Artificial Intelligence ( AAAI ), 2025 Chapter 7 is published with material from: 1. Thong Nguyen , Yi Bin, Xiaobao Wu, Zhiyuan Hu, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. Multi-Scale Contrastive Learning for Video Temporal Grounding . Proceedings of AAAI Conference on Artificial Intelligence ( AAAI ), 2025 Chapter 8 is published with material from: 1. Thong Nguyen , Zhiyuan Hu, Xu Lin, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding . In submission to Conference on Neural Information Processing ( NeurIPS ), 2025 Other publications during my PhD study: 1. Cong-Duy Nguyen, Xiaobao Wu, Thong Nguyen , Shuai Zhao, Khoi Le, Viet-Anh Nguyen, Feng Yichao, Luu Anh Tuan. Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation . Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics ( NAACL ), 2025 2. Thong Nguyen , Xiaobao Wu, Xinshuai Dong, Cong-Duy T Nguyen, See-Kiong Ng, Luu Anh Tuan. Topic Modeling as Multi-Objective Contrastive Optimization . Proceedings of International Conference on Learning Representations ( ICLR ), 2024 3. Xiaobao Wu, Fengjun Pan, Thong Nguyen , Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, Luu Anh Tuan. On the affinity, rationality, and diversity of hierarchical topic modeling . Proceedings of AAAI Conference on Artificial Intelligence ( AAAI ), 2024 4. Xiaobao Wu, Thong Nguyen , Luu Anh Tuan. A survey on neural topic models: methods, applications, and challenges . Artificial Intelligence Review, 2024 5. Xiaobao Wu, Thong Nguyen , Delvin Ce Zhang, William Yang Wang, Luu Anh Tuan. FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm . Proceedings of Advances in Neural Information Processing Systems ( NeurIPS ), 2024 6. Cong-Duy Nguyen, Thong Nguyen , Duc Anh Vu, Luu Anh Tuan. KDMCSE: Knowledge distillation multimodal sentence embeddings with adaptive angular margin contrastive learning . Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics ( NAACL ), 2024 7. Thong Nguyen , Xiaobao Wu, Xinshuai Dong, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan. DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding . Proceedings of Empirical Methods in Natural Language Processing ( EMNLP ), 2023 (Findings) 8. Cong-Duy Nguyen, Thong Nguyen , Anh Vu, Luu Anh Tuan. Improving multimodal sentiment analysis: Supervised angular margin-based contrastive learning for enhanced fusion representation . Proceedings of Empirical Methods in Natural Language Processing ( EMNLP ), 2023 (Findings) 9. Xiaobao Wu, Xinshuai Dong, Thong Nguyen , Luu Anh Tuan. Effective Neural Topic Modeling with Embedding Clustering Regularization . Proceedings of International Conference on Machine Learning ( ICML ), 2023 10. Thong Nguyen , Xiaobao Wu, Xinshuai Dong, Luu Anh Tuan, Cong-Duy Nguyen, Zhen Hai, Lidong Bing. Gradient-Boosted Decision Tree for Listwise Context Model in Multimodal Review Helpfulness Prediction . Annual Meeting of the Association for Computational Linguistics ( ACL ), 2023 (Findings) 11. Xiaobao Wu, Xinshuai Dong, Thong Nguyen , Chaoqun Liu, Liangming Pan, Luu Anh Tuan. InfoCTM: A Mutual Information Maximization Perspective of Cross-lingual Topic Modeling . Proceedings of AAAI Conference on Artificial Intelligence ( AAAI ), 2023 12. Thong Nguyen , Xiaobao Wu, Tuan Luu, Cong-Duy Nguyen, Zhen Hai, Lidong Bing. Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Predictions . Proceedings of Empirical Methods in Natural Language Processing ( EMNLP ), 2022 Chapter 1 Introduction Video data is among the richest and most rapidly growing forms of digital information in the world. Every day, millions of videos, spanning hundreds of thousands of hours, are uploaded to online platforms. This steadily increasing data treasure is highly diverse, reflecting real-world complexity not only through static visual cues but also through temporal dynamics. There are two fundamental video types, i.e. natural footage and edited videos. Natural footage captures events as they unfold, whereas edited videos are shaped by human interventions such as cuts, splits, and transitions. In this work, we aim to develop artificial intelligence capable of understanding video content by constructing video understanding models. Different from image understanding, which primarily captures static visual information, video understanding must model both spatial and temporal aspects to determine what happens, when it happens, and how events evolve over time. Both natural and edited video understanding have extensive applications. For instance, analyzing natural footage enables tasks such as temporal grounding to localize video moments of anomalous activities, such as arson, shoplifting, and fighting. Similarly, video summarization can distill long meeting recordings into concise highlights to support business decision-making. Edited video understanding also offers practical applications. For example, text-video retrieval models can help users locate relevant movies based on queries, enhancing entertainment experiences. Unlike images, videos inherently encompass a temporal dimension, marking events as they unfold. Nevertheless, this temporal aspect is difficult to represent explicitly. To robustly capture this temporal aspect of videos, we develop strong temporal understanding capabilities for video understanding models. Humans naturally perceive temporal context, such as when we see a friend waving, we infer their movement; when a ball is suspended mid-air, we understand that it is moving. This awareness of change, sequence, and continuity stems from our lifelong exposure to dynamic events, and arguably from evolutionary experiences encoded in our genes. In contrast, instilling such temporal awareness in neural models is computationally demanding. Videos consist of numerous high-resolution frames, and processing hundreds of them often already exceeds the memory capacity of modern GPUs. Moreover, beyond frame-level analysis, effectively extracting and leveraging meaningful temporal cues remains a major challenge. As a result, video understanding has not reached the same level of success as achieved in image or language understanding, primarily due to the complexity of modeling temporal dynamics. Therefore, this thesis argues that explicitly modeling temporal signals for video understanding can substantially enhance video understanding. In particular, we concentrate on relations that only emerge when time is considered, e.g. relations between object motions or relations between video moments, and investigate techniques to capture these temporal cues across three dimensions, i.e. data, model, and training. While the datasets used in this dissertation are predominantly edited videos, our methods are designed to generalize to both natural footage and edited videos. We believe that tailoring approaches to either natural or edited videos offers a promising future research avenue. The following chapters of this thesis delve into the specifics of our temporal-oriented approach, demonstrate its advantages, and discuss its implications for advancing the field of video understanding. 1 Challenges of Video Understanding Performance Limits In Video Understanding. Prior research has established scaling laws in deep learning, showing that performance tends to improve as both model size and training data increase ( 79 ; 103 ) . This implies that, regardless of advancements in architecture or training techniques, meaningful performance gains will eventually stall without a corresponding increase in scale. Video understanding faces unique challenges in this regard: a single video can comprise hundreds or thousands of high-resolution frames, making processing computationally expensive. Moreover, videos demand significantly more storage than images due to their multi-frame nature. As a result, video models and datasets have lagged behind those in image and language domains in terms of scale. Building on established scaling laws, this thesis aims to push the boundaries of video understanding by proposing strategies to scale models and training datasets to enable the effective capture of implicit temporal relations among video elements for video understanding. Inefficient capture of temporal relations in video understanding models. Most video understanding architectures leverage self-attention-based modules originally designed for language understanding tasks. These modules require large volumes of training samples to capture useful inductive biases required for accurate semantic interpretation of text or video. Prior work has shown inferior performance under data scarcity ( 48 ) . Unfortunately, as mentioned, video datasets are generally far smaller than their image or language counterparts—a gap that widens in privacy-sensitive or long-tail scenarios. As a consequence, we must develop efficient techniques that enable models to quickly extract useful cues from limited training data. Ineffective capture of temporal relations in video understanding models. A key challenge addressed in this thesis is the under-utilization of temporal cues in video understanding. Since many prior works focus on static frame-level features, they overlook deeper semantics that arise from temporal phenomena such as motion continuity, event formation, and long-range temporal dependencies. These elements derive their semantics only when we consider core temporal properties: change ( e.g. detecting motion through variation across object masks), duration ( e.g. identifying the span of an event), and succession ( e.g. understanding how moments build into narratives). For instance, when an object shifts position across time—such as hand waving—we instinctively infer the motion, not just a sequence of static poses. Similarly, we interpret a stream of frames not as isolated images, but as a cohesive event, often organizing them into structured stories that unfold over time. In this thesis, we focus on emphasizing these semantics in representations generated by video understanding models by forming internal relations for motions, events, and long-term context. 2 Research Objectives Given the challenges, we seek to advance video understanding by addressing them. As such, we tackle the following objectives which respectively correspond to the aforementioned challenges. • Research Objective 1 (RO1): Pushing the Frontier of Video Understanding. Video understanding remains constrained by both architecture and dataset scale. We will expand these dimensions by (a) scaling up the model components most critical for temporal understanding, and (b) introducing an automatic annotation approach that automatically annotates a video by temporally linking sampled frames into a coherent video caption. • Research Objective 2 (RO2): Efficient Capture of Temporal Relations in Video Understanding Models. In certain scenarios—such as privacy-sensitive environments or settings with limited computational and storage resources—acquiring large-scale video training data, whether supervised or non-supervised, is often infeasible. In such cases, models may lack access to rich temporal cues, particularly if their architectures do not inherently target video data. Furthermore, these low-resource contexts might also demand high adaptability, requiring support for multiple downstream tasks such as video summarization or temporal grounding. Therefore, beyond simply improving effectiveness, we also design modular components or training strategies that facilitate efficient temporal modeling under resource constraints. Our proposed solution offers a lightweight, task-adaptable framework that maintains strong performance across diverse video understanding tasks while minimizing training and storage costs. • Research Objective 3 (RO3): Effective Capture of Temporal Relations in Video Understanding Models. We seek to integrate our inductive biases that emphasize temporal semantics in videos. These biases reflect common temporal relations among video elements: (i) Relation between Motions: When an object changes positions over time, it exhibits motion. Actions belonging to the same semantic category often share similar motion patterns. We encourage the model to learn representations that are close in feature space for actions of the same type, thereby reinforcing motion-level consistency. (ii) Relation between Events: As frames progress sequentially, temporal properties such as duration, succession, and order emerge, forming a coherent event. To encode the content of an event, models generate compact representations for temporally ordered frame sequences that capture event-level temporal dynamics. In our work, we encourage the model to map events of similar semantics into similar embeddings, while separating dissimilar events. (iii) Long-Term Context: Videos typically consist of temporally linked that form a coherent narrative or context. In longer videos, such as movies, modeling this long-term temporal context poses both computational and representational challenges. We enhance the capacity of video models to capture and utilize long-range temporal dependencies, and we validate our approach on datasets featuring extended video content. 3 Contributions and Outline of This Thesis In this section, we describe our contributions of this thesis with the according research objectives. For each contribution, we provide reference to the respective chapter in this thesis. The outline of the thesis is illustrated in Figure 1 . Figure 1 : Outline of the thesis. • Contribution 1 - RO1 - Chapter 3 : Meta-optimized Angular MArgin (MAMA) contrastive framework for pushing the limit of video understanding through escalating training data. Particularly, MAMA utilizes a pretrained large vision-language model to annotate a vast set of video data. Because automatically annotated data can contain noise, MAMA uses regularization training which consist of a subtractive angular margin integrated in the contrastive objective along with a multi-layer perceptron (MLP) to dynamically adjust model’s attention to training samples. This can alleviate the noisy effect of synthetic data upon video understanding model training. • Contribution 2 - RO2 - Chapter 4 : REcurrent ADapter (READ) for efficient capture of temporal relations in low-resource video understanding. In low-resource scenarios, there exists limited video data to provide sufficient temporal cues for video understanding models to learn from. To remedy this problem, we incorporate recurrent computation into low-dimensional adapters to encourage temporal modeling capability. We further implement a video-text alignment training objective to motivate the model to focus upon task-related information during video encoding. • Contribution 3 - RO3 - Chapter 5 : Encoding Global Video Semantics for effective capture of temporal long-term context in long videos. Prior works cope with excessive computational cost of long-form video understanding by selecting a small number of frames for video understanding model, which leads to significant information loss. We mitigate this problem by incorporating SSLs into model architectures to encode all densely sampled frames which can absorb video’s global semantics with reduced complexity. We target our SSL-augmented architecture for long-form VideoQA, and validate its superiority on our two collected benchmarks involving movies and egocentric videos. • Contribution 3 - RO3 - Chapter 6 : Motion-aware Contrastive Learning (MCL) for effective capture of relations between object motions. MCL employs a novel contrastive training objective to pull closer representations for pairs of object masks possess similar actions. We explicitly target dynamic actions such as “ run ”, “ walk ”, and “ throw ” with strong motion cues. Experiments demonstrate the effectiveness of our method for video panoptic scene graph generation, a comprehensive video understanding task whose aim is to segment a video into object masks and predict their relations. • Contribution 4 - RO3 - Chapter 7 : Multi-Scale Contrastive Learning (MSCL) for effective capture of relations between video moments. Our MSCL iteratively learns close representations for short-range and long-range video moments that are commonly described by an event description. By modeling the relations between short-range and long-range moments, we are able to deconfound the effect of surface properties of video elements, e.g. video moment length. Moreover, we can adopt a query-based sampling strategy for our framework to significantly reduce computational and memory cost, which are significant burdens of previous contrastive learning frameworks. • Contribution 5 - RO1 - Chapter 8 : Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding for pushing the limit of video understanding through advancing model architecture. With the advent of large language models (LLMs), a popular research avenue is to design LLM-oriented systems to inherit LLM’s impressive reasoning capability for video understanding. In our work, to further push the frontier of these systems, we first conduct extensive experiments to demystify essential components that significantly affect video understanding performance. As we discover that the vision-language interface is the most influential part, we devise a recipe centered on the interface. In our recipe, we progressively choose Q-Former as the vision-language interface, add temporal-oriented training schemes, incorporate temporal memory bank to store video representations, and augment mixture-of-experts (MoE) into the interface. These steps result in notable improvement for large vision-language models, as demonstrated by their strong performance on standard video understanding benchmarks. Chapter 2 Literature Review 4 Video Understanding Tasks Figure 2 : Level hierarchy of video understanding tasks. Text-video retrieval. Text-video retrieval is the task to search for the corresponding video given a language query (text-to-video), or oppositely search for the language description given a video (video-to-text). In practical applications, returning an entire video may not be desirable. Hence, video moment retrieval (VMR) has emerged with an aim to accurately locating relevant moments within a video based on user queries. VMR examines more nuanced and fine-grained understanding to capture different concepts and events in a video in order to pinpoint specific moments rather than capturing the overall theme in standard text-video retrieval. Video captioning. Video captioning is the task to generate a concise language description for a video. A video captioning model receives as input a video and optionally a language transcript transcribed from the audio in the video. Typically, a model produces a sentence-level caption for the whole video, or might also generate a paragraph as a more detailed summary. Video question answering (videoQA). Video question answering is the task to predict the correct answer based on a question q and a video v . There are two fundamental types of VideoQA, i.e. multi-choice VideoQA and open-ended VideoQA. In multi-choice VideoQA, a model is presented with a certain number of candidate answers and it will choose the correct answer among them. Open-ended VideoQA can be formulated as a classification problem, a generation problem, or a regression problem. Classification-based VideoQA associates a video-question pair with an answer from a pre-defined vocabulary set. Generation-based VideoQA is not restricted to a vocabulary set, in which a model can generate a sequence of tokens that represent the answer to a question. Regression-based VideoQA is often used for counting questions, e.g. counting the repetitions of an action or counting the number of an object in a video. Connections among video understanding tasks. These tasks constitute the three fundamental testbeds for evaluating video understanding (see Figures 4 and 5 for their examples). Despite their operational differences, they can be unified under a common framework. Formally, let V=\{f_{1},f_{2},\dots,f_{N}\} denote a video with N frames, and let T denote a textual input, such as a query, question, or prompt. A video-language model M learns a mapping from the video V and text input T into an output Y , i.e. f:(V,T)\rightarrow Y . In text-video retrieval, Y is a relevance score between V and T . In video captioning, Y is a sequence of words describing V . In VideoQA, Y is the predicted answer conditioned on V and T . This unified formulation highlights that video understanding tasks share a foundation of joint video-language representation learning. These video understanding tasks can also be organized into a hierarchy of understanding levels, as illustrated in Figure 2 . At the basic level, text-video retrieval globally aligns a video with its textual counterpart. At the intermediate level, video captioning requires mapping salient entities and events in a video to a coherent language description, make it a more difficult task than text-video retrieval. At the highest level, VideoQA demands reasoning over both modalities to generate accurate answers. Each level further extends to more fine-grained or inferential variants. In particular, video moment retrieval, or temporal grounding, extends text-video retrieval; dense video captioning ( 317 ) and video chapter generation ( 277 ) extend capitoning; and inference-focused VideoQA ( 257 ; 128 ) extends VideoQA. These advanced tasks pose greater challenges and are increasingly central to advancing video research towards human-level intelligence ( 53 ) . Figure 3 : Timeline of the established video understanding methods (TVR: Text-video retrieval, VC: video captioning, VQA: video question answering, TF: Transformer, LLM: large language model). From left to right, our legend table follows the order: pre-Transformer (Pre-TF), task-specific Transformer, multi-task Transformer, and LLM-augmented architectures. Figure 4 : Illustration of text-video retrieval, video captioning, and video question answer (videoQA) tasks. Figure 5 : More illustration of video moment retrieval, action recognition, action segmentation, dense video captioning, video chapter generation, and multimodal abstractive summarization tasks. 5 Challenges of Video Understanding The discussed video understanding tasks present unique challenges compared with image-language understanding, since a video incorporates an additional temporal channel. We summarize their important challenges as follows: Intra-modal and cross-modal interaction. While intra-modal interaction modeling within language can be directly taken from image-language understanding, intra-modal interaction modeling within video is different since it jointly consists of spatial interaction and temporal interaction. Spatial interaction delves into the relationships among pixels, patches, regions, or objects within an individual frame, whereas temporal interaction captures sequential dependencies among video frames or video segments. Longer video durations amplify the complexity of temporal modeling by necessitating the recognition of more objects and events in a higher number of video frames ( 289 ; 147 ) , and reasoning their long-term dependencies 313 . Particular video domains, such as egocentric videos, also complicate temporal interaction modeling, as objects undergo drastic appearance and disappearance dynamics over time, posing challenges in capturing their relationships ( 13 ; 216 ) . Given the larger semantic gap for video-language compared to image-language, cross-modal interaction plays a crucial role in video understanding. The interaction between visual and language features is pivotal for aligning the semantics of video and text query to associate them for text-video retrieval, or identifying relevant parts to answer the question and writing the caption in videoQA and video captioning, respectively. In addition, incorporating the interaction of motion and language features can mitigate the extraction of noisy information from videos ( 47 ) . 147 also discover that the interaction between audio and language features can compactly capture information related to objects, actions, and complex events, compensating for sparsely extracted video frames. Cross-domain adaptation. Given the infinitude of online videos, that our video understanding model will encounter testing scenarios which are identically distributed to our training data is an impractical assumption. Moreover, with the advent of LLM-augmented models that can tackle a variety video understanding tasks ( 131 ; 140 ) , it is currently more advisable to train a model that can effectively adapt to multiple tasks and domains than to obtain a model which specializes in a specific understanding task. Furthermore, since a video can be considered as a sequence of images, training a model on video-text data is more computationally expensive than image-text data. Combined with the large-scale of recent video understanding models ( 93 ; 275 ) , there is also a need to devise an efficient fine-tuning strategy to save the computational cost of fine-tuning these models. Data preparation. Although 120 only use image-text data to train models for video understanding tasks, in essence, video-text data are crucial for the effectiveness of these models. In particular, compared with a static image, a video offers richer information with diverse spatial semantics with consistent temporal dynamics ( 323 ) . As such, 37 find that training on videos outperforms training on images, but jointly training on both data achieves the best performance. As additional evidence, 293 shows that video-pretrained models outperform image-pretrained models in classifying motion-rich videos. However, video-text data takes up more storage cost than image-text data since a video comprises multiple images as video frames. Moreover, annotating a video is also more time-consuming and labor-intensive than annotating an image ( 264 ) . Therefore, video understanding models have been limited by the small size of clean paired video-text corpora in contrast to billion-scale image-text datasets ( 311 ) . Various efforts ( 311 ; 264 ) have been put into devising efficient and economical methods to curate and label video-text data. Addressing challenges. These identified challenges encompass three critical perspectives: model architecture, model training, and data preparation in the field of video understanding. In general, there should be a synergistic relationship among these components. Specifically, model architecture should be designed to effectively capture video-language interactions. Concurrently, model training should be tailored to enable the architecture to adapt to target domains with their captured video-language interactions. Lastly, data preparation plays a pivotal role in shaping model training, which in turn significantly impacts the development of an efficacious model architecture. 6 Model Architecture for Video- Language Understanding Figure 6 : Illustration of Transformer-based architectures. Addressing the challenge of intra-modal and cross-modal interaction is the key aim in designing video understanding model architectures, which can be divided into Pre-transformer and Transformer-based architectures . The advent of LLMs with remarkable zero-shot capability in addressing multiple tasks led to the design of LLM-augmented architectures that exhibit cross-domain adaptation ability to various video understanding tasks. 6.1 Pre-Transformer Architecture Pre-transformer architectures typically comprise unimodal video and language encoders for implementing intra-modal interactions and cross-modal encoders for cross-modal interactions. Unimodal encoders. A video encoder often encodes raw videos by extracting frame appearance and clip motion features as spatial and temporal representations, respectively. As each video frame can be considered as a single image, various works have utilized CNNs to extract spatial representations ( 205 ; 55 ; 312 ) . For temporal representations, the sequential nature of RNN makes it a popular choice in pre-transformer architectures ( 285 ; 310 ; 227 ; 229 ) . Furthermore, 3D CNNs with an additional temporal channel inserted to 2D CNN have also demonstrated effectiveness in extracting spatio-temporal representations ( 224 ; 24 ) . In addition to CNN and RNN, 33 , 63 , and 247 also build graphs to incorporate intra-modal relationships among video entities such as video segments or visual objects. These graph-structured works emphasize the reasoning ability of the model architecture. A common framework of language encoder is to extract pre-trained word embeddings such as word2vec ( 104 ; 291 ) or GloVe ( 221 ; 110 ) , then proceed with RNN-based modules such as LSTM or GRU. Such framework is taken from language model architectures before the era of Transformer. Cross-modal encoders. 61 and 299 apply element-wise multiplication to fuse the global video and question representations for video question answering. It demonstrates the advantage of a simple operation for video-language fusion. Attention has also been used to model video-language relations, in order to identify salient parts in video and language sentence ( 294 ) , or to refine the representation of the video based on the language question ( 265 ) . Pre-transformer video-language works have also combined attention with a wide variety of techniques, including hierarchical learning ( 14 ) , memory networks ( 51 ) , and graph networks ( 259 ; 244 ) . 6.2 Transformer-based Architecture Developed based on the self-attention mechanism, which exhaustively correlates every pair of input tokens with each other, Transformer-based architecture has the capacity to capture long-term dependencies and learn from web-scale data. It has demonstrated remarkable performance in many video-language tasks. Similar to the pre-transformer architecture, the Transformer-based framework also comprises unimodal encoders and cross-modal encoders to model intra-modal and cross-modal interactions, respectively. For unimodal encoders, several works find vision transformer for video encoding and BERT encoder for language encoding outperform RNN- and CNN-based encoding ( 56 ; 11 ; 200 ) . We then summarize fundamental types of Transformer-based architectures and illustrate them in Figure 6 . Shared Transformer. Motivated by the success of Transformer in language modeling ( 46 ) , 4 and 231 construct a shared Transformer encoder for video understanding. Their encoder architectures receive the concatenation of visual patches and language tokens, then jointly calculate their interactions in a BERT-based manner. 4 additionally incorporate modality embeddings which comprise three values to denote three kinds of input modalities, i.e. (video, audio, text). Stacked Transformer. 134 reveals that a shared Transformer encoder is weak in modeling temporal relations between videos and texts. To address this problem, they introduce a stacked Transformer architecture, with a hierarchical stack consisting of unimodal encoders to encode video and language inputs separately, and then a cross-modal Transformer to compute video-language interactions. A multitude of video understanding works follow such design to stack a cross-modal Transformer-based encoder above unimodal encoders ( 57 ; 135 ; 120 ; 245 ; 156 ; 179 ; 246 ) . To perform video captioning, 200 and 155 further insert a causal Transformer-based decoder that generates language tokens based on the encoded cross-modal representations. Dual Transformer. Dual Transformer architectures have been favored for text-video retrieval ( 156 ; 11 ; 12 ; 147 ; 273 ) . These architectures use two Transformer encoders to encode video and language separately, yielding global representations for each input modality, then applying simple operations such as cosine similarity to compute cross-modal interaction. Such a separate encoding scheme enables them to mitigate the computational cost of computing pairwise interactions between every pair of video and language inputs. They have accomplished not only efficiency but also effectiveness in text-video retrieval problems. 6.3 LLM-Augmented Architecture Large language models (LLMs) have achieved impressive results in simultaneously tackling multiple NLP tasks. Recent efforts have sought to apply LLMs for video understanding to extend its cross-domain adaptation ability to video-language settings ( 30 ; 131 ) . These efforts can be categorized into two approaches. The first approach employs LLM as a controller and video understanding models as helping tools. The controller will call the specific tool according to the language input instruction. The second approach utilizes LLM as the output generator and seeks to align video pre-trained models to the LLM. For video understanding, since the second approach dominates the first one with a long list of recent works ( 30 ; 131 ; 30 ; 140 ; 302 ; 159 ) , we review them as follows: LLM as Output Generator. The framework comprises a visual encoder, a semantic translator, and an LLM as the output generator. Regarding visual encoder, LLM-augmented architectures often use vision transformer and CNN models of the pre-Transformer and Transformer-based architectures ( 30 ) . Since an LLM has never seen a video during its training, a semantic translator is needed to translate the visual semantics of a video to the LLM’s semantics. For the translator, Video-LLaMA ( 302 ) and VideoChat ( 131 ) implement a Q-Former as a Transformer-based module that uses a sequence of query embeddings that interact with visual features of the video to extract informative video information. Instead of Q-Former, VideoLLM ( 30 ) , Video-ChatGPT ( 159 ) , and LLaMA-Vid ( 140 ) find that a simple linear projection that projects visual features into the LLM’s input dimension can achieve effective performance. Subsequently, these visual-based query embeddings or projected visual features are combined with the language instruction to become the input fed to the LLM to produce the final output. 6.4 Architecture Analysis In Figure 3 , we show the timeline of video understanding methodologies, categorized according to our defined architecture taxonomy and their affiliated downstream tasks. The evolution of pre-transformer models aligns with our hierarchy of video understanding levels, i.e. models for video captioning generally appear subsequent to those for text-video retrieval, followed by the development of videoQA models. Owing to their impressive capacity, Transformer-based models capable of addressing multiple tasks have been introduced concurrently with task-specific Transformer frameworks. Recently, large language models (LLMs) have gained prominence for their superior in-context learning ability, enabling them to handle diverse tasks without fine-tuning. Consequently, new LLM-augmented architectures have emerged to utilize this capability to address multiple understanding tasks. Among Transformer-based architectures, the dual Transformer stands out as the most effective for text-video retrieval, adeptly associating global semantics of video and language modality. On the other hand, the stacked Transformer architecture excels at facilitating intra-modal and cross-modal interactions through its specialized unimodal and cross-modal encoders. These encoders are particularly efficient at correlating video content with the question in videoQA. Additionally, for video captioning, cross-modal encoder plays a crucial role in translating video content into textual descriptions. Recent LLM-augmented models have begun to outperform Transformer-based architectures in videoQA, signalling their potential as the next frontier in video understanding research. We provide full details of performance in text-video retrieval, video captioning, and videoQA tasks in Table 1 , 2 , and 3 , respectively. Methods Model architecture Video Text R@1 R@5 R@10 VSE-LSTM ( 110 ) Pre-TF ConvNet/OxfordNet GloVe-LSTM 3.8 12.7 17.1 C+LSTM+SA-FC7 ( 221 ) VGG GloVe-LSTM 4.2 12.9 19.9 EITanque ( 104 ) VGG word2vec-LSTM 4.7 16.6 24.1 SA-G+SA-FC7 ( 221 ) VGG GloVe 3.1 9.0 13.4 CT-SAN ( 291 ) RN word2vec-LSTM 4.4 16.6 22.3 JSFusion ( 290 ) RN GloVe-LSTM 10.2 31.2 43.2 All-in-one ( 231 ) Shared TF Linear BT 37.9 68.1 77.1 VLM ( 267 ) Shared TF S3D BT 28.1 55.5 67.4 DeCEMBERT ( 218 ) Shared TF RN BT 17.5 44.3 58.6 ActBERT ( 321 ) Stacked TF Faster-RCNN BT 16.3 42.8 56.9 VIOLET ( 57 ) Stacked TF VS-TF BT 37.2 64.8 75.8 VindLU ( 37 ) Stacked TF ViT BT 48.8 72.4 82.2 HERO ( 134 ) Stacked TF RN+SlowFast BT 16.8 43.4 57.7 MV-GPT ( 200 ) Stacked TF ViViT BT 37.3 65.5 75.1 CLIP2TV ( 62 ) Dual TF ViT CLIP-text 32.4 58.2 68.6 CLIP-ViP ( 272 ) Dual TF ViT CLIP-text 49.6 74.5 84.8 CLIP4Clip ( 156 ) Dual TF ViT CLIP-text 44.5 71.4 81.6 Table 1 : Performance on text-video retrieval. (Pre-TF: Pre-transformer, Shared TF: Shared Transformer, Stack TF: Stack Transformer, Dual TF: Dual Transformer, RN: ResNet/ResNeXt ( 76 ; 263 ) , ViT: Vision Transformer ( 48 ) , BT: BERT ( 46 ) , ViViT: Video Vision Transformer ( 10 ) ). We report recall at rank 1 (R@1), 5 (R@5), and 10 (R@10). We choose MSRVTT as one of the most popular datasets for text-video retrieval. Methods Model architecture Video BLEU-4 METEOR CIDEr TA ( 286 ) Pre-TF Video: 3D-CNN 36.5 25.7 - h-RNN ( 287 ) Video: VGG 36.8 25.9 - MFATT ( 154 ) Video: RN+C3D 39.1 26.7 - CAT-TM ( 154 ) Video: RN+C3D 36.6 25.6 - NFS-TM ( 154 ) Video: RN+C3D 37.0 25.9 - Fuse-TM ( 154 ) Video: RN+C3D 37.5 25.9 - MARN ( 184 ) Video: RN - - 46.8 Res-ATT ( 136 ) Video: RN 37.0 26.9 40.7 DenseLSTM ( 322 ) Video: VGG 38.1 27.2 42.8 VIOLET ( 57 ) Stacked TF VS-TF - - 58.0 LAVENDER ( 135 ) VS-TF - - 57.4 VLAB ( 77 ) EVA-G 54.6 33.4 74.9 UniVL ( 155 ) S3D 41.8 28.9 50.0 MV-GPT ( 200 ) ViViT 48.9 38.7 60.0 CLIP-DCD ( 278 ) ViT 48.2 30.9 64.8 DeCEMBERT ( 218 ) RN 45.2 29.7 52.3 mPLUG-2 ( 266 ) ViT 57.8 34.9 80.3 Table 2 : Performance on video captioning. (Pre-TF: Pre-transformer, Stacked TF: Stacked Transformer, RN: ResNet/ResNeXt ( 76 ; 263 ) , ViViT: Video Vision Transformer ( 10 ) , EVA-G: 52 ). We report BLEU-4 and METEOR, which are two popular metrics for language generation. We choose MSRVTT as one of the most popular datasets for video captioning. Methods Architecture Video Text Dataset MSRVTT MSVD E-MN ( 265 ) Pre-TF VGG + C3D GloVe-LSTM 30.4 26.7 QueST ( 95 ) RN + C3D GloVe-LSTM 40.0 - HME ( 51 ) RN/VGG + C3D GloVe-GRU 34.6 36.1 HGA ( 97 ) RN/VGG + C3D GloVe-GRU 33.0 33.7 ST-VQA ( 88 ) RN+C3D GloVe-LSTM 35.5 34.7 PGAT ( 185 ) Faster-RCNN GloVe-LSTM 38.1 39.0 HCRN ( 116 ) RN GloVe-LSTM 35.6 36.1 HQGA ( 259 ) Faster-RCNN BERT-LSTM 38.6 41.2 All in one ( 231 ) Shared TF Linear BT 44.3 47.9 LAVENDER ( 135 ) Stacked TF VS-TF BT 45.0 56.6 DeCEMBERT ( 218 ) Stacked TF RN BT 37.4 - VindLU ( 37 ) Stacked TF ViT BT 44.6 - VIOLET ( 57 ) Stacked TF VS-TF BT 44.5 54.7 ClipBERT ( 120 ) Stacked TF CLIP-text BT 37.4 - VGT ( 260 ) Dual TF Faster-RCNN BT 39.7 - CoVGT ( 261 ) Dual TF Faster-RCNN BT 40.0 - LLaMA-Vid ( 140 ) LLM-Augmented EVA-G Vicuna 58.9 70.0 Table 3 : Performance on videoQA. (Pre-TF: Pre-transformer, Dual TF: Dual Transformer, RN: ResNet/ResNeXt ( 76 ; 263 ) , BT: BERT ( 46 ) , VS-TF: Video Swin Transformer ( 152 ) , EVA-G: 52 ). We report accuracy of the methods. We choose MSRVTT and MSVD as two of the most popular datasets for videoQA. 7 Model Training for Video Understanding Model training seeks to address the cross-domain adaptation ability of video understanding models. To achieve this goal, pre-training strategies have been devised to gain world knowledge that generalizes across multiple scenarios, then task-specific fine-tuning is conducted to specifically improve downstream task performance. 7.1 Pre-training for Video Understanding In this section, we mainly summarize pre-training strategies for video understanding models into three groups: Language-based pre-training. The most popular language-based pre-training task is masked language modeling (MLM) ( 120 ; 212 ; 37 ) , which randomly masks a portion of words in the language input and trains the model to predict the masked words based on unmasked language words and video entities. Instead of masking a portion of words, UniVL ( 155 ) and VICTOR ( 117 ) discover that masking the whole language modality benefits video captioning task. MLM can be combined with other language-based pre-training task, e.g. masked sentence order modeling which is to classify the original order of the shuffled language sentences ( 117 ) . Video-based pre-training. Video-based pre-training tasks help video-language models capture contextual information in the video modality. As a counterpart of MLM, masked video modeling (MVM) trains the model to predict the portion of masked video entities based upon the unmasked entities and language words. The continuous nature of videos leads to different choices of video entities, such as frame patches ( 134 ) or video frames ( 56 ) . In terms of the training objective, 134 use L2 regression loss to train the model to predict pre-trained features of the masked video frames extracted by ResNet and SlowFast models, while 56 use cross-entropy loss to train the model to predict the masked visual tokens, which are quantized by a variational autoencoder from visual frame patches. Video-text pre-training. Video-text pre-training is crucial for a model to capture video-language relation. 273 , 62 , and 11 utilize a framework of video-text contrastive learning to produce close representations for semantically similar video and language inputs. These works focus on creating a joint semantic space that aligns separate representations of video and language. Instead of separate representations, 218 , 56 , and 135 enable video and textual representations to interact with each other and use a single token to represent the cross-modal input, which is forwarded to predict whether the video-text pair is matched or not. In these two pre-training frameworks, not only video-text data but also image-text data are utilized during pre-training, in which an image is considered as a video with a single frame. Contrastive learning has revealed promising results ( 147 ; 62 ; 273 ; 178 ; 172 ; 174 ; 170 ; 169 ; 250 ; 252 ; 253 ) . MLM has contributed to enhancing VideoQA since the task resembles MLM in predicting the language word given a video-language pair (the question is the language input in videoQA). Compared to these pre-training strategies, MVM does provide performance gain for video understanding but its gain is less significant. For more details about pre-training, please refer to ( 37 ) . 7.2 Fine-tuning for Video Understanding Task-specific fine-tuning is commonly used by pre-Transformer architectures to train from scratch since these models do not have sufficient parameter capacity to learn generalizable features through pre-training. It is also widely adopted by Transformer-based architectures to improve the performance for a specific downstream task. Moreover, LLM-augmented architectures also utilize instruction tuning as a variant of fine-tuning, to adapt from the visual and audio spaces to the LLM language space. Fine-tuning strategies. Normally, all of the model parameters are updated during fine-tuning ( 61 ; 268 ; 7 ; 177 ; 251 ) . However, in cases computational resources or training data are limited, only adaptation layers such as low-rank adapters ( 180 ; 275 ; 176 ) or learnable prompt vectors ( 101 ) are fine-tuned to reduce training cost or prevent overfitting. Such risks also apply for LLM-augmented architectures discussed in Section 6.3 , since LLMs exhibit a billion scale of parameters, thus incurring excessively huge cost if full fine-tuning is conducted. For such models, 302 and 140 design a two-stage instruction tuning strategy which only fine-tunes the semantic translator. The first stage trains the model to generate the textual description based on the combined video and the language instruction, in order to align visual representations extracted by the visual encoder with the language space of LLM. The second stage is often performed on small-scale video-text pairs manually collected by the authors to further tailor the output features of the translator towards the target domains. 8 Data Perspective for Video Understanding In this section, we analyze data preparation approaches for video understanding models, and provide details of the datasets in Table 4 . Dataset Video source Annotation Tasks #Videos/#Routes MSVD ( 29 ) YouTube videos Manual TVR, VC, VideoQA 1.9K MSRVTT ( 269 ) Web videos Manual TVR, VC, VideoQA 7.2K ActivityNet ( 292 ) YouTube videos Manual AL, TVR, VC, VMR 5.8K FIBER ( 26 ) VaTeX ( 238 ) Manual VC, VideoQA 28K WildQA ( 25 ) YouTube videos Manual VideoQA 0.4K NExT-QA ( 257 ) VidOR 202 Manual VideoQA 5.4K CausalVid-QA ( 128 ) Kinetics-700 ( 23 ) Manual VideoQA 26K HowTo100M ( 164 ) YouTube videos Auto PT 1.2M HD-VILA-100M ( 272 ) YouTube videos Auto PT 3.3M YT-Temporal-180M ( 298 ) YouTube videos Auto PT 6M TGIF-QA ( 89 ) Animated GIFs Manual VideoQA 71K TGIF-QA-R ( 185 ) TGIF-QA ( 89 ) Manual, Auto VideoQA 71K DiDeMo ( 7 ) YFCC100M ( 219 ) Manual TVR 11K YouCook2 ( 316 ) YouTube videos Manual TVR, VC 2K HMDB-51 ( 115 ) Web videos Manual TVR, AR 6.8K Kinetics-400 ( 106 ) YouTube videos Manual AR 306K Kinetics-600 ( 22 ) Kinetics-400 ( 106 ) Manual AR, VG 480K Kinetics-700 ( 23 ) Kinetics-600 ( 22 ) Manual AR 650K VaTeX ( 238 ) Kinetics-600 ( 22 ) Manual TVR, VC 41K TVR ( 123 ) TVQA ( 121 ) Manual VMR 22K How2R ( 134 ) HowTo100M ( 164 ) Manual VMR 22K How2QA ( 134 ) HowTo100M ( 164 ) Manual VideoQA 22K YouTube Highlights ( 213 ) YouTube videos Manual VMR 0.6K TACoS ( 194 ) MPII Composites ( 197 ) Manual VMR 0.1K QVHighlights ( 118 ) YouTube vlogs Manual VMR 10K TVSum ( 209 ) YouTube videos Manual VMR 50 ViTT ( 85 ) YouTube-8M ( 2 ) Manual VMR 5.8K VidChapters-7M ( 276 ) YT-Temporal-180M ( 298 ) Auto VC, VMR 817K VideoCC3M ( 168 ) Web videos Auto PT 6.3M WebVid-10M ( 11 ) Web videos Auto PT 10.7M COIN ( 217 ) YouTube videos Manual AS 12K CrossTask ( 324 ) YouTube videos Manual AR 4.7K Alivol-10M ( 117 ) E-commerce videos Auto PT 10M LSMDC ( 196 ) British movies Manual TVR 72 EK-100 ( 45 ) Manual Manual AR, AL 7K SSV1 ( 65 ) Manual Manual AR 108K SSV2 ( 65 ) Manual Manual AR 221K Moments in Time ( 165 ) Web videos Manual AR 1M InternVid ( 239 ) YouTube videos Auto PT 7.1M How2 ( 199 ) YouTube videos Auto VC 13.2K WTS70M ( 210 ) YouTube videos Auto PT 70M Charades ( 61 ) Manual Manual AR, VMR, VideoQA 10K Table 4 : Video understanding datasets in the literature. (VMR: Video moment retrieval, TVR: text-video retrieval, VC: video captioning, AL: action localization, AR: action recognition, AS: action segmentation, VG: video generation, PT: pre-training). 8.1 Data curation Manual collection. To curate video-language data, multiple works search for publicly available online videos, which exhibit a wide diversity of content. Video-language datasets with online videos are mostly aimed for pre-training models to learn generalizable knowledge, e.g. HowTo100M ( 164 ) and YT-Temporal-180M ( 298 ) , or they can also be used for fine-tuning, e.g. MSRVTT ( 269 ) and YouCook2 ( 316 ) . To satisfy a certain requirement, videos different from the online ones can be inherited from existing datasets, e.g 257 utilize 6,000 videos from VidOR dataset and ( 128 ) inherit 546,882 videos from Kinetics-700 since they describe scenes of daily life and real world, respectively. Apart from making use of existing datasets’ and online videos, videos can also be recorded by human annotators to enable quality control ( 65 ; 45 ) . Data augmentation. Rather than manually collecting videos from external sources, 264 and 98 explore data augmentation techniques which are particularly designed for videos. In detail, their TubeTokenMix mixes two videos in which the mixing coefficient is defined upon the temporal dimension, and their temporal shift randomly shifts video frame features backward or forward over the temporal dimension. These techniques outperform standard augmentation approaches for image data, such as CutMix ( 295 ) , Mixup ( 304 ) , and PixMix ( 78 ) . 8.2 Label annotation Manual annotation. Several works ( 128 ; 118 ; 257 ) use human annotators since they provide high-quality labels. However, such approach is expensive, particularly when dealing with video data. For example, annotating QVHighlights dataset ( 118 ) costs approximately $16,000 for 10K videos and 3 months to complete. Similarly, NExT-QA ( 257 ) needs 100 undergraduate students and 1 year to annotate only 5K videos. Automatic generation. Directly taking language transcripts of YouTube videos as textual labels could reduce annotation cost ( 164 ; 272 ; 298 ) . However, these labels have been shown to be grammatically incorrect and temporally misalign with the video content ( 218 ) . Motivated by the success of LLMs, 311 train a system consisting of a TimeSformer-L visual encoder and a GPT-2XL decoder to write dense captions for videos. Moreover, 131 use GPT-4 to generate summaries for movie synopses. 9 Future Directions Fine-grained understanding. Existing methods excel at video understanding at a coarse-grained level, enabling effective responses to questions like “ what is ” or the recognition of global events without significant difficulty ( 257 ) . Nevertheless, limiting comprehension to this coarse level could hinder practical utility of existing systems. In real-world scenarios, a user might require a precise timestamp and location of an object within a video ( 94 ) , or request the AI agent to forecast potential alternative events, which is a common need in predictive analytics ( 257 ; 128 ) . These tasks necessitate an advanced understanding and inference capability regarding the causal and temporal relationships present in a video. At present, models exhibit a constrained visio-linguistic capacity to engage in temporal reasoning, categorizing them as image-sequence-and-language models rather than video-language models ( 107 ) . Therefore, future research in this direction deserves more attention and exploration. Long-form video understanding. Current understanding systems have demonstrated remarkable performance on short video clips lasting several seconds. However, they tend to struggle when switching to long-form videos which last several minutes or hours. To enhance the applicability of these systems, it is essential to enhance their capability of understanding long-form videos. Current approaches mainly feature reducing computational cost through architectures more efficient than Transformer-based ones such as state space models ( 284 ; 132 ) , which can be considered as linear RNN with specifically designed fixed weights, or compensating sparsely extracted video frames with additional information ( 147 ) . In general, how to effectively model long-form videos and adapt them to the joint context with language deserves more attention. Trustworthiness of video understanding models. Although modern video understanding systems have demonstrated remarkable performance, their black-box nature undermines our trust to deploy them. In particular, we still do not precisely understand what part of the video a videoQA model looks at to answer the question ( 142 ) , or how video and language semantic information flows into the common representation space of the video retrieval model ( 92 ) . Furthermore, adversarial noise sensitivity or hallucination of video understanding models are also open problems. Future trustworthiness benchmarks such as ( 258 ; 233 ) for video understanding are of great significance towards practical systems. Chapter 3 MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning 10 Introduction Figure 7 : Examples of video inputs and their textual descriptions. Figure 8 : Topic distribution of the MSRVTT dataset. We use Latent Dirichlet Allocation (LDA) to extract topics from manually annotated descriptions of videos. Learning vision and language representations has advanced state-of-the-art across multiple cross-modal tasks, such as video question answering (VideoQA) ( 120 ; 134 ; 56 ; 57 ; 37 ; 200 ; 231 ) and text-video retrieval ( 321 ; 218 ; 267 ; 119 ; 135 ; 232 ; 18 ) . The success of vision-language representation learning mainly results from the effectiveness of contrastive learning, which projects video and text inputs into a common latent space based on their semantic similarities. Data quality stands at the forefront of influencing the efficacy of video-language representation learning, particularly through its cleanliness and diversity, which are pivotal in optimizing model performance and bolstering generalization capabilities ( 311 ) . Within this context, we pinpoint two interrelated issues that significantly impact the cleanliness and diversity of video-text data: (1) the imperfection in alignment, characterized by a scarcity of fine-grained details, and (2) the imbalance among the concepts of data samples. First, a video and its textual description commonly does not perfectly align with each other, e.g. the description may omit certain details in the video, such as the green grass on a soccer field or the pink flowers at the background in two videos of Fig. 7 . As such, aggressively minimizing a contrastive loss to pull video and language representations together might result in distorted video-language representations that do not closely capture the semantic similarity of video-text pairs, thus compromising the interpretive video-language understanding. Second, as an example, Fig. 8 shows a majority of video-text pairs in the well-known MSRVTT dataset denote the video game or singing topic (the topic bars), while a minority of them describe the fashion topic (the bottom bar). This might skew the model’s exposure towards certain topics at the expense of others, undermining the model’s ability to perform uniformly across a broad spectrum of subjects ( 126 ) . Addressing these intertwined challenges is crucial for advancing video-language representation learning and facilitating more effective video-language understanding. Our method, called MAMA : M eta-optimized A ngular MA rgin Contrastive Learning, subtracts a margin between a positive video and text sample in the angular space. Our mathematical derivation shows that the subtracted margin can decay the gradient norm, thus providing a regularization effect to constrain positive but imperfectly aligned samples from reaching perfect similarity. For the imbalance issue, to enable the network to dynamically adjust its focus during training, MAMA introduces a sample reweighting strategy that maps loss values to sample weights. A natural idea is to assign higher weights to larger losses to emphasize minority classes for which the network is likely to make mistakes ( 214 ; 160 ) . However, with respect to the imperfect alignment issue, it is advisable to assign higher weights to smaller losses, as there is a greater chance that these samples better align with each other, hence forming cleaner samples which the network should focus on 308 ; 240 . To avoid the exhaustive effort in manually specifying a weighting function and enhance the function generalization, we parameterize our weighting function as a multi-layer perceptron (MLP) as theoretically a universal approximator for any continuous function ( 42 ) , and use a small unbiased validation set (meta-data) to train the MLP. Moreover, to further enhance the diversity in video-language representation learning, MAMA utilizes off-the-shelf large vision-language model (LVLM) to augment downstream video-text data. In particular, given an additional video input, we adopt the density peak-based clustering approach to extract its key frames, then concatenate the extracted frames into one grid image, and forward the image with a relevant input prompt to obtain the text pairing. Combined with the LVLM-augmented video-text data, MAMA considerably outperforms previous state-of-the-art video-language representation learning methods on standard MSRVTT, DiDeMo, ActivityNet, TGIF-QA-R, NExT-QA, and Causal-VidQA datasets. Figure 9 : Illustration of the proposed MAMA framework and its components. 11 MAMA Framework In this section, we explain MAMA, the meta-optimized angular margin contrastive framework with our augmentation strategy for video-language representation learning. We provide an overall illustration of MAMA in Figure 9 . 11.1 Video-Language Representation Learning We are given a corpus of video data, in which each video \mathbf{v}_{i} is attached with a textual description \mathbf{t}_{i} and possibly a label y_{i} . In the beginning, we embed the video input \mathbf{v}_{i} into a sequence of visual representations \mathbf{V}_{i}^{f}=\{\mathbf{h}_{i,j}^{f}\}_{j=1}^{N_{\mathbf{v}_{i}}} , where N_{\mathbf{v}_{i}} is the number of randomly sampled video frames in \mathbf{v}_{i} . We also embed the textual description into a sequence of representations \mathbf{T}^{w}_{i}=\{\mathbf{h}_{i,j}^{w}\}_{j=1}^{N_{\mathbf{t}_{i}}} , where N_{\mathbf{t}_{i}} is the number of words in \mathbf{t}_{i} . In this work, we conduct video-language representation learning for two types of video-language model, i.e. dual and bidirectional model. For the dual type, we will pool the visual representations \mathbf{V}^{f}_{i} and textual representations \mathbf{T}^{w}_{i} to obtain global representations \mathbf{h}^{V}_{i} and \mathbf{h}^{T}_{i} for the video and text input, respectively. We use cosine similarity to calculate the cross-modal similarity S_{\mathbf{v}_{i},\mathbf{t}_{i}} . Then, we aim to maximize the similarities S_{\mathbf{v}_{i},\mathbf{t}_{i}} of video-text pairs within the data, relative to those of the unaligned video-text pairs. The bidirectional model will concatenate the visual and textual representations into a sequence, then feed the sequence to Transformer attention layers to capture the video-text relations. Subsequently, we will forward the [CLS] token which represents the video-text sequence to a fully-connected (FC) layer. The FC layer will calculate the similarity S_{\mathbf{v}_{i},\mathbf{t}_{i}} if the video input \mathbf{v}_{i} has a text pairing \mathbf{t}_{i} , or calculate the log-likelihood \log q(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i}) of the answer if the video input \mathbf{v}_{i} has a label y_{i} . In the former case, similar to the dual type we also maximize the similarity of in-distribution video-text pairs, whereas for the latter case we maximize the log-likelihood of the answer label y_{i} . In general, for all video inputs, we maximize the similarity S_{\mathbf{v},\mathbf{t}} through minimizing the cross-modal contrastive loss as: \hskip-35.0pt\small\mathcal{L}^{v,t}_{\text{contrastive},i}=-\log\frac{e^{S_{\textbf{v}_{i},\textbf{t}_{i}}/\tau}}{\sum\limits_{j=1}^{B}e^{S_{\textbf{v}_{i},\textbf{t}_{j}}/\tau}},\;\;\;\;\mathcal{L}^{t,v}_{\text{contrastive},i}=-\log\frac{e^{S_{\textbf{t}_{i},\textbf{v}_{i}}/\tau}}{\sum\limits_{j=1}^{B}e^{S_{\textbf{t}_{i},\textbf{v}_{j}}}/\tau}, (1) where B is the batch size and \tau is the temperature hyperparameter. If a video \mathbf{v}_{i} additionally exhibits an answer label y_{i} , we jointly combine the cross-entropy loss to maximize the log-likelihood of the label with the above contrastive loss: \displaystyle\small\mathcal{L}_{\text{CE},i}=-\sum\limits_{y_{i}}p(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i})\log q(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i}), (2) \displaystyle\mathcal{L}_{\text{train},i}=\mathcal{L}^{v,t}_{\text{contrastive},i}+\mathcal{L}^{t,v}_{\text{contrastive},i}+\eta\cdot\mathcal{L}_{\text{CE},i}, (3) where \eta is a hyperparameter to balance both losses. It is worth noting that \eta=0 if a video input \mathbf{v}_{i} is not attached with y_{i} . 11.2 Meta-optimized Angular Margin Contrastive Framework Algorithm 1 Our meta-optimized learning framework 1: Training data \mathcal{D} , meta-data \widehat{\mathcal{D}} , training batch size B , meta-data batch size M , initialized video-language model parameter \Theta^{(0)} and MLP network parameter \theta^{(0)} 2: Video-language model parameter \Theta^{(K)} 3: for k=0 to K-1 do 4: Sample a training minibatch \{\textbf{v},\textbf{t},y\} . 5: Sample a meta-data minibatch \{\textbf{v}^{\text{meta}},\textbf{t}^{\text{meta}},y^{\text{meta}}\} . 6: Estimate \hat{\Theta}^{(t)} by Eq. ( 9 ). 7: Update \theta^{(k+1)} by Eq. ( 10 ). 8: Update \Theta^{(k+1)} by Eq. ( 11 ). 9: end for Subtractive angular margin contrastive learning. As cosine similarity is used to calculate S_{\mathbf{v}_{i},\mathbf{t}_{j}}\in[-1,1] , we denote the angle between the representation of video i and text j as: \small\lambda_{i,j}=\arccos\left(S_{\mathbf{v}_{i},\mathbf{t}_{j}}\right). (4) The original video-language representation learning minimizes \lambda_{i,i} to approach a 0-degree angle. However, since the video and its textual description commonly does not perfectly align with each other, we want the gradient to be regularized when the similarity of positive pair \textbf{v}_{i} and \textbf{t}_{i} becomes small. Particularly, we replace \mathcal{L}_{\text{contrastive},i}^{v,t} by a new training objective: \mathcal{L}_{\text{angular},i}^{v,t}=\begin{cases}-\log\frac{e^{\cos\left([\lambda_{i,i}-\mu]_{+}\right)/\tau}}{e^{\cos\left([\lambda_{i,i}-\mu]_{+}\right)/\tau}+\sum\limits_{j\neq i}e^{\cos(\lambda_{i,j})/\tau}},&\text{if }\lambda_{i,i}\leq\frac{\pi}{2}\\ -\log\frac{e^{\cos(\lambda_{i,i})/\tau}}{e^{\cos(\lambda_{i,i})/\tau}+\sum\limits_{j\neq i}e^{\cos(\lambda_{i,j})/\tau}},&\text{otherwise},\end{cases} A similar objective \mathcal{L}_{\text{angular},i}^{t,v} is formulated between \mathbf{t}_{i} and \mathbf{v}_{i} . As long as the angular difference \lambda_{i,i} is smaller than \mu , the similarity score will become 1. On the other hand, if \lambda_{i,i} is larger but starts to turn small, i.e. \mu\leq\lambda_{i,i}\leq\frac{\pi}{2} , the subtractive margin \mu will temporarily pull the positive video and text closer to regularize the gradient of the positive samples, thus restraining them from reaching perfect similarity. This intuition is formalized by the following theorem: Theorem 1 . Let \lambda_{i,j} denote the angle between the representation of two samples i,j , \mathcal{L}^{v,t}_{\textup{angular},i} and \mathcal{L}^{v,t}_{\textup{contrastive},i} denote the training objectives with and without the angular margin, respectively. Then, if \lambda_{i,i}\leq\frac{\pi}{2} , the following inequality holds: \small\left|\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}\right|\leq\left|\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}\right| (5) In order to speed up the training in the beginning and constrain the update of positive pairs in the latter stage, we adopt an adaptive strategy that gradually increases the margin \mu towards a limit: \small\mu^{(k)}=\frac{a_{0}}{a_{1}+e^{-a_{2}\cdot k}}, (6) where k denotes the training step, and a_{0} , a_{1} , and a_{2} denote hyperparameters. MLP-parameterized weighting function. To control the effect of the data imbalance issue, we construct a weighting function as a \theta -parameterized MLP to map each training loss to a sample weight: \small w(\theta,\mathcal{L}_{i}^{\text{train}}(\Theta))=\text{MLP}_{\theta}(\mathcal{L}_{i}^{\text{train}}(\Theta)),\;\;i\in\{1,2,…,B\}, (7) where \mathcal{L}_{i}^{\text{train}}(\Theta) denotes the training loss of sample i in a batch of size B , calculated using the \Theta -parameterized video-language model. Then, the final training objective is the sum of the training losses of data samples weighted by w : \small\mathcal{L}(\Theta,\theta)=\frac{1}{B}\sum\limits_{i=1}^{B}w(\theta,\mathcal{L}_{i}^{\text{train}}(\Theta))\cdot\mathcal{L}_{i}^{\text{train}}(\Theta). (8) Meta-optimized learning framework. At present, we have a \theta -parameterized MLP network and a \Theta -parameterized video-language model to train. Since jointly training these two models might be unstable ( 203 ) , we instead develop a meta-learning approach. In our procedure, we manually extract a small amount of unbiased meta-data set \{\textbf{v}_{i}^{\text{meta}},\textbf{t}_{i}^{\text{meta}},y_{i}^{\text{meta}}\}_{i=1}^{M} , i.e. with semantically aligned textual description and balanced label distribution, representing the meta-knowledge of the groundtruth sample-label distribution, where M is the number of the meta-samples and M\ll N . We first estimate the update of the video-language model parameters \Theta : \small\hat{\Theta}^{(k)}=\Theta^{(k)}-\frac{\alpha}{B}\sum\limits_{j=1}^{B}w_{i}\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\cdot\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right), (9) where \alpha is the learning rate for the video-language model. Subsequently, we update the MLP network using the estimated \hat{\Theta} : \small\theta^{(k+1)}=\theta^{(k)}-\frac{\eta}{M}\sum\limits_{i=1}^{M}\nabla_{\theta}\mathcal{L}^{\text{meta}}_{i}\left(\hat{\Theta}^{(k)}\right), (10) where \beta is the learning rate for the MLP network. Lastly, we obtain the new video-language model’s parameters as: \begin{split}\Theta^{(k+1)}=\Theta^{(k)}-\frac{\alpha}{B}\sum\limits_{j=1}^{B}w\left(\theta^{(k+1)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\cdot\nabla_{\Theta}\mathcal{L}^{\text{train}}_{j}\left(\Theta\right)\Big|_{\Theta^{(k)}}.\end{split} (11) We summarize our algorithm in Algorithm 1 . From an empirical perspective, we compare the effectiveness of joint learning and our meta-optimized framework in Section 12.3 . From a theoretical perspective, we conduct further derivation for Eq. ( 10 ), resulting in the following formulation: \displaystyle\theta^{(k+1)}=\theta^{(k)}+\frac{\alpha\beta}{BM}\sum_{i=1}^{M}\sum_{j=1}^{B}G_{ij}\frac{\partial w\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)}{\partial\theta}\Big|_{\theta^{(k)}}, (12) \displaystyle G_{ij}=\nabla_{\hat{\Theta}}\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}\right)\Big|_{\hat{\Theta}^{(k)}}\cdot\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta\right)\Big|_{\Theta^{(k)}}, (13) where G_{ij} denotes the coefficient of the gradient. The coefficient will increase for training samples whose gradient is in the same direction with the gradient of the meta-data (meta gradient), as this is more likely to be a clean learning direction. In contrast, the coefficient will decrease for those whose gradient is opposite from the meta gradient, since the learning direction is likely to be noisy. 11.3 Large Vision-Language Model for Augmentation To further enhance our video-language representation learning, we devise a strategy to utilize large vision-language model to augment additional video-text data. Our strategy consists of two stages, i.e. extracting key frames and generating textual descriptions, which we delineate as follows: Key frame extraction. Inspired by ( 99 ) , we adopt a density peak-based clustering approach to extract key video frames. Starting with a sequence of visual frame features \textbf{V}_{i}^{f}=\{\mathbf{h}^{f}_{i,j}\}_{j=1}^{N_{\textbf{v}_{i}}} , we calculate the locality density d_{i,j} of each feature \mathbf{h}^{f}_{i,j} based on its K -nearest neighbors: \small d_{i,j}=\exp\left(-\frac{1}{K}\sum\limits_{\textbf{h}^{f}_{i,l}\in\text{KNN}(\mathbf{h}^{f}_{i,j})}\left|\left|\textbf{h}^{f}_{i,l}-\textbf{h}^{f}_{i,j}\right|\right|^{2}\right), (14) where \text{KNN}(\mathbf{h}^{f}_{i,j}) denotes the K -nearest neighbors of \mathbf{h}^{f}_{i,j} . Subsequently, we estimate the distance index \gamma_{i,j} of each frame \mathbf{h}_{i,j}^{f} : \displaystyle\small\gamma_{i,j}=\begin{cases}\min\limits_{l:d_{i,j}>d_{i,l}}||\mathbf{h}^{f}_{i,l}-\mathbf{h}^{f}_{i,j}||^{2},&\text{if }\exists l\;\text{s.t.}\;d_{i,l}>d_{i,j}\\ \max\limits_{l}||\mathbf{h}^{f}_{i,l}-\mathbf{h}^{f}_{i,j}||,&\text{otherwise}\end{cases}. (15) Our intuition is that d_{i,j} denotes the local density of a video frame, and \gamma_{i,j} denotes the distance from other frames of high density. We proceed to extract the top Q video frames of the highest d_{i,j}\times\gamma_{i,j} values as key video frames. Textual description generation. Since large vision-language models have established impressive results on natural instruction tuning and visual reasoning capabilities, we leverage a LVLM to augment textual descriptions for video data ( 43 ; 129 ; 148 ) . Particularly, given the top Q video frames, we concatenate them into a single image I of the W\times H grid. We then forward the image I along with the textual prompt “ Write a short caption sentence for the video in order from left to right, top to bottom ” to a large vision-language model to generate the sentence-level textual description t , which we combine with the video v to form a video-text pair (\textbf{v},\textbf{t}) . 12 Experiments 12.1 Experimental Settings Downstream tasks. Following previous works ( 99 ; 212 ; 311 ; 161 ) , we evaluate our framework on two popular video question answering and text \leftrightarrow video retrieval tasks. • Video question answering (VideoQA): We experiment with two videoQA settings, i.e. open-ended and multi-choice videoQA. Open-ended videoQA classifies a pair of video and question into one of the pre-defined set of answer labels. Multi-choice videoQA chooses the correct answer from five choices given the video and the language question. We assess our video-language representations by the VideoQA task using the following five datasets, MSRVTT ( 269 ) , MSVD ( 29 ) , TGIF-QA-R ( 185 ) , NExT-QA ( 257 ) , and Causal-VidQA ( 128 ) . • Text \leftrightarrow video retrieval: The retrieval task is to extract the corresponding video given the textual query, or extract the textual description given the video. We evaluate the retrieval ability of our video-language representations using three datasets: MSRVTT ( 269 ) , DiDeMo ( 7 ) , ActivityNet ( 113 ) . Evaluation metrics. For the videoQA task, we use the answer accuracy as the evaluation metric. For the retrieval task, we employ Recall at rank K ( R@K ), with K\in\{1,5,10\} to evaluate the performance. Video-language backbones. To extensively validate the effectiveness of our framework, we conduct experiments on various models of bidirectional and dual architectures. Particularly, for the dual architecture, we experiment with CLIP-ViP ( 273 ) and video graph transformer (VGT) ( 261 ) , while for the bidirectional architecture, we experiment with the well-known VIOLET architecture ( 57 ) . Baseline methods. We compare our method with a comprehensive list of video-language understanding models, along with the approaches that use LVLM to augment video-text data: (i) CLIP4Clip ( 156 ) , a model that transfers image-text CLIP model ( 191 ) to text \leftrightarrow video retrieval tasks; (ii) MERLOT ( 298 ) , a method that trains on both spatial and temporal objectives to learn video-language representations; (iii) LAVENDER ( 135 ) , a model that learns video-language representations using a unified framework of masked language modeling; (iv) Singularity ( 119 ) , curates a pre-training dataset and uses an early fusion strategy to improve single-frame video-language representation learning; (v) OmniVL ( 232 ) : a model that is trained on both image-language and video-language data to enhance video-language representation learning; (vi) VindLU ( 37 ) , follows a recipe to select pre-training objectives, pre-training data, and model scale for effective video-language representation learning; (vii) All-in-one ( 231 ) , a model that embeds raw video and textual signals into hidden representations without using pre-trained unimodal encoders; (viii) DRL ( 235 ) , a disentangled representation method that decouples sequential from hierarchical representations to specifically improve both of them for cross-modal retrieval; (ix) CLIP2Video ( 234 ) , consists of two normalization methods to improve the effectiveness and robustness of cross-modal retrieval models; (x) LaViLa ( 311 ) , an approach that fine-tunes LLM to be conditioned on visual frames to generate additional textual descriptions, which are used to improve video-language representation learning; (xi) Vid2Seq ( 277 ) , a pre-trained dense video captioning model which is used to generate additional textual descriptions for video data, which are employed to enhance video-language representation learning. Implementation details. We utilize LLaVA model as the LVLM ( 148 ) to augment video inputs of the HowTo100M dataset ( 164 ) , which is a large-scale dataset for video-language representation learning but has been known for its weakly aligned video-text pairings ( 72 ) . To construct the input for the LVLM, we use K=6 and Q=12 , then concatenate key frames into an image as a grid of 3\times 4 frames. Based upon validation, we adopt \tau=1 , a_{0}=0.2,a_{1}=10 , and a_{2}=-0.1 for angular margin contrastive learning, and \eta=1 for our optimization objective. For videoQA, to fairly compare with previous methods, we fine-tune our VIOLET-based model using AdamW with an initial learning rate of \alpha=\beta=2e-5 , betas of (0.9, 0.98), weight decay of 1e-3, and batch size B of 4 for 10 epochs. Similarly, for the VGT-based model, we use Adam optimizer with an initial learning rate of 1e-5 of a cosine annealing schedule, and batch size B of 4 for 30 epochs. For the text \leftrightarrow video retrieval task, we fine-tune the model with an initial learning rate of 5e-6 and a fixed weight decay of 5e-2, using a batch size B of 4 for 5, 20, and 20 epochs on MSRVTT, DiDeMo, and ActivityNet datasets, respectively. 12.2 Main Results Table 5 : VideoQA results on MSRVTT, MSVD, and TGIF-QA-R. Open-ended videoQA is evaluated on MSRVTT, MSVD, and TGIF-Frame datasets. Multi-choice videoQA is evaluated upon TGIF-Action and TGIF-Transition. Method MSRVTT-QA MSVD-QA TGIF-QA-R Action Frame Transition MERLOT 43.1 51.9 61.4 69.3 84.0 LAVENDER 44.2 52.4 57.1 66.9 84.0 Singularity 43.5 49.6 53.1 65.1 81.5 OmniVL 44.1 51.0 62.0 69.5 85.5 VindLU 44.6 51.0 59.5 65.8 87.2 VGT 40.0 36.4 61.0 61.7 73.2 VIOLET 44.5 52.5 62.6 70.0 86.3 All-in-one 44.3 47.9 34.9 65.0 53.8 LaViLa-VIOLET 44.9 53.7 64.3 70.7 87.0 Vid2Seq-VIOLET 44.8 53.1 64.2 70.1 86.6 MAMA-VGT 41.6 37.1 61.7 62.7 74.0 MAMA-VIOLET 46.4 55.8 66.5 71.7 89.5 Table 6 : VideoQA results on NExT-QA and Causal-VidQA. All of the datasets target multi-choice videoQA. Method NExT-QA Causal-VidQA Descriptive Temporal Causal All Descriptive Explanatory Predictive Counterfactual MERLOT 66.6 62.5 58.2 59.4 67.2 65.7 57.2 57.0 LAVENDER 63.4 56.5 54.6 56.7 62.0 61.6 46.3 50.4 Singularity 60.0 61.1 50.9 54.8 54.3 49.6 41.3 46.9 OmniVL 67.1 63.8 55.9 59.9 67.0 66.0 56.5 57.1 VindLU 68.4 59.7 55.7 59.8 63.6 54.8 57.7 54.4 VGT 69.6 65.4 56.2 61.8 74.4 75.6 60.7 65.6 VIOLET 67.7 58.0 50.7 58.5 67.6 66.6 57.1 57.6 All-in-one 64.8 63.9 50.9 57.9 60.4 51.3 51.2 50.7 LaViLa-VGT 70.9 69.2 59.3 64.8 74.7 76.1 61.6 65.7 Vid2Seq-VGT 70.3 67.0 58.5 64.2 73.7 76.0 61.3 65.6 MAMA-VGT 72.7 70.6 62.2 66.3 75.3 77.1 62.2 68.2 MAMA-VIOLET 71.2 68.6 61.1 65.8 72.7 68.0 60.5 59.5 Table 7 : Text \rightarrow video retrieval results on MSRVTT, DiDeMo, and ActivityNet. Method MSRVTT DiDeMo ActivityNet R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 VIOLET 37.2 64.8 75.8 47.9 76.5 84.1 18.1 43.1 56.7 VindLU 48.8 72.4 82.2 59.8 86.6 91.5 55.9 82.3 90.9 CLIP4Clip 44.5 71.4 81.6 42.8 68.5 79.2 40.5 72.4 83.4 CLIP2Video 47.2 73.0 83.0 - - - - - - DRL 47.4 74.6 83.8 47.9 73.8 82.7 44.2 74.5 86.1 CLIP-ViP 55.7 77.7 86.6 55.7 78.1 86.1 59.1 83.9 91.3 LaViLa - CLIP-ViP 56.0 79.8 87.2 56.6 79.8 87.1 58.7 82.8 90.5 Vid2Seq - CLIP-ViP 55.3 77.9 86.1 57.6 79.9 88.4 55.1 79.0 87.4 MAMA - CLIP-ViP 60.0 82.2 89.2 62.7 89.9 96.0 60.0 85.0 91.7 Table 8 : Video \rightarrow text retrieval results on MSRVTT, DiDeMo, and ActivityNet. Method MSRVTT DiDeMo ActivityNet R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 VIOLET 36.6 64.1 75.1 44.8 72.9 82.4 15.8 39.8 54.8 VindLU 46.0 71.8 80.2 57.2 83.1 88.9 47.1 77.5 86.6 CLIP4Clip 43.1 70.5 81.2 17.5 37.5 49.4 42.6 73.4 85.6 CLIP2Video 43.5 72.3 82.1 - - - - - - DRL 45.1 72.9 83.5 45.4 72.6 82.1 42.2 74.0 86.2 CLIP-ViP 48.0 72.4 82.9 46.3 73.2 81.9 50.2 78.3 87.5 LaViLa - CLIP-ViP 49.0 74.5 83.4 47.1 74.2 83.0 50.3 78.7 88.4 Vid2Seq - CLIP-ViP 49.1 74.1 82.5 47.2 74.4 83.1 50.9 79.5 89.6 Our method - CLIP-ViP 50.1 76.8 84.8 48.1 78.1 85.5 52.3 80.8 90.1 We denote the results for videoQA in Table 5 and 6 , for text \rightarrow video retrieval in Table 7 , and for video \rightarrow text retrieval in Table 8 . VideoQA. In terms of open-ended videoQA, we substantially outperform the LVLM approaches, e.g. improve upon LaViLa with improvements of 1.5% on MSRVTT, 2.1% on MSVD, and 3.7% on TGIF-Frame. We also surpass previous video-language understanding models, e.g. VindLU by 1.8% on MSRVTT, VIOLET by 3.3% on MSVD, and OmniVL by 2.2% on TGIF-Frame. Moreover, for multi-choice videoQA, we consistently surpass both LVLM and video-language understanding models, e.g. achieving an overall gain of 1.5% on NExT-QA and 1.2% on Causal-VidQA over LaViLa, while the gains over VGT are 4.5% and 1.6%, respectively on these two datasets. Text \leftrightarrow video retrieval. Our observation in videoQA applies for the text \leftrightarrow video retrieval task. In the text \rightarrow video direction, we enhance upon CLIP-ViP by 2.6 R@10 points on MSRVTT, 5.5 R@1 points on DiDeMo, and 1.1 R@5 points on ActivityNet. We also significantly outperform the LVLM approaches, e.g. by 4.0 R@1 points and by 2.4 R@5 points on MSRVTT over LaViLa. Our superiority on video \rightarrow text retrieval is analogous to the text \rightarrow video case. These results substantiate that our framework is applicable to various video-language understanding tasks and model architectures. We hypothesize that we can better control the alignment between the positive video and language, and their effect upon the model training, thus leading to more reasonable video-language representations. 12.3 Ablation Study We ablate our meta-optimized framework to investigate which factor results in the demonstrated effectiveness and explore intriguing properties. We conduct all experiments of our ablation study on the MSRVTT and MSVD for videoQA, and MSRVTT and DiDeMo for the text \leftrightarrow video retrieval task. Varying the angular margin. The angular margin, represented as \mu , is critical to control the relationship between semantically close video and language inputs. To better understand \mu , we experiment with manually varying the margin \mu from 0.1 to 0.4 in Figure 10 (left). We observe that the optimal performance is achieved when \mu\sim 0.2 . Based upon this observation, we adopt a_{0}=2 and a_{1}=10 . Subsequently, we evaluate the impact of a_{2}\in\{0.01,0.05,0.1,0.15,0.2\} in Figure 10 (right). We discover that a_{2}=0.1 achieves the best performance and we adopt it for all experiments. Reweighting strategies. In addition to MLP, there exist various approaches to weigh training losses to control noisy pairings and data imbalance issues within the training data, e.g. focal loss ( 146 ) , self-paced learning (SPL) ( 96 ) , and L2RW ( 195 ) . We replace our MLP by these approaches and demonstrate the results in Table 10 . We find that our MLP-parameterized function significantly improves upon previous weighting functions that use manually designed formulation, i.e. focal loss and SPL, and also the meta-learning approach without using MLP, i.e. L2RW. The possible reason could be that MLP is a universal approximator, so that it can adaptively learn a reasonable weighting function based on data, thus outperforming methods that do not employ MLP. Table 9 : Experiments on the LVLM choice for our prompting strategy. LVLM choice VideoQA Text \rightarrow video retrieval MSRVTT MSVD MSRVTT DiDeMo R@1 R@5 R@10 R@1 R@5 R@10 No augmentation 45.6 54.7 59.0 80.3 87.6 58.0 82.9 89.6 BLIP-2 45.7 54.9 59.1 80.4 87.7 58.1 82.9 89.7 InstructBLIP 46.1 55.2 59.2 80.9 88.1 58.5 83.5 89.8 LLaVA-ChatGPT 46.1 55.0 59.0 80.3 87.7 58.7 83.9 90.0 LLaVA 46.4 55.8 60.0 82.2 89.2 62.7 89.9 96.0 Table 10 : Experiments on weighting functions. Weighting function VideoQA Text \rightarrow video retrieval MSRVTT MSVD MSRVTT DiDeMo R@1 R@5 R@10 R@1 R@5 R@10 Focal loss 45.0 54.1 58.3 80.0 86.9 59.7 85.6 91.8 SPL 45.3 54.3 58.7 80.8 88.1 61.4 87.5 93.8 L2RW 45.9 55.2 59.5 81.8 89.6 62.0 88.8 94.6 MLP 46.4 55.8 60.0 82.2 89.2 62.7 89.9 96.0 Optimization strategies. We explore different strategies to optimize the MLP and video-language model. Particularly, we experiment with the joint learning strategy, which simultaneously updates the parameters of the MLP and the video-language model to minimize the objective in Eq. 3 . As shown in Table 12 , our meta-learning approach outperforms the joint learning one and achieves the best performance. This empirically validates the effectiveness of our meta-learning strategy which enables the training to follow gradient of the meta-data. Table 11 : Experiments on the number of key frames Q and the concatenated grid. Q Grid VideoQA Text \rightarrow video retrieval MSRVTT MSVD MSRVTT DiDeMo R@1 R@5 R@10 R@1 R@5 R@10 1 1\times 1 45.0 55.1 58.2 79.3 86.2 60.2 85.8 93.2 2 1\times 2 45.7 55.3 58.8 80.0 87.0 60.5 85.8 93.3 2\times 1 45.9 55.4 58.4 79.2 87.0 60.9 86.1 94.2 4 1\times 4 45.9 55.2 59.1 80.3 88.4 61.3 86.3 94.6 2\times 2 45.6 55.2 59.2 80.4 88.4 61.3 86.7 94.6 8 2\times 4 45.9 55.6 59.8 81.3 88.7 61.8 89.0 95.3 4\times 2 45.8 55.5 59.3 80.7 88.5 61.8 88.9 95.3 12 3\times 4 46.4 55.8 60.0 82.2 89.2 62.7 89.9 96.0 2\times 6 45.8 55.6 59.3 81.7 89.1 62.7 89.2 95.6 16 2\times 8 45.8 55.1 58.3 79.7 87.6 61.9 88.7 94.8 4\times 4 45.7 55.1 58.9 80.8 88.7 62.6 89.1 95.9 Figure 10 : (Left) Validation videoQA accuracy on MSVD with respect to \mu ; (Middle) Validation videoQA accuracy on MSVD with respect to a_{0} ; (Right) Relationship between loss values and weight values generated by our MLP-parameterized weighting function. Video-text data augmentation. We assess the influence of different video-text data augmentation strategies. In particular, we ablate the LLaVA-generated samples and also explore different LVLM choices, i.e. BLIP-2 ( 129 ) and InstructBLIP ( 43 ) . In addition, we also use LLaVA model to generate image caption for each video frame, and ask ChatGPT to write a summary for the concatenated captions of all frames as the textual description. Table 9 shows that neglecting LVLM-generated data results in slight performance degradation, which can be resolved by using other LVLMs, but still lack behind LLaVA model. We conjecture that LLaVA can better follow the language instruction to produce more accurate textual descriptions for videos ( 148 ) . Moreover, LLaVA-ChatGPT approach also leads to performance degradation, possibly because it is still challenging for ChatGPT to infer temporal relations among captions of separate video frames. Table 12 : Experiments on the learning strategy. Training strategy VideoQA Text \rightarrow video retrieval MSRVTT MSVD MSRVTT DiDeMo R@1 R@5 R@10 R@1 R@5 R@10 Joint learning 45.5 54.9 59.3 81.1 88.4 59.2 84.9 90.2 Meta-learning 46.4 55.8 60.0 82.2 89.2 62.7 89.9 96.0 Figure 11 : a) Relative R@1 improvement in the text-video retrieval task on MSRVTT for each topic; b) Relative accuracy improvement in the videoQA task on MSRVTT with respect to the proportion of questions for which each label is the answer. Table 13 : Case study of similarity scores for video-text pairs. Video Caption CLIP-ViP score Our score a group of people dancing on a stage in front of a crowd. 0.1727 0.1698 the man in black reaches his dance partner. man walks to the woman in red the camera zooms out on the dancers. we see a dancer in the back left spinning 0.2175 0.3342 a boy in a red shirt and a blue hat is playing. 0.2594 0.2441 a girl throwing a rock. girl in the red shirt looks back over his shoulder the boy in a blue hat is seen for the first time. the boy in the baseball cap first looks at the camera boy wearing a blue ball cap 0.2869 0.3164 Number of extracted key frames. We explore the impact of varying number of extracted key frames and varying the approach to concatenate in Table 11 . While the performance is quite invariant to the concatenation strategy, we observe an increasing trend of the performance when the number of the extracted frames Q increases, since there exists more information for LVLM to put into the descriptions. However, after surpassing Q=12 frames, the performance dramatically decreases. The reason could be that when Q is excessively large, visual frames become excessively small for LVLM to precisely detect details. Furthermore, balanced grids, i.e. 3\times 4 and 4\times 4 , tend to outperform skewed grids, i.e. 2\times 6 and 2\times 8 , respectively. The cause might be that video details are more difficult for an LVLM to capture when the video frames are presented on a long rectangle than when they are presented on a balanced one. 12.4 Analysis Effect of angular margin contrastive learning. To better understand the effect of our angular margin contrastive method, we show in Table 13 examples of videos and language captions, along with the similarity scores which are generated by the baseline CLIP-ViP model and the model trained with our proposed subtractive contrastive learning strategy. We observe that when the caption consists of less details, i.e. it aligns more weakly with the video content, our model assigns a lower similarity score compared to CLIP-ViP. In contrast, when the language caption becomes more detailed, we assign a higher score while CLIP-ViP only slightly elevates the score. These examples demonstrate that our angular margin contrastive strategy can control the similarity level for weakly aligned video-text pairs while being able to adapt to cases of strongly aligned pairs. Effect of MLP-parameterized weighting function. To closely study the impact of our MLP-based weighting function, we show the relative R@1 and the relative accuracy improvement of the text-video retrieval tasks in Figure 11 a and 11 b, respectively. Both figures demonstrate that we accomplish higher level of performance improvement on samples that belong to minority groups, i.e. unpopular topic groups and answer labels. Such results intuitively substantiate the productivity of our MLP-based weighting function and showcase its capacity to learn the effective strategy to control the training effect of the training samples. 13 Summary In this paper, we propose a meta-optimized contrastive framework to enhance video-language representation learning. In particular, we propose a contrastive learning framework with a subtractive margin between positive video and language to regularize their representations from reaching perfect similarity. Our framework also utilizes an MLP network that maps training losses to sample weights to enable the model to dynamically adjust the focus on data samples during training. Combined with a strategy to utilize large-vision language model to augment video-text data, our framework achieves superior results on commonly used video question answering and text-video retrieval tasks. Our framework is also applicable to a wide array of model architectures, which can promote its implementation in practical applications. Chapter 4 READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling 14 Introduction Video-language modeling is a challenging problem since it involves understanding both video and language modalities. For example, temporal language grounding (TLG) model comprehends video detail and language query to localize semantically related video moments (Figure 12 (left)), or video-language summarization (VLS) model extracts information from both video content and language transcript to write the summary (Figure 12 (right)). Previous video-language modeling methods ( 151 ; 118 ; 288 ) employ pretrained Transformer models such as Unified Multimodal Transformer (UMT) ( 151 ) and Vision-Guided BART (VG-BART) ( 288 ) , and fine-tune all the parameters of these models for every single task. This results in substantial storage overhead since each task demands storing a separate model ( 300 ) . Moreover, because of the difficulty of collecting video-language data ( 180 ) , fully fine-tuning these over-parameterized models in low-resource scenarios, where limited training data is available, leads to instability and sub-optimal performance ( 93 ; 86 ) . Figure 12 : Examples of the TLG and VLS problems. TLG model needs to understand the meaning of language entities such as proposal or girl , and the existence of expression in video frames. VLS model is expected to recognize salient information, e.g. crank bolt , bottom bracket from the language, and bicycle from the video. To address these shortcomings, adapters are proposed as a parameter-efficient solution for finetuning video-language pretrained transformers ( 93 ; 300 ; 283 ; 215 ; 32 ) . The strategy is to add additional adaptation module to each layer of the pre-trained network and only the adaptation modules are trained during fine-tuning to improve the parameter-performance trade-off. These modules rely on non-linear projections to downproject video-language inputs into low-dimensional space then up-project them back to the original high-dimensional space. However, such projections consider video frames and textual words as separate tokens, thus ignoring the intrinsic temporal dependency among video frames or textual words. Without such dependency information, it is difficult to reason about temporal context in the video to properly ground the language ( e.g. in Figure 12 , determine the expression of the girl after , not before , the proposal ), or coherently link the entities in the summary ( e.g. in Figure 12 , recap the chronological order of bolt removing and puller threading ). Moreover, because at fine-tuning time only adaptation modules are trained using limited video-language data, little attention is paid to the information flow that starts from the raw video-language inputs till the low-dimensional space of the adaptation modules. This may result in losing essential task-related information and carrying noise into these modules ( 225 ; 73 ) . Figure 13 : Comparison of our proposed READ method with the full fine-tuning and other parameter-efficient fine-tuning methods. For each method, we denote the mAP gain averaged over the domains of the YouTube Highlights dataset together with the number of trainable parameters. To resolve the first issue, we propose a novel adapter architecture, RE current AD apter (READ), for video-language modeling tasks. The key idea is to incorporate the recurrent modeling ability into the adaptation module to capture the temporal dependency of video frames and language entities ( 64 ) . As such, we formulate READ as a parameter-efficient bottleneck with a sequence of operations including feature dimension reduction, recurrent modeling, and feature dimension recovery. Since the incorporated recurrent computation works in the low dimension ( e.g. 4-dimensional), our READ module stands as a lightweight design and can be cheaply integrated throughout the Transformer architecture for enhancing video-language modeling, using only up to 1.20% trainable parameters. As for the second issue, we propose P artial V ideo- L anguage A lignment (PVLA), a novel objective to explicitly encourage the alignment between video and language representations, thus capturing invariant aligned information across modalities that are critical for downstream tasks. The key concept is to minimize the Partial Optimal Transport (POT) distance between the distribution over video frame representations and the distribution over textual word representations. The rationale for our partial implementation of optimal transport lies in that video and language do not exhibit complete one-to-one correspondence. Typically, the language does not describe all aspects of the video, and only part of the language sequence is strongly related to part of the video frames, e.g. in Figure 12 the language input about the girl’s expression is only related to the target grounding. As such, utilizing POT for distribution matching is to focus on essential masses that are strongly related between modalities, hence optimizing towards better video-language alignment and gaining more control over video-language information passed into our READ modules. Based on our novel proposals, we construct the READ framework that can be employed to finetune various pre-trained Transformer architectures such as multimodal transformer (UMT ( 151 ) , Moment-DETR ( 118 ) ), and generative vision-guided transformer models (VG-BART ( 125 ) and VG-T5 ( 192 ) ). Through freezing these pre-trained models and fine-tuning only our READ modules with PVLA objective, we outperform standard fine-tuning and other parameter-efficient methods with substantially fewer tunable parameters (Figure 13 ) for low-resource video-language tasks, including temporal language grounding and video-language summarization. To sum up, our contributions can be summarized as: • We propose RE current AD apter (READ), a novel adapter architecture, that better captures temporal information for modeling video-language tasks. • We propose P artial V ideo- L anguage A lignment (PVLA) objective to encourage the alignment between video and language modalities during the adaptation process. • We validate our READ framework by extensive experiments using multiple low-resource temporal language grounding and video-language summarization datasets, where READ outperforms all existing fully or parameter-efficient fine-tuning strategies with only up to 1.20% parameters tunable. Figure 14 : Overall illustration of the proposed recurrent adapter (READ) and partial video-language alignment (PVLA) framework. 15 Methodology We present recurrent adapter (READ) to effectively develop the temporal modeling capability and efficiently transfer large pre-trained transformer models for video-language downstream tasks. We also introduce the partial video-language alignment (PVLA) task to optimize the alignment of in-distribution video-language inputs for better supporting video-language adaptation under low-resource settings. Our overall framework is illustrated in Figure 27 . 15.1 Preliminary – Transformer Architecture for Video-Language Modeling We concentrate our work upon the Transformer architecture ( 226 ) . The architecture consists of an embedding layer and M consecutive Transformer blocks. As inputs to the Transformer model, we extract N_{V} frames and N_{L} words from the video and language input, respectively. The embedding layer would encode the extracted frames and words into sequences of initial video and language representations H_{V}^{(0)}=\{\mathbf{h}_{v,i}^{(0)}\}_{i=1}^{N_{V}} and H_{L}^{(0)}=\{\mathbf{h}_{l,j}^{(0)}\}_{j=1}^{N_{L}} , respectively. The transformer then forwards these sequences into consecutive Transformer blocks, each of which is typically composed of a multi-head self-attention (MHSA) layer, a residual connection with normalization (Add & Norm) layer, a feedforward layer, and another Add & Norm layer. In MHSA for video-language modeling, the language representations are linearly projected into the query tensor \mathbf{Q}\in\mathbb{R}^{N_{L}\times d} , whilst the video representations into the key \mathbf{K}\in\mathbb{R}^{N_{V}\times d} and value tensors \mathbf{V}\in\mathbb{R}^{N_{V}\times d} : \displaystyle\mathbf{Q}^{(m)}=\text{Linear}\left(\mathbf{H}_{L}^{(m)}\right),\;\mathbf{K}^{(m)}=\text{Linear}\left(\mathbf{H}_{V}^{(m)}\right), \displaystyle\mathbf{V}^{(m)}=\text{Linear}\left(\mathbf{H}_{V}^{(m)}\right), where m denotes the index of the current Transformer block and d the hidden dimension. Then, the self-attention computation is conducted upon these vectors as: \begin{split}\mathbf{X}^{(m)}=\text{Attention}\left(\mathbf{Q}^{(m)},\mathbf{K}^{(m)},\mathbf{V}^{(m)}\right)=\\ \text{Softmax}\left(\frac{\mathbf{Q}^{(m)}\cdot(\mathbf{K}^{(m)})^{\top}}{\sqrt{d}}\right)\cdot\mathbf{V}^{(m)}.\end{split} (16) The attention output \mathbf{X}^{(m)} is subsequently sent to an Add & Norm layer: \mathbf{P}^{(m)}=\text{LN}\left(\mathbf{X}^{(m)}+\mathbf{H}^{(m)}_{L}\right), (17) where LN denotes the layer normalization layer. Subsequently, \mathbf{P}^{(m)} is forwarded to a FeedForward block to produce the output representation \mathbf{O}^{(m)} , which will be passed to another Add & Norm layer to create the video-informed language representation for the next transformer block: \displaystyle\mathbf{O}^{(m)}=\text{GeLU}\left(\text{Linear}\left(\mathbf{P}^{(m)}\right)\right), (18) \displaystyle\mathbf{H}_{L}^{(m+1)}=\text{LN}\left(\mathbf{P}^{(m)}+\mathbf{O}^{(m)}\right),\;\mathbf{H}_{V}^{(m+1)}=\mathbf{H}_{V}^{(m)}. (19) The video-language representation of the last Transformer block \mathbf{H}_{L}^{(M+1)} is finally adopted to perform a specific downstream task. 15.2 Recurrent Adapter (READ) The objective of our READ is to incorporate the temporal modeling capability for the adaptation module. To this end, we construct a recurrent-based bottleneck layer which is composed of a downprojection layer, a recurrent neural network (RNN) layer, and an up-projection layer. Formally, given the FeedForward output \mathbf{O} , our recurrent adapter can be expressed as: \displaystyle\mathbf{\tilde{O}}=\mathbf{O}+\text{GELU}\left(\text{RNN}\left(\mathbf{O}\cdot W_{\text{down}}\right)\right)\cdot W_{\text{up}}, (20) where W_{\text{down}}\in\mathbb{R}^{d\times k},W_{\text{up}}\in\mathbb{R}^{k\times d} , and k\ll d . Subsequently, we combine \mathbf{P} and \tilde{\mathbf{O}} via residual connection to generate the output \mathbf{H} : \mathbf{H}=\text{LN}\left(\mathbf{\tilde{O}}+\mathbf{P}\right). (21) In addition to RNN, we also experiment with other recurrent architectures in Table 21 and observe that the performance is insensitive to the architectural choice. Therefore, for simplicity, we decide to implement the RNN architecture in our READ layer. Fine-tuning. During the fine-tuning stage, we preserve the weights of the pre-trained Transformer model and only optimize our introduced READ layers. In detail, the original model components (blue blocks in Figure 27 ) are frozen, while the parameters of READ (the yellow block in Figure 27 ) are updated with respect to the task-specific and the partial video-language alignment losses, which will be delineated in the upcoming section. Testing. During testing, we maintain the shared parameters of the pre-trained Transformer model and only load those of our extra READ modules that are fine-tuned in the previous phase. This would keep the storage cost from burgeoning because the number of added parameters is tiny. 15.3 Partial Video-Language Alignment (PVLA) To encourage the control towards the information flow of video frames and language words, we propose to optimize the alignment between the in-distribution video and language representations H_{V} and H_{L} at all Transformer blocks. We consider video and language as two discrete distributions \boldsymbol{\mu} and \boldsymbol{\nu} , whose H_{V} and H_{L} are their supports, respectively. We formulate this setting as \boldsymbol{\mu}=\sum\limits_{i=1}^{N_{V}}\mathbf{a}_{i}\delta_{\mathbf{h}_{v,i}} and \boldsymbol{\nu}=\sum\limits_{j=1}^{N_{L}}\mathbf{b}_{j}\delta_{\mathbf{h}_{l,j}} , with \delta_{\mathbf{h}_{l,j}} and \delta_{\mathbf{h}_{l,j}} being the Dirac functions respectively centered upon \mathbf{h}_{v,i} and \mathbf{h}_{l,j} . The weight vector of the supports is \mathbf{a}=\frac{\mathbf{1}_{N_{V}}}{N_{V}} , and \mathbf{b}=\frac{\mathbf{1}_{N_{L}}}{N_{L}} . Based upon the above setting, we propose the partial video-language alignment (PVLA) task, which is to minimize the following \mathcal{L}_{\text{PVLA}} loss equal to the partial optimal transport (POT) distance \mathcal{D}_{\text{POT}} between \boldsymbol{\mu} and \boldsymbol{\nu} as: \displaystyle\mathcal{L}_{\text{PVLA}}=\mathcal{D}_{\text{POT}}(\boldsymbol{\mu},\boldsymbol{\nu})=\min_{\mathbf{T}\in\Pi(\mathbf{a},\mathbf{b})}\sum\limits_{i=1}^{N_{V}}\sum\limits_{j=1}^{N_{L}}\mathbf{T}_{i,j}\cdot c\left(\mathbf{h}_{v,i},\mathbf{h}_{l,j}\right), (22) \displaystyle\begin{split}\text{s.t}\quad\Pi(\mathbf{a},\mathbf{b})=\{\mathbf{T}\in\mathbb{R}^{N_{V}\times N_{L}}_{+}\mid\mathbf{T}\mathbf{1}_{N_{L}}\leq\mathbf{a},\mathbf{T}^{\top}\mathbf{1}_{N_{V}}\leq\mathbf{b},\\ \mathbf{1}_{N_{V}}^{\top}\cdot\mathbf{T}\cdot\mathbf{1}_{N_{L}}=s,\quad 0\leq s\leq\min(N_{L},N_{V})\}.\end{split} (23) Because the exact minimization over the transport plan \mathbf{T} is intractable, we adopt the Sinkhorn-based algorithm to compute \mathbf{T} . We explicate our algorithm to calculate the partial video-language alignment loss in Algorithm 2 . Our PVLA formulation is flexible where it allows only s samples from one distribution to be transported to the other, and enables the algorithm to decide the value of s , in case the input language only corresponds to certain video aspects (or vice versa). Training Strategy. For training, we jointly optimize the video-language task-specific loss and our PVLA loss. It is worth noting that we only update our introduced READ layers while keeping the remaining components frozen. 16 Experiments Algorithm 2 Computing the PVLA loss \mathbf{C}=\{\mathbf{C}_{i,j}=c\left(\mathbf{h}_{v,i},\mathbf{h}_{l,j}\right)\mid 1\leq i\leq N_{V},1\leq j\leq N_{L}\}\in\mathbb{R}^{N_{V}\times N_{L}} ,  temperature \tau,\;\mathbf{a}\in\mathbb{R}^{N_{V}},\;\mathbf{b}\in\mathbb{R}^{N_{L}},\;s,\;N_{\text{iter}} \mathcal{L}_{\text{PVLA}}=\infty for s=1 to \min(N_{L},N_{V}) do \mathbf{T}=\text{exp}\left(-\frac{\mathbf{C}}{\tau}\right) \mathbf{T}=\frac{s}{\left(\mathbf{1}_{N_{V}}\right)^{\top}\cdot\mathbf{T}\cdot\mathbf{1}_{N_{L}}}\mathbf{T} for i=1 to N_{\text{iter}} do \mathbf{p}_{a}=\min\left(\frac{\mathbf{a}}{\mathbf{T}\mathbf{1}_{N_{L}}},\mathbf{1}_{N_{V}}\right) , \mathbf{T}_{a}=\text{diag}\left(\mathbf{p}_{a}\right)\cdot\mathbf{T} \mathbf{p}_{b}=\min\left(\frac{\mathbf{b}}{\mathbf{T}_{a}^{\top}\mathbf{1}_{N_{V}}},\mathbf{1}_{N_{L}}\right) , \mathbf{T}_{b}=\text{diag}\left(\mathbf{p}_{b}\right)\cdot\mathbf{T}_{a} \mathbf{T}=\frac{s}{\left(\mathbf{1}_{N_{V}}\right)^{\top}\cdot\mathbf{T}\cdot\mathbf{1}_{N_{L}}}\mathbf{T}_{b} end for \mathcal{L}_{\text{PVLA}}=\min\left(\mathcal{L}_{\text{PVLA}},\;\sum\limits_{i=1}^{N_{V}}\sum\limits_{j=1}^{N_{L}}\mathbf{T}_{i,j}\mathbf{C}_{i,j}\right) end for return \mathcal{L}_{\text{PVLA}} Method #params (M) Dog Gym Par. Ska. Ski. Sur. Avg. Full 283.97 (100%) 65.90 ‡ 75.20 ‡ 82.20 ‡ 71.80 ‡ 72.30 ‡ 81.15 ‡ 74.76 ‡ Bias 0.51 (0.18%) 46.23 ‡ 61.19 ‡ 56.73 ‡ 31.36 ‡ 61.14 ‡ 49.77 ‡ 51.07 ‡ Partial 38.75 (13.65%) 48.28 ‡ 63.26 ‡ 59.71 ‡ 32.66 ‡ 64.58 ‡ 56.22 ‡ 54.12 ‡ Proj 5e-4 (1.76e-4%) 57.05 ‡ 65.70 ‡ 63.03 ‡ 71.83 ‡ 65.45 ‡ 79.71 ‡ 67.13 ‡ LoRA 13.12 (4.62%) 60.97 ‡ 67.68 ‡ 72.53 ‡ 66.62 ‡ 71.24 ‡ 79.15 ‡ 69.70 ‡ Prompt 0.02 (0.01%) 48.28 ‡ 63.26 ‡ 59.71 ‡ 35.67 ‡ 35.67 ‡ 64.61 ‡ 46.87 ‡ Adapter 13.11 (4.62%) 62.89 ‡ 67.09 ‡ 74.56 ‡ 62.56 ‡ 68.10 ‡ 78.73 ‡ 68.98 ‡ READ 0.16 (0.06%) 67.65 78.05 83.25 72.40 72.98 82.36 76.12 Table 14 : TLG results on the YouTube Highlights dataset. We report the mean average precision (mAP) and the number of trainable parameters (#params). ‡ means the gain of READ is statistically significant at the 0.05 level. Method #params (M) VT VU GA MS PK PR FM BK BT DS Avg. Full 285.28 (100%) 84.17 ‡ 81.50 ‡ 88.20 ‡ 71.54 ‡ 81.40 ‡ 84.31 ‡ 72.30 ‡ 76.53 ‡ 78.86 ‡ 77.70 ‡ 79.65 ‡ Bias 0.25 (0.09%) 38.08 ‡ 69.62 ‡ 60.87 ‡ 31.25 ‡ 68.84 ‡ 51.71 ‡ 50.72 ‡ 65.38 ‡ 54.42 ‡ 59.05 ‡ 54.99 ‡ Partial 38.75 (13.58%) 57.27 ‡ 62.57 ‡ 58.08 ‡ 52.35 ‡ 61.58 ‡ 63.94 ‡ 50.82 ‡ 62.36 ‡ 58.05 ‡ 47.79 ‡ 57.48 ‡ Proj 5e-4 (1.75e-4%) 57.65 ‡ 65.80 ‡ 64.40 ‡ 55.57 ‡ 64.67 ‡ 67.07 ‡ 59.08 ‡ 74.70 ‡ 63.29 ‡ 49.48 ‡ 62.17 ‡ LoRA 13.28 (4.66%) 77.87 ‡ 77.01 ‡ 77.82 ‡ 66.38 ‡ 80.21 ‡ 82.23 ‡ 66.89 ‡ 72.31 ‡ 69.58 ‡ 72.09 ‡ 74.24 ‡ Prompt 0.02 (0.007%) 61.67 ‡ 71.98 ‡ 64.07 ‡ 35.54 ‡ 72.74 ‡ 48.70 ‡ 52.97 ‡ 67.59 ‡ 57.28 ‡ 38.60 ‡ 57.11 ‡ Adapter 13.29 (4.66%) 78.46 ‡ 76.38 ‡ 77.36 ‡ 67.12 ‡ 80.33 ‡ 82.51 ‡ 67.77 ‡ 71.71 ‡ 69.58 ‡ 71.24 ‡ 74.25 ‡ READ 0.14 (0.05%) 88.30 85.15 89.76 75.80 86.69 86.62 74.99 82.38 84.65 79.60 83.39 Table 15 : TLG results on the TVSum dataset. We report the mean average precision (mAP) and the number of trainable parameters (#params). ‡ means the gain of READ is statistically significant at the 0.05 level. Method #params (M) mAP Full 15.88 (100%) 36.14 ‡ Bias 0.06 (0.38%) 24.89 ‡ Partial 1.05 (6.61%) 26.37 ‡ Proj 7.31 (46.03%) 32.71 ‡ LoRA 0.19 (1.20%) 33.96 ‡ Prompt 0.04 (0.25%) 25.86 ‡ Adapter 0.20 (1.26%) 33.61 ‡ READ 0.19 (1.20%) 36.74 Table 16 : TLG results on the QVHighlights dataset. We report the mean average precision (mAP) and the number of trainable parameters (#params). ‡ means the gain of READ is statistically significant at the 0.05 level. We conduct extensive experiments to evaluate the effectiveness of our READ framework. We first describe the experimental settings, covering the downstream tasks, evaluation metrics, pre-trained backbones, baseline approaches, and implementation details. We then present the numerical results of our method with baseline models, then provide ablation study and thorough analysis to explore various configurations. Eventually, we perform qualitative assessments to further elucidate the behavior of our framework. 16.1 Experimental Settings Downstream tasks. We assess the effectiveness on the temporal language grounding and video-language summarization tasks. The corresponding datasets to each task are presented as follows: • Temporal Language Grounding (TLG): The TLG’s task is to localize temporal boundaries of the video frames that semantically relate to the language query. The evaluation is performed upon three datasets, i.e. YouTube Highlights ( 213 ) , TVSum ( 209 ) , and QVHighlights ( 118 ) . YouTube Highlights consists of 40 video-language training inputs for each of the 6 domains. TVSum comprises 10 domains, each of which possesses 5 video-language training inputs. The QVHighlights benchmark includes 7,218 language-annotated video segments for training, 1,550 for development, and 1,542 for testing. Following previous work on low-resource experiments ( 17 ) , we keep our training size at 700 samples, which is less than 10% of the full data for the QVHighlights dataset, while preserving the original splits on the TVSum and YouTube Highlights datasets. • Video-Language Summarization (VLG): Given a video-language input, the VLS’s target is to generate a summary which takes into account both video and language content ( 288 ) . We consider the How2 dataset ( 199 ) , from which we randomly draw 2,000 out of 73,993 samples for training, i.e. less than 3% of the full data, to simulate the low-resource settings, while maintaining 2,520 samples for validation, and 2,127 samples for testing. Evaluation metrics. For the TLG task, we follow previous works ( 118 ; 151 ) to use the mean average precision (mAP) metric. Regarding VLS, we utilize the ROUGE score, which is a popular metric for summarization ( 305 ; 288 ) . Method #params (M) R1 R2 RL Full 249.67 (100%) 35.72 ‡ 11.88 ‡ 30.00 ‡ Bias 0.20 (0.08%) 30.51 ‡ 8.20 ‡ 23.00 ‡ Partial 16.54 (6.62%) 31.55 ‡ 8.63 ‡ 13.65 ‡ Proj 38.60 (15.46%) 32.76 ‡ 9.11 ‡ 30.01 ‡ LoRA 1.19 (0.48%) 40.04 ‡ 20.36 ‡ 35.98 ‡ Prompt 0.05 (0.02%) 31.55 ‡ 8.63 ‡ 14.90 ‡ Adapter 1.20 (0.48%) 41.52 ‡ 20.75 ‡ 36.88 ‡ READ 1.17 (0.47%) 44.01 21.91 37.91 Table 17 : VLS results on the How2 dataset with the VG-BART model. We report the ROUGE-1, ROUGE-2, and ROUGE-L scores, with the number of trainable parameters (#params). ‡ means the gain of READ is statistically significant at the 0.05 level. Method #params (M) R1 R2 RL Full 333.16 (100%) 32.37 ‡ 8.07 ‡ 26.53 ‡ Bias 0.07 (0.02%) 27.03 ‡ 4.53 ‡ 19.51 ‡ Partial 16.52 (4.96%) 27.83 ‡ 4.92 ‡ 10.49 ‡ Proj 24.67 (7.40%) 29.16 ‡ 5.50 ‡ 26.76 ‡ LoRA 1.93 (0.58%) 36.63 ‡ 16.72 ‡ 32.77 ‡ Prompt 0.18 (0.05%) 28.33 ‡ 5.12 ‡ 11.75 ‡ Adapter 1.95 (0.59%) 37.83 ‡ 17.45 ‡ 33.87 ‡ READ 1.91 (0.57%) 40.12 18.71 34.42 Table 18 : VLS results on the How2 dataset with the VG-T5 model. We report the ROUGE-1, ROUGE-2, and ROUGE-L scores, with the number of trainable parameters (#params). ‡ means the gain of READ is statistically significant at the 0.05 level. Method mAP - YouTube Highlights R2 - How2 No VLA 73.80 18.22 VLA 74.41 20.01 PVLA 76.12 21.91 Table 19 : Partial video-language alignment (PVLA) ablation experiments on YouTube Highlights and How2. We color the settings we implement for our READ method. Video Language query POT distance AP Girl’s expression after getting proposal 27.17 90.28 A boy showing his arm after being stung at the beach 56.38 30.00 Table 20 : Case study on the temporal language grounding benchmark. We extract the POT distance between video and language of two inputs with different language queries and measure the respective AP performance change. Recurrent architecture #params - UMT (M) mAP - YouTube Highlights #params - VG-BART (M) R2 - How2 GRU 0.16 76.07 1.17 21.85 LSTM 0.16 76.08 1.17 21.87 RNN 0.16 76.12 1.17 21.91 Table 21 : Recurrent architecture ablation experiment on YouTube Highlights and How2. We color the settings we implement for our READ method. Distance method mAP - YouTube Highlights R2 - How2 AvgPool - Cosine 71.65 20.32 MaxPool - Cosine 74.37 21.36 AvgPool - L2 72.11 20.73 MaxPool - L2 74.39 21.02 Partial OT 76.12 21.91 Table 22 : Distance method ablation experiments on YouTube Highlights and How2. We color the settings we implement for our READ method. Pre-trained backbones. We adopt the Transformer encoder-decoder architecture ( 226 ) pre-trained with both supervised and self-supervised objectives. Specifically, for TLG, we use the unified multimodal transformer (UMT) ( 151 ) and Moment-DETR ( 118 ) models pre-trained upon the automatic speech recognition task. For VLS, we carry out the parameter-efficient adaptation on the generative vision-guided BART (VG-BART) and T5 (VG-T5) ( 288 ) pre-trained upon reconstruction and masked language modeling tasks ( 192 ) . Baseline methods. We compare our method with a comprehensive list of baseline approaches for efficient video-language transfer learning: • Full : update all parameters of the pre-trained backbone. • Partial : only update the last layers of the encoder and decoder in the Transformer model. • Bias ( 296 ) : only fine-tune the bias terms in the Transformer backbone. • Proj : fine-tune only the last linear projection layer in the Transformer. • LoRA ( 84 ) : solely fine-tune the decomposition matrices introduced to the linear weights of the Transformer model. • Prompt ( 91 ) : Append a sequence of learnable prompt tokens to both video and language inputs and only fine-tune the appended sequence. • Adapter ( 83 ) : update only the adaptation modules consisting of downprojection and up-projection layers inserted into the Transformer model. Implementation details. For the TLG task, we use the SlowFast ( 54 ) and video encoder of CLIP ( 191 ) to extract features every 2 seconds. For the VLS task, we use a 3D ResNeXt-101 model to extract a 2048-dimensional embedding for every 16 non-overlapping frames. Similar to previous works ( 83 ; 32 ) , to support training stability, we initialize the weights of the down-projection layer W_{\text{down}} with the Kaiming normal ( 75 ) method, whereas those of the up-projection W_{\text{up}} , recurrent layer RNN, and biases of our READ layers are configured with zero initialization. In our PVLA framework, we implement the cost distance c\left(\mathbf{h}_{v,i},\mathbf{h}_{l,j}\right) as the cosine distance c\left(\mathbf{h}_{v,i},\mathbf{h}_{l,j}\right)=1-\frac{\mathbf{h}_{v,i}\cdot\mathbf{h}_{l,j}}{||\mathbf{h}_{v,i}||_{2}\cdot||\mathbf{h}_{l,j}||_{2}} , and set the maximum number of iterations N_{\text{iter}} to 1,000 and the temperature \tau to 0.05. We fine-tune all models leveraging the AdamW optimizer on 4 NVIDIA Tesla V100 GPUs and report average results of 5 runs. Specific details about the epoch, batch size, learning rate, and the number of Transformer blocks for each task can be found in the Appendix. 16.2 Main Results For main comparison of our READ with baseline methods, we denote the results of YouTube Highlights in Table 14 , TVSum in Table 15 , QVHighlights in Table 16 , and How2 in Tables 17 and 18 . Temporal language grounding (TLG). For the YouTube Highlights dataset, our READ framework substantially outperforms the Full fine-tuning approach ( e.g. 1.36% on average, 1.75% in the Dog domain, and 1.21% in the Surfing domain), while updating far less parameters (0.16M vs. 283.97M). We significantly surpass all other efficient fine-tuning methods as well, e.g. with an improvement of 10.96% over the Adapter in the Gym category, or 5.78% over LoRA in the Skating category. For the TVSum dataset, we observe that our method enhances the Full fine-tuning direction with only 0.14M versus 285.28M tunable parameters. For instance, we obtain an increase of 4.26% in the MS subset and 3.74% on average. Compared with the best parameter-efficient approach, i.e. the Adapter, we achieve a gain of 15.07% in BT, 10.67% in BK, and 8.77% in the VU domain. Our improvement also generalizes across different pre-trained backbone. On the QVHighlights dataset, in which we work with the Moment-DETR architecture, we accomplish a gain of 0.6% over the standard fine-tuning method, while our tunable parameters are only 0.19M versus its 15.88M. We also surpass the efficient approach LoRA with an enhancement of 2.78% in mAP. These results demonstrate that our READ framework can efficiently model video-language inputs to polish the low-resource temporal language grounding performance of various pre-trained Transformer models. Video-language summarization (VLS). Analogous to the TLG experiment, on the VG-BART backbone, we improve upon the full fine-tuning approach with 8.29 points of ROUGE-1, 10.03 points of ROUGE-2, and 7.91 points of ROUGE-L. Importantly, we only update 1.17M parameters, which account for 0.47% total parameters of the overall model. On the VG-T5 backbone, we exceed the full approach by 7.75 points in ROUGE-1, 10.64 points in ROUGE-2, and 7.89 points in ROUGE-L, whilst keeping 99.43% parameters frozen. In addition, our framework substantially outperforms other fine-tuning strategies, e.g. LoRA with 3.97 points in ROUGE-1, 1.55 points in ROUGE-2, and 1.93 points in ROUGE-L on the VG-BART architecture, along with 3.49 points in ROUGE-1, 1.99 points in ROUGE-2, and 1.65 points in ROUGE-L on the VG-T5 one. These results substantiate that our method is applicable to diverse benchmarks and model architectures, particularly not only multimodal Transformers for temporal language grounding but also generative Transformers for video-language summarization. We hypothesize that our advantages are due to the recurrent adapter’s ability to model temporal information and the PVLA task to align video and language signals to maintain more essential information during the efficient fine-tuning stage. 16.3 Ablation Studies We ablate our READ framework to discover what factors result in the demonstrated efficiency and observe several intriguing properties. Our ablation studies are all conducted on the YouTube Highlights and How2 test set. Effects of video-language alignment. We evaluate our framework without the assistance of the PVLA task and with the one of the VLA variant that requires all masses of one distribution to be transferred (we set s=\min\left(N_{V},N_{L}\right) in formulation ( 56 )). As shown in Table 19 , the performance drops dramatically when we remove the PVLA task from the fine-tuning procedure. We conjecture that the model has become deficient in managing the information injected into the low-dimensional space of the READ layers, thus passing detrimental noise to the downstream task. Moreover, the VLA variant brings slight performance decrease, which could be due to the VLA’s restrictive nature of transporting all masses from the language distribution to the video one or vice versa. Effects of the recurrent architecture. In addition to RNN, there exist various recurrent architectures in the literature, particularly the gated recurrent unit (GRU) ( 39 ) and long short-term memory (LSTM) ( 80 ) . We experiment with different recurrent choices and explicate the results in Table 21 . As can be observed, the performance is insensitive to the choice of recurrent design. Therefore, we select the simplest option, i.e. recurrent neural network (RNN) for our READ layers. Distance methods. We further ablate on the distance metrics to estimate the distance between video and language distributions. Technically, we perform the average- and max-pooling of the video and language representations. Then, we consider the cosine distance or the L2 distance of the two pooled vectors as the video-language distance. Results in Table 22 substantiate the superiority of our POT distance for the PVLA objective. Such success illustrates the POT-based PVLA’s advantage of modeling the relationship nature between video and language representations 16.4 Qualitative Assessment Case study. We display a TLG example on the YouTube Highlights dataset, along with the POT distance estimated by our PVLA framework and the AP score in Table 20 . We observe that when the language query semantically corresponds to a moment in the video, i.e. a girl expression after she gets the proposal , the POT distance is small and correlates with the high value of AP. In contrast, when we replace the original query with an out-of-distribution one, the POT distance burgeons significantly, causing the AP to decrease from 90.28% to 30.00%. Therefore, we conclude that our READ framework is capable of intelligently adjusting the information flowing through the READ layers in order to produce the final output consistent with the video-language input and downstream tasks. 17 Conclusion We propose a novel READ-PVLA framework for parameter-efficient transfer learning to video-language modeling tasks. Our READ-PVLA utilizes recurrent computation component to enable temporal modeling capability and partial video-language alignment objective to preserve critical information for bottleneck adaptation modules. Experiments demonstrate that READ-PVLA consistently outperforms both the full fine-tuning and competitive strategies, whilst bringing the benefit of parameter-efficiency (at most 1.20% trainable parameters). Our method is also applicable to diverse pre-trained models, which has the potential to employ more powerful video-language models in the future. Chapter 5 Encoding and Controlling Global Semantics for Long-form Video Question Answering 18 Introduction VideoQA has been extensively studied to develop systems to assist humans in daily activities ( 66 ; 124 ) , e.g. , remind users of their past actions, help users locate their belongings, and provide assistance with complex tasks. To implement these functions, we expect videoQA systems to understand and extract relevant information from long-form videos with diverse objects and complex spatial-temporal interactions. Figure 15 : Long-form videoQA examples, with videos taken from MAD ( 207 ) and Ego4D ( 66 ) datasets, respectively. Question in video 1 requires the model to reason about the relation chain of replacing ingot of palladium to activate the rt unit that powers the armored suit and protects person’s health . Question in video 2 necessitates an understanding of the overall theme in video 2. Compared with short clips, long-form videos pose more challenges for videoQA. They consist of a higher number of objects and events. As such, comprehensively encoding information from them requires expensive computations. Moreover, a high amount of information may be unrelated to the posed question. To address these problems, recent studies ( 11 ; 231 ; 60 ) adaptively select a subset of video frames and visual regions associated with the question. Nevertheless, if a question necessitates a reasoning of the entire sequence of events ( e.g. video 1’s Figure 15 ), or an understanding of the overall video narration ( e.g. video 2’s Figure 15 ), a mere handful of selected frames or regions might not sufficiently encapsulate necessary details. To tackle these problems, we introduce a state space layer (SSL). Before forwarding video frames to selection modules, SSL fixes long-term dependency patterns for integrating global information into visual representations. Such global information offers the selected frames the global context within the video, so that they can relate to other frames even though those frames are not selected for attention computation. However, a considerable amount of unrelated global information may flow into visual representations. Therefore, we first equip SSL with a gating mechanism to provide more controllability over the flow of global semantics into visual representations, resulting in our G ated S tate space M ulti-modal T ransformer (GSMT) architecture. Furthermore, we promote global semantics that is more aligned with the question. In particular, we introduce C ross-modal C ompositional C ongruence (C 3 ) objective that compares visual attention with its version transitioned to the language basis via cross-modal attention, effectively measuring cross-modal congruence between intra-modal relations. Our rationale behind focusing on intra-modal relations is because videoQA models often need to understand spatial and temporal relationships between entities and events posed by the question ( 58 ) , thus we encourage globally informed visual representations to maintain compositional consistency between visual patches and question entities. Remarkably, we observe that recent long-form videoQA works ( 60 ; 87 ) still mostly evaluate on videos lasting at most one minute or two, and use short-natured questions which necessitate watching only a short period of video, i.e. , about 100 seconds ( 162 ) , to determine the answer. To more rigorously evaluate long-form videoQA capacity, we introduce a construction procedure which utilizes large language model (LLM) to generate questions and associated answers for egocentric and movie videos whose average lengths are 17.5 minutes and 1.9 hours, respectively. Additionally, we also conduct automatic and manual filtering to obtain high-quality questions which require watching a video up to 1200 seconds to answer, longer than any existing long-form videoQA benchmarks ( 257 ; 248 ; 162 ) . To sum up, our contributions are as follows: • We propose a Gated State space Multi-modal Transformer (GSMT) with state space layer (SSL) to integrate global information into visual representations for long-form videoQA. • We equip SSL with a gating mechanism to provide controllability over the flow of global video semantics and a Cross-modal Compositional Congruence (C 3 ) objective to encourage question-aligned visual representations. • We curate two new datasets with excessively long video lengths and long-natured questions for long-form videoQA. Comprehensive experiments on our curated and five standard datasets substantiate the superiority of our framework over various competitive baselines. 19 Methodology Figure 16 : Illustration of the GSMT architecture empowered by gated SSL and C 3 training objective. The formulation of video question answering (videoQA) is to predict the answer y for a question q about a video \mathcal{V} as follows: \tilde{y}=\argmax_{y\in\mathcal{A}}\mathcal{F}_{\theta}(y|q,\mathcal{V},\mathcal{A}), (24) where \tilde{y} denotes the predicted answer chosen from the set of candidate options \mathcal{A} , and \theta denotes the trainable parameters of a videoQA model. With this videoQA task formulation, we explain our proposed GSMT architecture as the model \mathcal{F}_{\theta} and C 3 objective to support GSMT. The overall framework is illustrated in Figure 16 . 19.1 Gated State Space Multi-Modal Transformer (GSMT) Our GSMT takes videos and questions as input, divides each video frame into visual patches and a question into textual words, and then forwards visual patches and textual words into video and text embedder to extract initial representations. Input Embedder. For video embedder, we utilize frozen pre-trained vision-language Transformer to extract patch-level features X=\{\mathbf{x}_{0},\mathbf{x}_{1},...,\mathbf{x}_{L-1}\} , of all T frames, where t -th frame consists of N patch-level features \{\mathbf{x}_{t,j}\}_{j=0}^{N-1} , hence L=NT , and \mathbf{x}\in\mathbb{R}^{d} . For text embedder, a similar frozen pre-trained vision-language Transformer is used to obtain word-level features W=\{\mathbf{w}_{0},\mathbf{w}_{1},...,\mathbf{w}_{M-1}\} , where \mathbf{w}_{0} corresponds to the [CLS] token and \mathbf{w}_{1},...,\mathbf{w}_{M-1} correspond to words in the question. Gated State Space Layer (Gated SSL). Inspired by ( 68 ) , we define a sequence-to-sequence map SSL(X) from a sequence of patch-level features to d_{S} -dim hidden states, parameterized by learnable state matrices A\in\mathbb{R}^{d_{S}\times d},B\in\mathbb{R}^{d_{S}\times d},C\in\mathbb{R}^{d\times d_{S}} , and step size \Delta as: \displaystyle\mathbf{g}_{t+1}=\overline{A}\cdot\mathbf{g}_{t}+\overline{B}\cdot\mathbf{x}_{t+1},\quad\mathbf{o}_{t+1}=\overline{C}\cdot\mathbf{g}_{t+1}, (25) \displaystyle\overline{A}=e^{A\Delta},\quad\overline{B}=(\overline{A}-I)A^{-1}B,\quad\overline{C}=C. (26) We unroll the mapping to obtain: \mathbf{o}_{t}=\sum\limits_{j=0}^{t}\overline{CA}^{j}\overline{B}\cdot\mathbf{x}_{t-j}. (27) This can be written as a convolutional representation O=\overline{\Gamma}*X , where \overline{\Gamma}=\left(\overline{CB},\overline{CAB},...,\overline{CA}^{L-1}\overline{B}\right) denotes the convolutional kernel, * the discrete convolution operator, X the input sequence, O the corresponding output sequence. This convolution denotes the fixed global dependency pattern that facilitates the computation of global information among visual patches. We use Fast Fourier Transformer (FFT) ( 41 ) to compute the convolution in parallel provided that \bar{\Gamma} has been obtained. Computing the kernel \bar{\Gamma} is non-trivial since it requires L distinct matrix powers. Instead, inspired by ( 70 ) , we initialize A to be a diagonal matrix \text{diag}\left(\lambda_{1},\lambda_{2},...,\lambda_{d_{S}}\right) and B to all-one matrix \mathbbm{1}_{d_{S}\times d} . Due to this initialization, the kernel can be computed as: \overline{\Gamma}=\left(C\odot E\right)\cdot P, (28) where E=\left(\frac{e^{\lambda_{1}\Delta-1}}{\lambda_{1}},\frac{e^{\lambda_{2}\Delta-1}}{\lambda_{2}},...,\frac{e^{\lambda_{d_{S}}\Delta-1}}{\lambda_{d_{S}}}\right)\in\mathbb{R}^{d_{S}} , in which P_{i,j}=e^{\lambda_{i}\cdot j\cdot\Delta}\in\mathbb{R}^{d_{S}\times L} , and \odot denotes the element-wise multiplication. To equip SSL with the control over which global semantics to integrate into visual representations, we construct a gating unit as: \displaystyle U=\phi\left(\text{Linear}(X)\right),\;\;V=\phi\left(\text{Linear}(X)\right), (29) \displaystyle O=\text{Linear}\left(\text{SSM}\left(U\right)\right), (30) \displaystyle H=\text{Linear}\left(O\odot V\right), (31) where U,V,O\in\mathbb{R}^{L\times d_{\text{gating}}} denote the intermediate gating representations, H\in\mathbb{R}^{L\times d_{h}} the visual representations output by gated SSL, \phi the non-linear activation, and d_{\text{gating}}