VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova · 2025
Read VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models by Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03970.txt.
Introduction
Dahun Kim, AJ Piergiovanni, Ganesh Mallya, and Anelia Angelova build a test for whether video models truly follow what happens, moment by moment, in a 2025 paper. Many benchmarks use single-event clips, which cannot expose a model that mixes up the order of actions or swaps one action for another.
Their benchmark stitches multi-event videos with captions for each stretch. Trap answers shuffle the order, swap action words, or cover only part of the action, so lazy models fail. A new training loss rewards exact timing and punishes each grade of disruption in steps. To ease the shortage of long labeled video, they also fake long sequences by joining short clips. Top models still stumbled, marking room to grow.
Catalog record
| CREATOR | Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova Abstract We introduce VideoComp, a benchmark and learning framework for advancing video-text compositionality understanding, aimed at improving vision-language models (VLMs) in fine-grained temporal alignment. Unlike existing benchmarks focused on static image-text compositionality or isolated single-event videos, our benchmark targets alignment in continuous multi-event videos. Leveraging video-text datasets with temporally localized event captions ( e.g. ActivityNet-Captions, YouCook2), we construct two compositional benchmarks, ActivityNet-Comp and YouCook2-Comp. We create challenging negative samples with subtle temporal disruptions such as reordering, action word replacement, partial captioning, and combined disruptions. These benchmarks comprehensively test models’ compositional sensitivity across extended, cohesive video-text sequences. To improve model performance, we propose a hierarchical pairwise preference loss that strengthens alignment with temporally accurate pairs and gradually penalizes increasingly disrupted ones, encouraging fine-grained compositional learning. To mitigate the limited availability of densely annotated video data, we introduce a pretraining strategy that concatenates short video-caption pairs to simulate multi-event sequences. We evaluate video-text foundational models and large multimodal models (LMMs) on our benchmark, identifying both strengths and areas for improvement in compositionality. Overall, our work provides a comprehensive framework for evaluating and enhancing model capabilities in achieving fine-grained, temporally coherent video-text alignment. Dataset available at: https://github.com/google-deepmind/video_comp . Figure 1 : Illustration of our video-text compositionality benchmark, introducing challenging disruptions for fine-grained alignment. Starting from a video and its positive text description (top row), we apply subtle disruptions including temporal reordering (purple), action word replacement (orange), and segment-level mismatch (yellow text matched with blue video crop) for evaluation, along with combined disruptions used during training. These perturbations test the model’s ability to distinguish coherent video-text pairs from disrupted ones. 1 Introduction Compositionality, the capacity to understand and integrate attributes, relations, and entity states, is essential for vision-language models (VLMs) to achieve a comprehensive understanding of complex scenes. While traditional benchmarks largely focus on compositionality in static image-text pairs, extending this to video data requires models to capture both temporal and structural relationships within event sequences. In video compositionality, models must align temporally ordered and semantically rich video segments with text, discerning evolving relationships among entities within and across scenes. Although video-text understanding has gained attention, few benchmarks specifically address the dynamic, fine-grained temporal compositionality needed to process multi-event coherence in video. While compositional reasoning has been explored in image-text models, it has primarily focused on static scenes using contrastive losses and negative samples from text augmentations or alternative image sampling [ 48 , 6 ] . These methods lack the understanding of temporal complexity required for video contexts. Recently, benchmarks such as PerceptionTest [ 30 ] , VITATECS [ 20 ] , ViLMA [ 14 ] , and ICSVR [ 26 ] have extended compositional evaluations to video, introducing counterfactual and concept-based tests. Yet, these benchmarks largely focus on isolated, single-event scenarios, which do not fully capture the continuous, multi-event coherence needed for real-world video understanding. Addressing this gap, our benchmark introduces multi-event video-text sequences with nuanced disruptions such as video segment cropping, action word replacements, and subtle temporal reordering to evaluate models’ ability to capture compositional alignment across cohesive video-text sequences. In this work, we study video-text compositionality and temporal understanding by introducing a benchmark and learning framework that extends VLM capabilities. We base our benchmark on dense video captioning datasets like ActivityNet-Captions [ 15 ] and YouCook2 [ 53 ] , which offer temporally localized captions for multiple events within each video. Using these datasets, we create structured modifications to evaluate video-text compositionality. For each video, we first obtain a positive sample by arranging event captions chronologically into a single, cohesive paragraph. To study compositional sensitivity, we then generate challenging negative samples with subtle disruptions, including temporal reordering (shuffling captions out of sequence), action word replacements (altering key verbs to shift meaning), segment-based misalignments (pairing partial video segments with mismatched captions), and combined disruptions. These nuanced modifications present models with difficult distinctions between coherent and disrupted sequences, particularly given the longer, interdependent annotations in our benchmark. To improve model performance on video-text compositionality, we introduce a hierarchical pairwise preference loss alongside the standard InfoNCE objective. This preference loss guides models to prioritize similarity for temporally accurate pairs, while progressively reducing similarity scores for increasingly disrupted pairs. This approach enhances alignment with accurately composed samples and strengthens the model’s ability to recognize compositionality at varying levels of granularity. One challenge in training for video-text compositionality is the limited availability of densely captioned, multi-event video datasets. To address this, we implement a pretraining strategy that leverages video-short text datasets. By concatenating short video clips and captions to simulate multi-event sequences, this pretraining method applies the preference loss to these pseudo-long-form pairs, enabling the model to develop a more robust understanding of temporal and compositional structure, even with limited dense-captioned data. This comprehensive approach allows models to better capture fine-grained, temporally coherent video-text alignment. Finally, we evaluate large multimodal models (LMMs) on our benchmark. This evaluation provides insights into the current capabilities of LMMs in handling video-text compositionality, particularly in distinguishing temporally accurate alignments from corrupted ones. The contributions of this work are as follows: • We introduce a video-text compositionality benchmark featuring complex negative samples that disrupt the temporal alignment, providing a rigorous test for VLMs to capture fine-grained compositional structures. • We propose a hierarchical pairwise preference loss that adjusts video-text similarity scores based on levels of disruption, enhancing fine-grained alignment between video and text. • To facilitate video compositionality learning and address the scarcity of densely captioned video data, we explore a pretraining strategy that concatenates short video-caption pairs to simulate long-text data, promoting stronger temporal compositionality learning. • Our evaluation of large multimodal models reveals specific areas for improvement in compositional alignment. 2 Related Work 2.1 Video-Text Alignment and Understanding Video-text alignment models [ 16 , 2 , 38 , 22 , 39 , 5 , 7 ] are fundamental for tasks like retrieval, captioning, and question answering. Dual-encoder models [ 43 , 22 , 37 ] , which adapt the CLIP [ 31 ] framework for video by incorporating temporal elements, are efficient for large datasets and perform well in video-text retrieval tasks. However, they often lack the fine-grained temporal and compositional understanding needed for complex, multi-event sequences. While cross-attention models [ 45 , 18 , 23 ] can capture more detailed interactions between video frames and text tokens, they typically involve higher computational costs. Commonly used datasets, such as MSR-VTT [ 44 ] , MSVD [ 4 ] , LSMDC [ 33 ] , WebVid [ 3 ] , InternVid [ 41 ] and VideoCC3M [ 27 ] , support high-level alignment tasks but generally focus on short, single-event captions, limiting their effectiveness in evaluating temporal coherence within extended, multi-event contexts. Several works [ 49 , 25 , 9 ] have addressed multi-event retrieval by linking video events with specific text descriptions, focusing mainly on event identification rather than fine-grained temporal coherence within events. Our benchmark extends these efforts, using dense captioning datasets with multi-event narratives to evaluate compositional coherence and temporal alignment in video-text models. By introducing targeted disruptions, we provide a comprehensive assessment of models’ capabilities in maintaining