SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning
Fida Mohammad Thoker, Letian Jiang, Chen Zhao, Piyush Bagad, Hazel Doughty, Bernard Ghanem, Cees G. M. Snoek · 2025
Read SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning by Fida Mohammad Thoker, Letian Jiang, Chen Zhao, Piyush Bagad, Hazel Doughty, Bernard Ghanem, Cees G. M. Snoek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05706.txt.
Introduction
Fida Mohammad Thoker and co-authors stress-test how well self-taught video models travel beyond their training pond, in this 2025 paper. Video models that learn without labels shine on standard action tests, but those tests use near-identical training and scoring sets. The team broadens an earlier sensitivity study to modern transformer models, both video-only and video-text, plus older convolutional rivals. Over eleven hundred runs span eight datasets and seven tasks.
Four pressures are applied: new domains, fewer training samples, finer action classes, and new task types. No method held up across all four. Video-only transformers coped best with domain shift, plain convolutional nets won on fine-grained actions, and video-text models often lagged despite giant pretraining. Newer transformers did not reliably beat older ones.
Catalog record
| CREATOR | Fida Mohammad Thoker, Letian Jiang, Chen Zhao, Piyush Bagad, Hazel Doughty, Bernard Ghanem, Cees G. M. Snoek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05706
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning Fida Mohammad Thoker, Letian Jiang, Chen Zhao, Piyush Bagad, Hazel Doughty, Bernard Ghanem, Cees G. M. Snoek Abstract Continued advances in self-supervised learning have led to significant progress in video representation learning, offering a scalable alternative to supervised approaches by eliminating the need for manual annotations. Despite strong performance on standard action recognition benchmarks, existing video self-supervised learning methods are predominantly evaluated within narrow protocols—typically pre-training on Kinetics-400 and finetuning on similar datasets—limiting our understanding of their generalization capabilities in real-world settings. In this work, we present a comprehensive evaluation of modern video self-supervised learning models, focusing on generalization across four key downstream factors: domain shift, sample efficiency, action granularity, and task diversity. Building on our prior work analyzing benchmark sensitivity in CNN-based contrastive learning, we extend the study to cover current state-of-the-art transformer-based video-only and video-text representation models. Specifically, we benchmark 12 transformer-based methods (7 video-only, 5 video-text) and compare them against 10 CNN-based methods, resulting in over 1100 experiments across 8 datasets and 7 downstream tasks. Our analysis reveals that, despite architectural advancements, transformer-based models remain sensitive to downstream conditions. No single method generalizes consistently across all factors; for instance, video-only transformers are more robust to domain shift, CNN-based models perform better on tasks requiring fine-grained temporal reasoning, and video-text transformers underperform both in several downstream settings despite large-scale pre-training. We also observe that recent transformer-based approaches do not universally outperform earlier methods. These findings provide a detailed understanding of the capabilities and limitations of current video self-supervised learning approaches and establish an extended benchmark for evaluating generalization in video representation learning. Our benchmark offers a unified protocol for future research aimed at developing robust, transferable video models. Code will be made available at https://github.com/fmthoker/SEVERE-BENCHMARK-plus-plus . keywords Self-supervised learning, Video self-supervised Learning, Video-text Pre-training 1 Introduction Video self-supervised learning has progressed at a tremendous pace in recent years, e . g ., Wang et al. (2021) ; Afouras et al. (2020) ; Qian et al. (2021) ; Piergiovanni et al. (2020) ; Peihao et al. (2021) ; Patrick et al. (2021) ; Tong et al. (2022) ; Thoker et al. (2023) ; Tong et al. (2022) ; Fan et al. (2023) ; Wang et al. (2023) ; Huang et al. (2023) ; Sun et al. (2023) ; Hwang et al. () ; Salehi et al. (2025) ; Bardes et al. (2024) , offering a crucial starting point for learning rich, generalizable video representations without relying on manual annotations. This is especially important in the context of video understanding, where large-scale datasets are essential for training, but annotating such datasets is both costly and time-consuming. At the same time, vast amounts of unlabeled video data are readily available online, presenting an untapped resource for learning meaningful representations. Beyond cost and scalability concerns, human annotations are susceptible to errors and inconsistencies, especially in tasks involving subtle motion, ambiguous actions, or domain-specific activities. These challenges introduce annotator bias Haliburton et al. (2024) ; Lampert et al. (2016) and reduce label quality, limiting the effectiveness of supervised learning. In contrast, self-supervised learning (SSL) leverages the inherent structure and redundancy in videos — such as temporal continuity, cross-modal correlations, and motion cues — to define pretext tasks that guide representation learning without explicit labels. Learning strong video representations without labels is vital in real-world applications where annotated data is scarce or hard to obtain, such as in medical videos, surveillance footage, scientific research domains, or niche industries. Furthermore, downstream tasks may involve new or rare action categories, unusual environments, or non-standard viewpoints that are not well-represented in existing labeled datasets. In these cases, SSL enables models to learn transferable and robust features that can generalize across tasks and domains. Ultimately, self-supervised learning unlocks the potential to scale video understanding to massive, diverse datasets while reducing reliance on costly annotations, making it an essential paradigm for advancing the field. Despite rapid advancements in video self-supervised learning, the majority of the methods, e . g . Xu et al. (2019) ; Misra et al. (2016) ; Morgado et al. (2021) ; Asano et al. (2020) ; Pan et al. (2021) ; Tong et al. (2022) ; Fan et al. (2023) ; Sun et al. (2023) ; Hwang et al. () ; Huang et al. (2023) , remain evaluated with a narrow scope. The standard benchmarking protocol for many years Xu et al. (2019) ; Misra et al. (2016) ; Morgado et al. (2021) ; Asano et al. (2020) ; Pan et al. (2021) was pre-training on the unlabeled Kinetics-400 dataset Kay et al. (2017) and evaluating by finetuning on UCF-101 Soomro et al. (2012) and HMDB-51 Kuehne et al. (2011) . More recently Tong et al. (2022) ; Fan et al. (2023) ; Sun et al. (2023) ; Hwang et al. () ; Huang et al. (2023) , the evaluation has changed to finetuning on Kinetics-400 Kay et al. (2017) and Something-Something-v2 Goyal et al. (2017) . Although these benchmarks have driven significant advancements in video self-supervised learning, they provide a limited perspective on generalizability. The pre-training and downstream datasets share similar visual characteristics and action types, making it unclear how well these methods transfer to more diverse or challenging real-world scenarios. The narrow scope of current benchmarks poses a critical limitation: it fails to reflect the diverse real-world conditions where robust video representations are most needed, such as domain-shift, egocentric perspectives, scenarios with limited labeled finetuning data, fine-grained interactions, or complex tasks beyond standard classification. Recognizing this gap, some works have begun to evaluate on additional datasets such as Dave et al. (2022) ; Wang et al. (2021) ; Thoker et al. (2023) ; Fan et al. (2023) on Diving-48 Li et al. (2018) , Xiao et al. (2021) ; Yang et al. (2020) ; Feichtenhofer et al. (2021) ; Tong et al. (2022) ; Wang et al. (2023) on AVA Gu et al. (2018) and Yang et al. (2020) on EPIC-Kitchens-100 Damen et al. (2021) . However, these evaluations are often ad hoc, typically involving only one additional dataset and frequently without standardized baselines or comparisons to other approaches. Some works have tried to address this problem for image self-supervised learning by performing benchmarking studies Cole et al. (2022) ; Islam et al. (2021) ; Kotar et al. (2021) ; Ericsson et al. (2021) ; Goyal et al. (2019) ; Yang et al. (2020) ; Kolesnikov et al. (2019) ; Zhai et al. (2019) ; Asano et al. (2020) ; Newell and Deng (2020) ; Sariyildiz et al. (2021) ; Van Horn et al. (2021) ; Ericsson et al. (2021) to investigate model transferability Islam et al. (2021) ; Ericsson et al. (2021) ; Newell and Deng (2020) ; Wallace and Hariharan (2020) or the importance of factors like pre-training dataset Cole et al. (2022) ; Kotar et al. (2021) ; Goyal et al. (2019) and backbone architecture Kolesnikov et al. (2019) . Unfortunately, lessons from these works do not directly transfer to video self-supervised learning. First, video self-supervised tasks are distinct from those of images as they are designed to understand the temporal dimension of video Peihao et al. (2021) ; Dave