Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman · 2026
Read Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models by Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19137.txt.
Introduction
Svetlana Orlova and two colleagues ask whether video models can be built by reusing a frozen image foundation model and training only a temporal module. Video foundation models usually need massive video pretraining, while image models already offer spatial representations. The authors freeze a pretrained image model as a spatial encoder and train only a recurrent temporal module for streaming video. Across several video understanding tasks, temporal performance emerges without large-scale video pretraining, suggesting future recurrent video models could be built this cheaper way.
Catalog record
| CREATOR | Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47222 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman Abstract Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen . 1 Introduction Figure 1 : Video Foundation Model vs . Image Foundation Model + Recurrent Head. Comparison of a frozen Video Foundation Model (RVM Zoran et al. 2025 ) vs . a frozen Image Foundation Model (DINOv3 Siméoni et al. 2025 ) with a fine-tuned recurrent temporal head, GatedMambaMix (GMMix). DINOv3 achieves similar performance across different tasks without large scale video pre-training. Recent years have seen rapid progress in video foundation models, which aim to learn general-purpose representations for a wide range of video understanding tasks Tong et al. 2022 ; Bardes et al. 2024 ; Carreira et al. 2024 ; Zoran et al. 2025 . Most competitive models adopt large architectures that are massively pre-trained end-to-end on extremely large video datasets, often comprising millions to billions of clips. While this pre-training is required for strong performance, it comes with substantial costs in terms of data collection, storage, and computational resources. At the same time, image foundation models have reached an unprecedented level of capability. Trained on billions of images, these models provide powerful spatial representations that transfer well across tasks and domains Oquab et al. 2023 ; Siméoni et al. 2025 ; Tschannen et al. 2025 . This raises an important question: to what extent is large-scale video pre-training actually necessary if strong spatial representations are readily available? A promising alternative is to leverage a pre-trained image backbone and focus the more computationally intensive video training exclusively on temporal modeling. Instead of learning spatial and temporal representations jointly from scratch on video data, a model could inherit spatial knowledge from an image foundation model and learn temporal reasoning via video pre-training of a temporal module processing the image foundation model’s representations. Such a strategy could dramatically reduce both the data and compute requirements needed to develop powerful video models. In this work, we explore this idea in the context of recurrent video models. We do not yet perform video pre-training of the temporal module, but, in this work, we explore the feasibility of this approach before making the required compute investment. For this, we aim to address two research questions: (1) is image pre-training of the spatial encoder competitive with video pre-training?, and (2) do we actually need large-scale video pre-training for the temporal module? To answer these questions, we conduct experiments using different image foundation models and temporal architectures. Our evaluation spans several representative tasks: action recognition (Something-Something v2 Goyal et al. 2017 ), object tracking (Waymo Open Sun et al. 2020 ), point tracking (Perception Test Patraucean et al. 2023 ), depth estimation (ScanNet Dai et al. 2017 ), and camera pose estimation (NuScenes Caesar et al. 2020 ), allowing us to comprehensively assess whether strong temporal reasoning can emerge without large-scale video pre-training. The results show that spatial representations obtained from frozen image foundation models are stronger than those obtained with video pre-training, but at the same time, the results also indicate that video pre-training of the temporal module (not yet done in this work) is likely needed to surpass current SotA video foundation models across all settings and tasks. This paper should therefore be viewed as a work in progress toward a new pre-training paradigm for recurrent video models. Our goal is to provide initial empirical evidence and insights that inform future efforts to scale this approach into a full video foundation model. We make the following contributions: 1. Empirical study of temporal learning with frozen spatial representations. Across multiple image encoders, temporal architectures, and diverse video tasks, we demonstrate that effective temporal reasoning emerges even with a fixed image-level backbone. 2. Evidence towards data-efficient video pre-training. Our results indicate that modern image foundation models already contain much of the spatial capacity needed for video understanding, thereby supporting a future paradigm in which only the temporal module requires video pre‑training. 1.1 Small Data Statement The end goal of our line of research is to reduce the data and computational requirements for pre-training recurrent video foundation models. We explore decoupling the pre-training of the spatial (image) encoder from that of the temporal module. By using an off-the-shelf image foundation model as a spatial encoder, we hypothesize that less video data is needed to pre-train the temporal module. This work reports our current findings, which do not yet include temporal pre-training, but it verifies that our paradigm is promising and that investing in temporal pre-training is a promising direction for obtaining more data-efficient pre-training strategies for recurrent video foundation models. 2 Related Work 2.1 Image Foundation Models Vision Transformers Dosovitskiy et al. 2020 trained with self-supervised objectives have become the dominant paradigm for visual representation learning. Contrastive methods such as CLIP Radford et al. 2021 learn aligned image-text representations, while masked autoencoders He et al. 2022 learn directly from pixels via reconstruction. Self-distillation approaches have proven particularly effective as frozen feature extractors: DINOv2 Oquab et al. 2023 produces general-purpose features that achieve strong results across classification, segmentation, and depth estimation with only task-specific heads trained on top. DINOv3 Siméoni et al. 2025 extends this further, establishing a single frozen ViT as a universal vision backbone that sets state-of-the-art results on multiple vision tasks, without any task-specific fine-tuning of the encoder. We use such image foundation models as frozen spatial feature extractors and investigate whether adding a learned temporal module on top is sufficient to match video foundation models. 2.2 Video Foundation Models Video Vision Transformers. The dominant approach to video modeling extends ViT by tokenizing video clips into spatio-temporal patches and using pre-training with self-supervised objectives on large-scale video data. VideoMAE Tong et al. 2022 applies tube masking to spatio-temporal patches, V-JEPA Bardes et al. 2024 predicts masked representations in latent space, and 4DS Carreira et al. 2024 simplifies the objective and focuses on scaling, showing consistent improvement on geometric and temporal tasks up to 22B parameters. These architectures process fixed-length clips and are inherently non-causal, requiring the entire temporal window upfront. All are trained end-to-end on billions of video samples, with 4DS demonstrating that performance continues to improve with further scaling. Recurrent Architectures.