DINO-Foresight : Looking into the Future with DINO

Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis · 2024

Read DINO-Foresight : Looking into the Future with DINO by Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11673.txt.

Introduction

Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis introduced DINO-Foresight in 2024, a method to predict future scenes from video. Forecasting what a scene will look like matters for self-driving cars and robots. Pixel-level forecasts are costly and waste effort on details that do not matter.

Their method works in the feature space of a large pretrained vision model instead of raw pixels. A transformer learns, without labels, to forecast how those features evolve over time. Standard task heads can then read the forecast features for jobs such as segmentation of future frames. Tests showed strong performance, robustness, and scale. Code and a project page were released with the study.

Catalog record

CREATOREfstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11673

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DINO-Foresight : Looking into the Future with DINO Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis Abstract Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce DINO-Foresight , a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code at https://dino-foresight.github.io/ . 1 Introduction Predicting future states in video sequences is a key challenge in computer vision and machine learning, with important applications in autonomous systems like self-driving cars and robotics ( Finn et al., 2016 ; Dosovitskiy and Koltun, 2017 ) . These systems must navigate dynamic environments safely, yet predicting future states remains difficult—especially in complex scenarios involving multi-object interactions over long time horizons. Recent approaches focus on generating realistic RGB future frames using latent generative modeling ( Rombach et al., 2022 ) . These methods first compress RGB data into a latent space, such as continuous ( Rombach et al., 2022 ) or discrete ( Esser et al., 2021 ) Variational Autoencoder (VAE) representations. Then, they train generative models—like diffusion models ( Zheng et al., 2024 ; Gao et al., 2024 ; Ho et al., 2022a ; Ho et al., 2022b ; Harvey et al., 2022 ; Brooks et al., 2024 ) , autoregressive models ( Hu et al., 2023 ; Hong et al., 2023 ; Kondratyuk et al., 2024 ; Wang et al., 2024 ) , or masked video generation ( Yu et al., 2023 ; Yu et al., 2024 ) —to predict future states in this compressed space. While this reduces dimensionality and improves training stability ( Rombach et al., 2022 ) , VAE latents often lack semantic alignment, making them hard to interpret or use in downstream scene understanding tasks (see Table 2 ). Moreover, these methods must model both low-level appearance and high-level semantics, even though decision-making systems (e.g., self-driving cars) primarily need semantic scene understanding—what objects exist and where they are. Latent generative approaches may waste capacity on irrelevant low-level details, compromising temporal semantic accuracy. In contrast, vision foundation models (VFMs) have revolutionized scene understanding with robust, transferable features ( Oquab et al., 2024 ; Radford et al., 2021 ; Venkataramanan et al., 2025 ; Bardes et al., 2024 ) . This raises a key question: can VFM features serve as a semantically rich, high-dimensional latent space for precise future prediction? In this work, we propose precisely this idea. Instead of predicting future low-level VAE latents, we forecast the temporal evolution of VFM features directly. This shift brings several important advantages: (a) Beyond low-level latents. It leverages semantically meaningful representations, inheriting strong scene understanding. (b) Dynamic semantics over raw frames. It avoids full-frame synthesis, letting the model focus on meaningful dynamics, reducing complexity and sidestepping challenges like multimodal pixel distributions. (c) Scalable multi-task support and modular integration. Forecasted features align with downstream tasks, enabling plug-and-play integration with pretrained classifiers, segmenters or task-specific heads—without the need to retrain the core feature predictor (see Figure 1 ). This represents a significant departure from prior semantic feature prediction methods ( Luc et al., 2017 ) , which face significant challenges with multi-task scalability. Such approaches either require training a separate model for each task ( Nabavi et al., 2018 ; Chiu et al., 2020 ; Terwilliger et al., 2019 ) or involve predicting features from multiple task-specific models simultaneously ( Hu et al., 2020 ; Karypidis et al., 2025 ) , resulting in more complex and less scalable architectures. Figure 1: Forecasting VFM Features for Future Frames. At the core of our approach is the prediction of future VFM feature evolution. To this end, we train a masked transformer model in a self-supervised manner to forecast these features from context frames, minimizing SmoothL1 loss between predicted and actual future features. By forecasting these rich and versatile features, task-specific prediction heads—such as semantic segmentation, depth, and surface normals—can be effortlessly employed at test time, enabling modular and efficient multi-task scene understanding. In this work we show that forecasting high-dimensional VFM features is not only feasible but also achieves strong performance, offering a new path toward semantically grounded future prediction. In summary, our contributions are: (1) We introduce DINO-Foresight , a self-supervised method for semantic future prediction that builds on the key idea of forecasting VFM features—our core contribution. Unlike latent generative methods that rely on low-level VAE features, our approach delivers precise future scene understanding without modeling unnecessary appearance details, enabling a unified model for diverse scene understanding tasks. (2) To realize this idea, we design an efficient masked feature transformer (see Figure 1 ) that propagates multi-layer, high-resolution features critical for achieving strong task performance. (3) Experimental results demonstrate a unique advantage of our approach - our single model successfully handles multiple future-frame understanding tasks (semantic segmentation, instance segmentation, depth prediction, and surface normal prediction) where previous approaches required multiple specialized models. Furthermore, as we show in Appendix Subsection A.2 intermediate features within our masked transformer can further improve downstream task performance, highlighting its potential as a self-supervised learning strategy that enhances the already strong VFM features. 2 Related Work Video Prediction is an extensively studied problem. Early approaches based Convolutional LSTMs ( Nabavi et al., 2018 ; Xu et al., 2018 ; Wang et al., 2018 ; Castrejon et al., 2019 ; Lee et al., 2021 ; Wu et al., 2021 ; Gao et al., 2022 ) struggled to maintain both visual quality and temporal consistency. Subsequent developments introduced generative adversarial networks and variational autoencoders ( Yan et al., 2021 ; Vondrick et al., 2016b ; Castrejon et al., 2019 ; Lee et al., 2018 ; Babaeizadeh et al., 2018 ) , alongside diffusion models ( Ho et al., 2022a ; Ho et al., 2022b ; Gao et al., 2024 ; Harvey et al., 2022 ) , to enhance spatial-temporal coherence and improve the quality of predictions. Furthermore, transformer-based models have also been adapted to videos, utilizing auto-regressive and masked modeling objectives to capture video dynamics ( Yu et al., 2024 ; Yu et al., 2023 ; Gupta et al., 2023 ; Wang et al., 2024 ) . Latent Generative Models Current state-of-the-art video prediction methods ( Gupta et al., 2023 ; Yu et al., 2023 ; Yu et al., 2024 ; Gao et al., 2024 ) build upon latent generative approaches ( Rombach et al., 2022 ) . These frameworks employ an autoencoder to compress RGB frames into a latent space, then train generative models to predict future states in this compressed representation. While sharing superficial similarities with our approach—including the use of latent spaces and masked transformers in some cases ( Gupta et al., 2023

Read the complete text