From Image to Video: An Empirical Study of Diffusion Representations

Pedro Vélez 1 Luisa F. Polanía 1 Yi Yang Chuhan Zhang Rishabh Kabra Anurag Arnab Mehdi S. M. Sajjadi · 2025

Read From Image to Video: An Empirical Study of Diffusion Representations by Pedro Vélez 1 Luisa F. Polanía 1 Yi Yang Chuhan Zhang Rishabh Kabra Anurag Arnab Mehdi S. M. Sajjadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-07001.txt.

Introduction

Pedro Vélez and co-authors compare what image and video diffusion models learn inside. Diffusion models make images and clips of striking realism by clearing noise. Their inner features may also serve tasks like sorting images or judging depth. The team trained the same design for images and for video, then read its features on four tasks.

Video models beat image models on all tasks, from sorting photos to tracking motion. The size of the lead varied a lot by task, a striking spread. Features from distinct layers and noise levels were also checked. Model size and training budget shaped both making and reading quality. The study is the first head-to-head test of the two training goals for understanding tasks.

Catalog record

CREATORPedro Vélez 1 Luisa F. Polanía 1 Yi Yang Chuhan Zhang Rishabh Kabra Anurag Arnab Mehdi S. M. Sajjadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-07001

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Image to Video: An Empirical Study of Diffusion Representations Pedro Vélez 1 Luisa F. Polanía 1 Yi Yang Chuhan Zhang Rishabh Kabra Anurag Arnab Mehdi S. M. Sajjadi Abstract 0 0 footnotetext: 1 1 footnotemark: 1 Equal contribution. 2 2 footnotemark: 2 Corresponding authors: [email protected] , [email protected] , [email protected] . Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This success has sparked interest in leveraging their representations for visual understanding tasks. While recent works have explored this potential for image generation, the visual understanding capabilities of video diffusion models remain largely uncharted. To address this gap, we systematically compare the same model architecture trained for video versus image generation, analyzing the performance of their latent representations on various downstream tasks including image classification, action recognition, depth estimation, and tracking. Results show that video diffusion models consistently outperform their image counterparts, though we find a striking range in the extent of this superiority. We further analyze features extracted from different layers and with varying noise levels, as well as the effect of model size and training budget on representation and generation quality. This work marks the first direct comparison of video and image diffusion objectives for visual understanding, offering insights into the role of temporal information in representation learning. 1 Introduction The field of computer vision has made remarkable strides towards imbuing machines with the ability to see and interpret the visual world. A key challenge in realizing this goal lies in learning effective representations that capture both the rich semantic information and dynamic 4D structure (3D & motion) inherent in real-world visual data. Leading approaches in visual representation learning have primarily focused on contrastive learning and reconstruction . Contrastive models can be exemplified by DINO [ 8 ] which applies knowledge distillation between augmented views of images, and CLIP [ 41 ] which aligns image and text representations, while MAE [ 23 ] and I-JEPA [ 1 ] are commonly used reconstruction models that predict masked pixels and features, respectively. Figure 1 : Video vs. image diffusion representations – The diffusion model V-WALT trained on generating videos learns better features than the same model trained on generating images (I-WALT, normalized to 100% here) as measured across a range of readout tasks. See Sec. 4.2 for details. Meanwhile, generative models have demonstrated an unprecedented ability to synthesize novel, photo-realistic imagery [ 27 , 29 ] . Among generative approaches, diffusion models [ 25 ] have emerged as the state-of-the-art for both image and video generation, achieving remarkable results in synthesizing high-quality visual content [ 19 ] . Image diffusion models have also demonstrated their capacity to learn powerful representations for downstream tasks such as image classification [ 28 ] , depth estimation [ 47 ] and semantic keypoint matching [ 24 , 50 ] , suggesting that the denoising process enables these models to acquire a deep understanding of visual semantics and structure. However, despite the recent emergence of video diffusion models capable of generating high-fidelity video content, the representational power of these models remains largely unexplored, particularly in the context of dynamic scene understanding. This leaves a crucial question unanswered: how effectively do video diffusion models capture the interplay between motion and spatial scene understanding, and how do they perform compared to image diffusion models? In this work, we delve into the representational power of the diffusion model WALT [ 22 ] which is particularly well-suited for a direct comparison thanks to its hybrid architecture, allowing us to specifically analyze the implications of the video versus image pre-training objective on downstream performance. Our key contributions are as follows: 1. We compare the same model architecture trained on image vs. video generation and demonstrate the superiority of video for learning representations across diverse downstream tasks. 2. We provide a qualitative analysis of video diffusion representations, highlighting the role of motion in the video representation space. 3. We present findings on the relationship between training budget, model size, visual generation quality and downstream performance of the learned representations. To the best of our knowledge, this is the first work to analyze and compare the internal representations of the same diffusion model pre-trained for video versus image generation across a diverse range of tasks. 2 Related Work Diffusion models [ 25 ] have revolutionized the fields of image and video synthesis as they offer unprecedented generation quality [ 27 , 29 ] . Whereas they once relied heavily on U-Nets [ 43 ] , transformer-based diffusion models have since gained traction [ 39 ] . We refer the reader to Zhang et al. [64] for a survey on diffusion-based image generation. Image diffusion for visual understanding. Beyond their impressive generative capabilities, image diffusion models have proved remarkably effective in perception tasks. For example, DDPM-Seg [ 4 ] utilizes intermediate features of a U-Net-based diffusion model and excels at semantic segmentation. It has further been extended to different architectures and showed that the best-performing features are found in the middle upsampling layers using small noise levels [ 59 ] . Pre-trained text-to-image diffusion models have also been applied to panoptic segmentation [ 60 ] . In terms of correspondence tasks, DIFT [ 50 ] , a method based on extracting features from diffusion models, showed that features extracted in earlier layers with larger timesteps tend to be more semantically meaningful, whereas lower-level features with smaller timesteps focus more on low-level details. A related work found that merging diffusion representations with DINOv2 [ 36 ] features improves performance on semantic correspondence [ 63 ] . Luo et al. [30] demonstrated that having an aggregation network that learns weights for all features maps across all layers and timesteps performs better than manually selecting layers and timesteps. Clark and Jaini [14] proposed a mechanism that makes use of the denoising scores for all possible labels, thereby creating a zero-shot ImageNet classifier which is however computationally impractical. Another work compares various types of readout architectures with similar parameter count and has found attentive readout to perform best [ 34 ] . Further works propose diffusion models as a general foundational model for a suite of image understanding tasks. For example, Zhao et al. [66] showed that diffusion models excel at both semantic segmentation and depth estimation, while Yang and Wang [61] , showed state-of-the-art performance on several image classification, semantic segmentation and landmark detection benchmarks using diffusion models. We refer the reader to a recent survey on the connection between diffusion models and image representation learning for an extensive overview [ 19 ] . Video representation learning. Building on image representation learning, research is naturally progressing to video analysis for enhanced spatial understanding [ 37 ] . Contrastive and masked modeling are popular choices here: V-JEPA [ 5 ] predicts video embeddings from a set of random crops in a self-supervised student-teacher setup, while VideoMAE [ 51 , 18 , 57 ] pre-trains vision transformers by masking and reconstructing random video patches. VideoPrism [ 65 ] combines both: it uses video-language contrastive learning in a first stage to capture semantic content, while a second stage uses masked modeling with global-local distillation and token shuffling. Recent works have started to explore the

Read the complete text