Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos

Youngseo KimDohyun KimGeonhee Han, Paul Hongsuck Seo[4pt] Dept. of CSE, Korea University { xwsa568 , a12s12 , rtrt505 , phseo }@korea.ac.kr · 2025

Read Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos by Youngseo KimDohyun KimGeonhee Han, Paul Hongsuck Seo[4pt] Dept. of CSE, Korea University { xwsa568 , a12s12 , rtrt505 , phseo }@korea.ac.kr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19936.txt.

Introduction

Youngseo Kim, Dohyun Kim, Geonhee Han, and Paul Hongsuck Seo find a tracking skill hidden inside image generators. Diffusion models trained to make images also learn rich semantic structure, and their self-attention maps can act as kernels that spread labels between matching pixels. The team asks whether that spreading can follow an object through video frames.

Extending the maps across frames gives a temporal kernel for zero-shot tracking by segmentation, with no tracking training. Test-time tools such as DDIM inversion, textual inversion, and adaptive head weighting make the labels steadier. Their DRIFT framework adds mask refinement guided by the Segment Anything Model and reaches top zero-shot scores on standard video segmentation benchmarks.

Catalog record

CREATORYoungseo KimDohyun KimGeonhee Han, Paul Hongsuck Seo[4pt] Dept. of CSE, Korea University { xwsa568 , a12s12 , rtrt505 , phseo }@korea.ac.kr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19936

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos Youngseo KimDohyun KimGeonhee Han, Paul Hongsuck Seo[4pt] Dept. of CSE, Korea University { xwsa568 , a12s12 , rtrt505 , phseo }@korea.ac.kr Abstract Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate their self-attention maps can be reinterpreted as semantic label propagation kernels, providing robust pixel-level correspondences between relevant image regions. Extending this mechanism across frames yields a temporal propagation kernel that enables zero-shot object tracking via segmentation in videos. We further demonstrate the effectiveness of test-time optimization strategies—DDIM inversion, textual inversion, and adaptive head weighting—in adapting diffusion features for robust and consistent label propagation. Building on these findings, we introduce Drift , a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement, achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks. 2 2 footnotetext: Corresponding author. 1 Introduction Diffusion models ( 39 ; 18 ; 40 ; 41 ; 29 ) have emerged as the dominant paradigm in generative modeling, achieving remarkable fidelity and diversity across modalities including images, audio, and video. Originally developed for image synthesis ( 36 ; 35 ; 38 ; 31 ) , they have since inspired a broad range of research beyond generation, driven by the observation that their intermediate representations encode rich semantic structures ( 17 ; 27 ; 43 ) . This arises naturally from the denoising process itself: to reconstruct coherent visual content from noisy inputs, the model must implicitly learn semantic correspondences among image regions. Recent studies have demonstrated that diffusion models encode rich semantic representations, enabling their features to be utilized for visual recognition ( 44 ; 10 ; 47 ) and localization tasks ( 46 ; 57 ; 55 ; 1 ) . In this work, we further investigate whether the capabilities learned by diffusion models from static images can capture temporal dynamics in videos by repurposing self-attention into cross-frame attention, revealing that diffusion features inherently support temporal reasoning beyond spatial recognition. We leverage these attention maps as a propagation kernel, enabling pixel-level correspondence and temporally consistent mask evolution without any additional supervision. We further demonstrate that test-time optimization effectively adapts image diffusion representations for object tracking in videos, enhancing temporal stability and object fidelity. By jointly applying DDIM inversion for semantically preserved noise, mask-specific textual inversion for object-aware adaptation, and adaptive head weighting to combine complementary attention patterns, we achieve stable and consistent mask propagation across frames. Building on these findings, we present Drift , a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement ( 21 ) , achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks. Our contributions can be summarized as follows: • We investigate that the self-attention in pretrained image diffusion models exhibits emergent temporal propagation, enabling zero-shot object tracking in videos. • We demonstrate that three test-time optimization techniques applied to diffusion models further improve cross-frame label propagation. • We propose Drift , a framework for object tracking in videos that demonstrates state-of-the-art zero-shot performance through extensive experiments on four standard video object segmentation benchmarks. (a) Original Image (b) Coarse Map (c) Self-attention Affinity (d) Propagated Mask (e) GT Mask Figure 1 : Visualization of Label Propagation via Self-Attention from a Text-to-Image Diffusion Models. Given an input image (a), the coarse map (b)—which corresponds to the cross-attention response for the token “cat”—provides approximate object localization based on the text prompt, while the self-attention map (c) captures semantic affinities across image regions to refine the coarse localization. Leveraging the self-attention map as a learned label propagation kernel, the coarse map is propagated to yield the final mask (d), which achieves substantially improved spatial precision, closely aligning with the GT mask (e). 2 Related Work Vision Tasks Using Image Diffusion Features. Image diffusion model features have been applied to various vision tasks beyond image generation, including correspondence ( 17 ; 27 ; 43 ) and object detection ( 16 ) . For segmentation, many studies have leveraged diffusion models ( 44 ; 46 ; 10 ; 56 ; 51 ) . In particular, DiffSeg ( 44 ) , DiffewS ( 56 ) , and DiffSegmenter ( 46 ) use attention maps in diffusion model within the image domain to derive or refine spatial masks. While prior works focus on static-image analysis, our method transfers the representational knowledge of pretrained image diffusion models to the video domain, leveraging diffusion features for object tracking via segmentation and achieving temporally consistent mask propagation without task-specific training. Video Diffusion Models. Video diffusion models are trained on videos and jointly model multiple frames in a sequence, using spatiotemporal attention to produce temporally coherent motion and appearance ( 3 ; 15 ; 5 ; 2 ) . Analogous to how image diffusion features are used in image tasks, video diffusion models are used in video applications such as point tracking ( 28 ) and video editing ( 33 ; 50 ; 9 ; 13 ) by extracting multi-frame video features. In contrast, we focus on pretrained image diffusion models that have never seen video data, yet we demonstrate that they can still capture meaningful temporal dynamics in videos. Diffusion-based Object Tracking. Recent works have explored leveraging diffusion priors for video understanding and tracking. VD-IT ( 57 ) simply extends prior diffusion-feature–based approaches to the video domain by training an additional segmentation head on top of video-diffusion features. In contrast, we show that pretrained image diffusion models can perform object tracking through their inherent self-attention, without video data and task-specific supervision. Diff-Tracker ( 55 ) localizes objects over time using cross-attention maps from a pretrained text-to-image diffusion model, where cross-attention conditioned on textual prompts delineates object regions within each frame. To capture motion beyond these spatial correspondences, it introduces a learned motion encoder and an online prompt updater that dynamically adapt the attention to temporal changes. SMITE ( 1 ) also relies on cross-attention features tied to learned text embeddings and fine-tunes those layers for video adaptation, while integrating an external point tracker ( 20 ) to enforce temporal coherence across frames. Both approaches depend on spatially grounded cross-attention and require auxiliary components—either learned motion encoders or explicit trackers—to model temporal dynamics. In contrast, we show that the self-attention layers of a pretrained text-to-image diffusion model inherently encode transferable motion cues across frames, enabling temporal label propagation without any task-specific training, or external tracking mechanism. Frame t\rightarrow t^{\prime} (a) Input Frames (b) Cosine Similarity (c) Aggregated Self-attention (d) Per-head Self-attention Figure 2 : Comparison of Cosine Similarity vs. Self-attention for Label Propagation. (a) The blue dot in the frame t is propagated to frame t{{}^{\prime}} . (b) Cosine similarity produces dispersed activations scattered across unrelated regions. (c) The aggregated self-attention map, in contrast, focuses sharply on the corresponding object region. (d) Individual attention heads exhibit complementary but distinct patterns, highlighting the diverse semantic relationships captured by multi-head self-attention. 3 Method In this work, our primary goal is to show that pretrained text-to-image diffusion models can be repurposed as object trackers without task-specific finetuning. The key insight is that

Read the complete text