Characterizing Motion Encoding in Video Diffusion Timesteps

Vatsal Baherwani, Yixuan Ren, Abhinav Shrivastava · 2025

Read Characterizing Motion Encoding in Video Diffusion Timesteps by Vatsal Baherwani, Yixuan Ren, Abhinav Shrivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22175.txt.

Introduction

Vatsal Baherwani, Yixuan Ren, and Abhinav Shrivastava ask when, during video generation, a diffusion model decides motion versus surface appearance. Practitioners long assumed early denoising steps shape motion and layout while later steps polish looks, but nobody had measured the split carefully. The authors measure it by injecting new conditions during chosen step ranges and tracking the trade-off between changed appearance and preserved motion.

Across several model designs, they always find an early motion-dominant regime and a later appearance-dominant one, with a usable boundary between them. They then simplify one-shot motion customization by training and running only inside the motion-dominant window. The recipe transfers motion strongly without extra debiasing modules or special objectives, and can plug into current editing methods.

Catalog record

CREATORVatsal Baherwani, Yixuan Ren, Abhinav Shrivastava
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Characterizing Motion Encoding in Video Diffusion Timesteps Vatsal Baherwani, Yixuan Ren, Abhinav Shrivastava Abstract Text-to-video diffusion models synthesize temporal motion and spatial appearance through iterative denoising, yet how motion is encoded across timesteps remains poorly understood. Practitioners often exploit the empirical heuristic that early timesteps mainly shape motion and layout while later ones refine appearance, but this behavior has not been systematically characterized. In this work, we proxy motion encoding in video diffusion timesteps by the trade-off between appearance editing and motion preservation induced when injecting new conditions over specified timestep ranges, and characterize this proxy through a large-scale quantitative study. This protocol allows us to factor motion from appearance by quantitatively mapping how they compete along the denoising trajectory. Across diverse architectures, we consistently identify an early, motion-dominant regime and a later, appearance-dominant regime, yielding an operational motion-appearance boundary in timestep space. Building on this characterization, we simplify current one-shot motion customization paradigm by restricting training and inference to the motion-dominant regime, achieving strong motion transfer without auxiliary debiasing modules or specialized objectives. Our analysis turns a widely used heuristic into a spatiotemporal disentanglement principle, and our timestep-constrained recipe can serve as ready integration into existing motion transfer and editing methods. 1 Introduction Figure 1 : Spatiotemporal disentanglement in video diffusion models. Our finding reveals that motion is primarily encoded in the early denoising timesteps. Given a reference video (top) and its ground truth caption (blue), we perform DDIM inversion and then denoise with a new prompt that modifies only the subject (yellow). The resampled videos show different subject editing and motion preservation results by applying the original or new prompts at different timesteps. Diffusion models 11 have achieved remarkable performance in image and video synthesis, and large-scale pre-trained foundations now support many controllable generation tasks such as editing 43 ; 30 and customization 27 ; 5 . For videos, a key challenge is that temporal motion and spatial appearance are entangled in the generative process, while many applications require modifying only one factor and preserving the other. Understanding how motion is encoded along the denoising trajectory is therefore central to controllable video generation. Recent work has analyzed what different timesteps and layers represent in image diffusion models, revealing coarse-to-fine synthesis and frequency-specific behaviors 8 ; 21 ; 41 ; 25 ; 16 ; 19 ; 36 . In the video domain, several methods 38 ; 17 ; 35 empirically exploit that early denoising steps tend to determine motion and layout, while later steps refine appearance. This heuristic has been used to design subject editing and motion transfer pipelines, but it remains informal: there is no systematic quantification of how motion and appearance trade off across timesteps, how consistent this pattern is across architectures and datasets, or where an operational boundary between motion-dominant and appearance-dominant regimes lies. In this work, we provide an in-depth characterization of motion encoding across timesteps in text-to-video diffusion models. Given a reference video and its text prompt, we obtain a denoising trajectory and resample the video while replacing only the appearance-related part of the prompt over specified ranges of timesteps, keeping the original prompt at other steps. Although this procedure is not intended as a high-quality editing method, it produces tampered RGB videos that reveal how strongly appearance can be edited and how well the original motion is preserved when re-conditioning at different timesteps. By measuring appearance alignment and motion preservation, we use the resulting trade-off as a large-scale quantitative proxy for how motion is encoded in video diffusion timesteps. Sweeping over timestep ranges reveals a consistent spatiotemporal structure: an early motion-dominant regime where re-conditioning strongly affects temporal dynamics, and a later appearance-dominant regime where re-conditioning mainly changes spatial details while largely keeping the motion. This pattern induces an operational motion-appearance boundary in timestep space, defined as the range where appearance editing becomes effective yet motion preservation remains high. We validate this behavior across three text-to-video architectures: ModelScope 31 , a U-Net with dedicated spatial and temporal attention blocks; Latte 23 , a Transformer with decoupled spatial and temporal attentions; and CogVideoX 40 , a Transformer with unified spatiotemporal attention. Despite their architectural varieties, all models display similar motion-dominant and appearance-dominant regimes. Based on this characterization, we derive a simplified design principle for motion-centric adaptation: restrict modeling to motion-dominant timesteps. We instantiate this principle in a one-shot video motion customization framework, where a single reference video provides the target motion that should be transferred to new subjects and scenes with temporal diversity. Prior works typically introduce auxiliary debiasing modules or specialized losses to suppress unwanted spatial signals 47 ; 26 . In contrast, we fine-tune temporal attention in pre-trained text-to-video diffusion models using the vanilla diffusion loss, while constraining both training and inference to early timesteps. This timestep constraint effectively prevents appearance leakage despite using full reconstruction losses, and it enables efficient partial-attention tuning and even direct full-rank fine-tuning without triggering spatial overfitting. In summary, our main contributions are: • We introduce a prompt-tampering probe and quantitatively analyze the trade-off between appearance editing and motion preservation across timesteps in pre-trained text-to-video diffusion models. • We identify consistent motion-dominant and appearance-dominant regimes across diverse architectures and characterize an operational motion-appearance boundary in timestep space. • Guided by this boundary, we propose a timestep-constrained one-shot motion customization framework that requires no auxiliary debiasing modules and naturally supports partial-attention tuning and direct tuning. 2 Related Works 2.1 Diffusion Attribute Disentanglement Attribute disentanglement in diffusion models is increasingly studied as a way to interpret internal representations and gain finer control. For image generation, several works analyze how information is organized across timesteps and layers. Aggregating multi-timestep and multi-scale features reveals complementary geometric and semantic cues for correspondence 22 , and spectral analyses show that low-frequency content dominates early steps while high-frequency refinements appear later, motivating non-uniform timestep sampling and frequency-aware manipulation 15 ; 36 . Other methods make timesteps explicit supervision axes through timestep-aware representations and step-aware preference alignment 42 ; 2 ; 29 , while per-step editing demonstrates that intervening at selected timesteps can separate layout from style 9 . Recent interpretability work further shows that semantic concepts are structured across layers and timesteps 13 . These studies, however, focus on spatial attributes in image diffusion and do not characterize how motion is encoded along timesteps in video models. For video diffusion models, timestep-wise disentanglement is less developed and mostly used in a heuristic way. 17 ; 35 inject new appearance into reference videos often bypass early steps to reduce motion interference, implicitly assuming that motion is encoded early and appearance later, but without quantifying where motion and appearance respectively dominate. 1 studies how camera trajectories are encoded over timesteps and separates camera from scene content, whereas our goal is to understand general object and scene motions and their interaction with appearance. 44 learns frequency-aware embeddings across all timesteps for image-to-video (I2V) models, whose appearance has been mostly debiased by the image image. 38 ; 20 extract motion-aware features from pre-trained T2V models and guide motion by feature alignment without tuning, while our motion module models the reference motion signal and is able to adapt it to any novel scenarios with temporal diversity. Moreover, we systematically characterize

Read the complete text