RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers

Ahmet Berke Gökmen, Yiğit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar · 2025

Read RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers by Ahmet Berke Gökmen, Yiğit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13344.txt.

Introduction

Ahmet Berke Gokmen and co-authors present a 2025 method that copies motion from one video into a newly made one. Their method needs no fresh training of the video model. It works by changing how the model marks positions in space and time. First, dense motion flow is pulled from the source video. Those shifts then warp the position marks so the new video inherits the motion. A further tuning step during generation aligns speeds with the target path.

A guard term keeps the output true to the text prompt and stops doubled scenes. That guard smooths a pattern drawn from the source video's frequency parts. Tests on standard benchmarks beat recent rival methods in both scores and looks. The authors say position marks are an overlooked lever for motion control. Their method runs on top of existing video models without retraining them.

Catalog record

CREATORAhmet Berke Gökmen, Yiğit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers Ahmet Berke Gökmen, Yiğit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar Abstract We propose RoPECraft, a training-free video motion transfer method for diffusion transformers that operates solely by modifying their rotary positional embeddings (RoPE). We first extract dense optical flow from a reference video, and utilize the resulting motion offsets to warp the complex-exponential tensors of RoPE, effectively encoding motion into the generation process. These embeddings are then further optimized during denoising time steps via trajectory alignment between the predicted and target velocities using a flow-matching objective. To keep the output faithful to the text prompt and prevent duplicate generations, we incorporate a regularization term based on the phase components of the reference video’s Fourier transform, projecting the phase angles onto a smooth manifold to suppress high-frequency artifacts. Experiments on benchmarks reveal that RoPECraft outperforms all recently published methods, both qualitatively and quantitatively. Reference RoPECraft A robotic courier zips through a maze of an eco-friendly, automated warehouse, packages whizzing past on conveyor belts. The silhouette of a ballerina leaps across a sunlit studio, delicate shadow dancing on polished hardwood floors. A young woman riding a small skateboard through the misty rainforest. A vintage steam locomotive rolls past a snowy station, its black iron body steaming in the frosty air. A camel is seen walking in an abandoned, moonlit amphitheater. A vintage biplane loops gracefully above an airfield. Figure 1: Our method successfully transfers the motion from reference videos. 1 Introduction Diffusion transformers (DiT) have become a leading approach for conditional video generation, producing realistic and coherent content across diverse scenarios [ 6 , 16 , 20 , 50 , 21 , 38 , 44 ] . While text conditioning provides a convenient interface, they are often too ambiguous to specify detailed spatio-temporal dynamics such as body movement, camera motion, or interactions. As generative quality improves, so does the demand for more precise and controllable motion synthesis. To address the limitations of text-based motion control, earlier methods introduced explicit structural cues such as masks, bounding boxes, or depth maps to guide motion [ 9 , 43 , 40 ] . These approaches assume consistent geometry between the reference and generated videos, which often fails under domain shifts [ 45 ] . More recent work has shifted toward leveraging latent representations within generative models. Some methods extract motion features from internal activations [ 14 , 45 , 42 ] , while others modify the latent prior [ 4 ] to better align reference and generated motions. A prominent example, Go with the Flow (GWTF) [ 4 ] , uses a pretrained optical flow model [ 37 ] to generate motion priors that warp the initial noise input while maintaining its Gaussianity. This reportedly stabilizes and speeds up convergence. However, directly warping noise disrupts the intended latent distribution of the pre-trained DiT, as seen in Fig. 2 . Even with inference-time latent optimization, the method struggles with generalization and domain shifts. As a result, GWTF requires costly fine-tuning, demanding around 40 GPU days [ 4 ] . DiTFlow [ 31 ] offers a more efficient alternative by optimizing latents or positional embeddings at test time without model retraining. However, it incurs high computational costs due to its reliance on a full-size attention-based feature computation. We extend DiTFlow’s approach by updating only positional embeddings, avoiding latent space deviation and content leakage. Our method introduces motion-augmented rotary positional embeddings, warped via optical flow-derived displacements to embed motion cues ( Section 4.1 ). We enhance this with flow-matching-guided optimization during early time steps, enabling stable and precise generation ( Section 4.2 ). We further ensure spatiotemporal consistency through a Fourier phase regularization ( Section 4.3 ). Unlike GWTF, our method requires no backbone training, drastically cutting computational costs. Compared to DiTFlow, it delivers higher efficiency and better motion quality. In addition, to evaluate motion alignment, we propose Fréchet Trajectory Distance (FTD) ( Section 5.2 ). Our method outperforms recent approaches in qualitative and quantitative assessments. Our contributions are: • An efficient motion transfer, RoPECraft, that leverages motion-augmented rotary positional embeddings in a training-free setting, without requiring any backbone fine-tuning. • A novel use of optical flow displacements to warp rotary positional embeddings, encoding spatial motion cues in attention calculations. • A unified optimization strategy combining flow matching velocity prediction and phase constraint regularization to enhance motion accuracy and ensure temporal coherence. • A new evaluation metric, Fréchet Trajectory Distance (FTD), for quantifying motion alignment between generated and reference videos. Input Latent warp + optimization + train DiT Ours A motorcycle is seen riding on a track, kicking up smoke as it goes. Figure 2: Latent warping [ 4 ] without an expensive fine-tuning of the DiT fails (Column 2), and latent optimization is not adequate to recover the domain shift (Column 3). Our approach keeps the latent space intact, and performs successful motion transfer, all without model re-training (Column 4). 2 Related Work Text-to-video models. Following the successful application of diffusion models in image generation [ 12 , 30 , 29 , 34 ] , efforts have been made to extend this approach to video generation. Initial efforts used U-Net-based architectures for this purpose, utilizing temporal modules [ 16 ] or inflated convolutions [ 1 , 3 , 20 , 39 ] to transfer the image prior to the video domain. More recently, DiT pipelines [ 21 , 44 , 38 , 25 , 50 , 28 ] have gained attention due to their superior capabilities in temporal modeling and enhanced quality. Motivated by these, we also adopt a DiT backbone. Motion transfer. The goal of motion transfer is to synthesize videos whose dynamics match a reference clip while disentangling motion from appearance. Earlier methods injected explicit structure (masks or depth maps) [ 9 , 43 , 40 ] . Subsequent work learned dedicated motion embeddings and fed them to the generator [ 23 , 22 ] . Recent approaches exploit the dense motion signals already present in backbone features [ 42 , 15 , 45 , 14 ] , or condition on trajectories extracted from the reference [ 48 , 46 , 41 ] . The current state of the art include Go With The Flow [ 4 ] , which warps the initial noise with reference flow and fine-tunes the DiT on this prior, and DiTFlow [ 31 ] , which derives displacement maps from cross-frame attention and updates either latents or positional embeddings. Building on DiTFlow’s insight, we dynamically update RoPE to guide attention toward reference motion while keeping the backbone frozen. Unlike prior work, we initialize RoPE with our motion-augmentation algorithm and regularizers, enabling fast and accurate motion transfer, without requiring model fine-tuning [ 4 ] , inversion [ 42 , 45 , 14 ] , masks [ 14 ] , and high GPU memory during tuning [ 31 ] . Positional embeddings in vision transformers. Transformers lack inherent order awareness, so Vision Transformers (ViTs) [ 11 ] rely on positional embeddings to encode spatial relationships among patches. In early ViT’s fixed sinusoidal or learnable absolute embeddings were used [ 11 , 8 ] . These failed to generalize across varying input

Read the complete text