Let Your Image Move with Your Motion! – Implicit Multi-Object Multi-Motion Transfer

Yuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, [0.2em] Yun Sing Koh, Cheng Yan, Xinyu Zhang, [0.2em] Tianjin University · 2026

Read Let Your Image Move with Your Motion! – Implicit Multi-Object Multi-Motion Transfer by Yuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, [0.2em] Yun Sing Koh, Cheng Yan, Xinyu Zhang, [0.2em] Tianjin University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01000.txt.

Introduction

Li and eight colleagues tackle multi-object motion transfer for controllable video, where one image's objects each need a different motion from reference clips. Their FlexiMMT framework pulls motion patterns apart and assigns each to the right object, allowing free recombination and arbitrary motion-to-object pairings. Object-specific masks keep motion and text tokens inside their regions, and masks derived from diffusion attention spread across frames, giving precise compositional results.

Catalog record

CREATORYuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, [0.2em] Yun Sing Koh, Cheng Yan, Xinyu Zhang, [0.2em] Tianjin University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47484
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01000

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Let Your Image Move with Your Motion! – Implicit Multi-Object Multi-Motion Transfer Yuze Li, Dong Gong, Xiao Cao, Junchao Yuan, Dongsheng Li, Lei Zhou, [0.2em] Yun Sing Koh, Cheng Yan, Xinyu Zhang, [0.2em] Tianjin University Abstract Motion transfer has emerged as a promising direction for controllable video generation, yet existing methods largely focus on single-object scenarios and struggle when multiple objects require distinct motion patterns. In this work, we present FlexiMMT, the first implicit image-to-video (I2V) motion transfer framework that explicitly enables multi-object, multi-motion transfer. Given a static multi-object image and multiple reference videos, FlexiMMT independently extracts motion representations and accurately assigns them to different objects, supporting flexible recombination and arbitrary motion-to-object mappings. To address the core challenge of cross-object motion entanglement, we introduce a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to constrain attention, ensuring that motion and text tokens only influence their designated regions. We further propose a Differentiated Mask Propagation Mechanism that derives object-specific masks directly from diffusion attention and progressively propagates them across frames efficiently. Extensive experiments demonstrate that FlexiMMT achieves precise, compositional, and state-of-the-art performance in I2V-based multi-object multi-motion transfer. Our project page is: https://ethan-li123.github.io/FlexiMMT_page/ Figure 1 : Illustration of FlexiMMT. Given multiple reference videos, our FlexiMMT independently extracts each motion and applies them to images with any number of objects, enabling precise and compositional multi-object multi-motion transfer. For clarity, we show only simplified motion labels here; full text prompts and vivid videos are provided in the Supplementary Material. † † footnotetext: ♠ Project Lead. † † footnotetext: † Corresponding Author 1 Introduction Recent advancements in video diffusion models 10 ; 3 ; 49 ; 41 ; 47 ; 21 have significantly accelerated the development of controllable video generation 58 ; 16 ; 50 ; 23 ; 13 ; 7 ; 4 ; 11 ; 57 ; 17 ; 12 ; 8 . Among various controllable tasks, motion transfer has emerged as a key direction, aiming to capture motion dynamics from reference guidances and apply them to target subjects. Some motion transfer methods use explicit motion guidance, such as pose 12 ; 6 ; 25 ; 42 ; 48 , optical flow 4 ; 28 ; 38 , or trajectories 15 ; 27 ; 8 ; 46 ; 57 ; 54 . While effective, these approaches rely on additional estimators and require accurate motion extraction, which can be computationally expensive. In contrast, implicit-motion methods encode motion directly from reference videos into latent embeddings or learned parameters 19 ; 53 ; 58 ; 16 ; 50 ; 23 ; 43 ; 31 ; 2 ; 24 ; 26 . This paradigm avoids explicit motion preprocessing and provides greater flexibility and ease of use. In this paper, we focus on implicit motion transfer from reference videos. Based on the type of input conditions, existing motion transfer methods can be grouped into two major directions, i.e . , text-to-video (T2V) 58 ; 16 ; 50 ; 23 ; 13 ; 7 and image-to-video (I2V) 12 ; 4 ; 8 . T2V methods typically transfer motion from reference sequences while synthesizing semantic content guided by text descriptions. I2V methods are more challenging as they must simultaneously preserve the appearance of a given first-frame image and follow the reference motion, requiring much stronger spatial and structural consistency. Most recent studies 19 ; 29 ; 53 ; 7 focus on single-concept ( i.e . , one object or one motion) motion transfer. However, simultaneously transferring distinct motions to multiple objects within the same image remains largely unexplored and continues to be an open challenge. In this paper, we propose FlexiMMT , a novel implicit motion transfer framework that, to the best of our knowledge, is the first to explicitly tackle multi-object, multi-motion challenges in I2V generation. Given a multi-object input image and multiple reference videos, our method independently extracts each motion and accurately assigns it to the corresponding target object, supporting flexible, compositional, and arbitrary motion–object rearrangements. The central challenge in this setting is the motion entanglement across different objects. To this end, we introduce a Motion Decoupled Mask Attention Mechanism (MDMA), which leverages object-specific masks to perform explicit motion disentanglement within the attention layers. This mechanism enforces that text and motion tokens exclusively attend to their corresponding objects from video tokens rather than interfering with others, thereby enabling precise and independent motion control. To efficiently obtain object-specific masks, we introduce a Differentiated Mask Extraction Mechanism (DMEM). Unlike approaches that rely on external segmentation models 18 ; 34 ; 35 , DMEM directly derives object masks from the model’s own attention activations, providing an efficient training-time solution. At inference, we further mitigate cross-object mask confusion through a dynamic Regressive Mask Propagation Mechanism (RMPM) that progressively and efficiently transfers first-frame masks to subsequent frames, maintaining clean and consistent object boundaries throughout the generation process. Quantitative and qualitative experiments on 200 video-image pairs demonstrate that our FlexiMMT enables independent and flexible motion assignment across multiple objects. As illustrated in Fig. 1 , different motion patterns extracted from multiple reference videos can be accurately transferred to distinct objects in the input image, and these motions can be arbitrarily swapped or recombined across objects. Our contributions are as follows: • We introduce FlexiMMT, the first implicit-motion I2V framework capable of multi-object, multi-motion transfer. • We design a Motion Decoupled Mask Attention Mechanism that uses object-specific masks to disentangle motion across objects, enabling precise and flexible per-object motion assignment. • We present a Differentiated Mask Extraction Mechanism that derives object-specific masks directly from diffusion attention with dynamic progressive propagation scheme for stable multi-object control. • Extensive experiments demonstrate that FlexiMMT offers strong flexibility, compositionality, and state-of-the-art performance in multi-object I2V motion transfer. 2 Related Work Figure 2 : Overview of FlexiMMT. (a) Training: Given one reference video, insert trainable motion tokens into text and video tokens. Get the object mask through a simple QK multiplication method, then mask out M2M, M2V and T2M parts in attention map. (b) Inference: Given a multi-object conditional image, we first segment each object’s mask with semantic segmentation model 35 . Concatenate pre-trained motion tokens into text and video tokens for inference. Extract each object’s latent-space mask in subsequent frames via Dynamic Regressive Mask Propagation Mechanism (Dynamic RMPM), and apply it to Motion parts and Text parts in attention map. 2.1 Explicit Motion Representation Models Explicit motion representation models achieve controllable video generation through predefined motion signals such as pose skeletons, optical flow fields, and trajectories. The core idea is to enforce explicit constraints to ensure that the generated motion aligns with the desired behavior. In the T2V domain, methods are categorized by motion signals: pose-driven method like FollowYourPose 25 focus on fusing text and pose sequences to address the limitation of text in specifying fine-grained human motion; depth/edge-based methods like ControlVideo 56 and SparseCtrl 9 adopt ControlNet-like 52 structures to condition generation on text and depth/edge signals for temporal coherence; trajectory-based method like Peekaboo 15 use interactive frameworks with text and foreground trajectories to guide target motion along predefined paths. In the I2V domain, models emphasize appearance-motion synergy: pose-driven methods such as AnimateAnyone 12 and MagicAnimate 48 fuse pose and reference image features, with enhancements to temporal consistency;

Read the complete text