SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing

Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang · 2026

Read SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing by Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19228.txt.

Introduction

Zhang and co-authors edit videos from text instructions while keeping the motion of the source clip. Their SAMA method splits the task into two parts. Semantic anchoring fixes the structure at sparse key frames from the instruction alone. Motion alignment pre-trains the same backbone on repair tasks drawn from raw video. No paired edit data is needed for the first stage. The team reports top scores among open models. It also rivals leading closed systems like Kling-Omni.

Catalog record

CREATORXinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42239
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang Abstract Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors ( e . g ., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized S emantic A nchoring and M otion A lignment), a framework that factorize video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring which establish a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g. Kling-Omni). Code, models, and datasets will be released. † † footnotetext: ∗ Equal Contribution † Project Leader ‡ Corresponding Author Figure 1: Teaser and overview. Top: qualitative comparisons on VIE-Bench, comparing SAMA with representative open- and closed-source systems. Bottom left: illustration of SAMA’s semantic–motion training objectives. Bottom right: fine-grained VIE-Bench performance comparison. 1 Introduction Diffusion models have enabled interactive, instruction-guided image editing with impressive fidelity and controllability [ 6 , 73 , 77 , 70 , 33 , 46 , 14 , 58 , 63 ] . Extending this paradigm from single images to videos, however, remains substantially more challenging. A practical instruction-guided video editor must (i) apply fine-grained semantic changes that follow the instruction, while (ii) preserving temporally coherent motion of the edited subject, background, and camera. In current models, these two requirements often conflict: aggressive semantic changes induce localized artifacts, identity drift, and texture popping, whereas enforcing temporal consistency can dilute the intended edit and reduce instruction fidelity (Fig. 1 top). This tension has been widely observed in diffusion-based video editing and adaptation works [ 64 , 41 , 32 , 39 ] . To mitigate these issues, a prevailing trend in existing approaches is to rely on injecting explicit external priors, such as VLM-extracted semantic conditions [ 35 , 52 ] or structural signals like skeletons and depth maps [ 75 , 9 ] . We argue that this over-reliance reflects a significant bottleneck, which constrains the diffusion backbone from learning inherent semantic-motion representations for precise semantic editing and faithful motion alignment with the source video dynamics. Instead, we attribute the core difficulty of instruction-guided video editing to the lack of factorization between semantic structure planning and motion modeling [ 38 , 7 , 25 , 1 , 16 ] . Semantic edits are typically sparse and temporally stable: a small number of anchor frames is often sufficient to determine the desired visual modification. In contrast, motion coherence follows physical and temporal dynamics that can be learned from large-scale raw videos without explicit editing supervision. Based on this observation, we propose SAMA (factorized S emantic A nchoring and M otion A lignment), a framework that encourages the model to learn semantic structure planning and motion modeling as two complementary capabilities. First, we introduce Semantic Anchoring which predicts semantic tokens together with video latents to support instruction-aware structural planning in the semantic space while retaining high-fidelity rendering in the latent space. Second, Motion Alignment strengthens temporal reasoning through motion-centric video restoration tasks, encouraging the backbone to internalize coherent temporal dynamics directly from raw videos. To realize this factorized learning paradigm, we train SAMA with a two-stage strategy. In the first stage, a factorized pre-training process encourages the model to internalize semantic anchoring and motion dynamics as two complementary capabilities, without requiring paired instruction-guided video editing data. Remarkably, we find that this stage alone already induces strong zero-shot video editing behavior. This observation suggests that robust instruction-guided video editing can naturally emerge once a model learns to jointly reason about semantic intent and temporal dynamics. In the subsequent supervised fine-tuning stage, the model is trained on paired video editing datasets to resolve residual semantic–motion conflicts and improve visual fidelity. Consequently, SAMA achieves state-of-the-art performance among open-source models while delivering results comparable to leading commercial systems (e.g. Kling-Omni [ 52 ] , Runway [ 43 ] ). • We propose a factorized perspective on instruction-guided video editing that separates semantic planning from motion modeling, reducing reliance on brittle external priors. • We introduce Semantic Anchoring and Motion Alignment via motion-centric video restoration pre-training, enabling the diffusion backbone to internalize robust semantic and temporal representations. • SAMA achieves state-of-the-art performance among open-source video editing models and is competitive with leading commercial systems. Code, models, and datasets will be publicly released. 2 Related Work 2.1 Instruction-Guided Video Editing Instruction-guided video editing aims to edit an input video following a text instruction, with the key challenge of preserving temporal consistency. Early diffusion-based attempts [ 15 , 39 , 13 , 66 , 44 , 12 , 64 , 32 ] in instruction-guided video editing mainly follow zero-shot or one-/few-shot paradigms, where pretrained text-to-image diffusion models are repurposed for videos with additional temporal modeling to maintain consistency. With the release of large-scale instruction-guided video editing datasets such as Señorita-2M [ 79 ] , InsViE-1M [ 65 ] , Ditto-1M [ 3 ] , ReCo-Data [ 76 ] , and OpenVE-3M [ 17 ] , recent research has shifted toward data-driven video editing models trained end-to-end. Ditto [ 3 ] builds its large-scale synthetic data pipeline by combining a strong image editing model with an in-context video generation model, and then trains a model on Ditto-1M to improve instruction-guided and temporal consistency. OpenVE-3M [ 17 ] expands supervision across diverse editing categories, while ReCo-Data [ 76 ] focuses on region-aware instruction editing to improve local controllability. Several recent works [ 69 , 29 , 22 , 50 , 10 , 30 , 23 , 67 , 76 ] further explore unified and in-context formulations for video editing. UNIC [ 69 ] unifies different video editing tasks by converting the noisy video latents, source video tokens, and multi-modal condition tokens into a single sequence, so a Diffusion Transformer can learn editing behaviors in-context without task-specific adapters or DDIM inversion. VACE [ 22 ] explores a unified and controllable editing formulation that supports diverse edit operations, improving the generality and robustness of instruction-guided video editing. ICVE [ 30 ] proposes a low-cost pretraining strategy that uses unpaired video clips to learn general editing ability in-context, and then refines the model with a small amount of paired editing data. EditVerse [ 23 ] proposes a unified framework for image/video generation and editing by representing text, images, and videos in a shared token space, enabling strong

Read the complete text