Mojito: Motion Trajectory and Intensity Control for Video Generation
Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang · 2024
Read Mojito: Motion Trajectory and Intensity Control for Video Generation by Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08948.txt.
Introduction
Xuehai He and co-authors present Mojito, a video model that takes orders on motion path and strength in two thousand twenty-four. Text-to-video tools now make fine clips, but steering how an object moves, and how hard, stays hard. Training such control in is also costly.
A direction module steers the object's path through cross-attention, with no extra training. A second module reads flow maps from real clips to grade motion strength and guide the output. Tests show the model follows set paths and strength levels closely while staying cheap to run. The motion it makes matches natural movement in real scenes, which suits ads, film drafts, and design work.
Catalog record
| CREATOR | Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08948
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mojito: Motion Trajectory and Intensity Control for Video Generation Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang Abstract Recent advancements in diffusion models have shown great promise in producing high-quality video content. However, efficiently training video diffusion models capable of integrating directional guidance and controllable motion intensity remains a challenging and under-explored area. To tackle these challenges, this paper introduces Mojito, a diffusion model that incorporates both mo tion tra j ectory and i ntensi t y contr o l for text to video generation. Specifically, Mojito features a Directional Motion Control (DMC) module that leverages cross-attention to efficiently direct the generated object’s motion without training , alongside a Motion Intensity Modulator (MIM) that uses optical flow maps generated from videos to guide varying levels of motion intensity. Extensive experiments demonstrate Mojito’s effectiveness in achieving precise trajectory and intensity control with high computational efficiency, generating motion patterns that closely match specified directions and intensities, providing realistic dynamics that align well with natural motion in real-world scenarios. Project page: https://sites.google.com/view/mojito-video . Figure 1: Mojito generates videos that accurately follow specified directions, locations, and trajectories, while adapting to varying input motion intensities. (a) Directional Motion Control : the object ( leaf, firefly ) in the generated videos can follow input bounding boxes or trajectories over time. (b) Motion Intensity Modulator : increasing input motion intensity levels results in a corresponding increase in motion, transforming a relatively static scene into one with more dynamic movement. Additional examples can be found at https://sites.google.com/view/mojito-video . 1 Introduction “Movement is life; control brings mastery.” —Adapted from Aristotle’s philosophy Figure 2: Overview of the Mojito framework. In the training pipeline (top), Mojito uses a VAE Encoder to transform input frames into latent features, processed by Spatial and Temporal Transformers within the U-Net. Motion intensity control is introduced through the Motion Intensity Modulator , consisting of the Optical Flow Map Generator and the Motion Intensity Encoder. The Directional Motion Control module interprets object phrases within the prompt to align attention with specified trajectories. During inference (bottom), Mojito generates videos following user-defined motion intensity and directional guidance. Text-to-video (T2V) generation ( Blattmann et al., 2023a ; He et al., 2022 ; Zheng et al., 2024 ; Lab & etc., 2024 ; Xu et al., 2024 ; Yang et al., 2024 ; Vchitect, 2024 ) aims to produce diverse, high-quality videos with given text prompts. Unlike image generation ( Rombach et al., 2022 ; Saharia et al., 2022b ; Nichol et al., 2021 ; Ramesh et al., 2021 ; Yu et al., 2022 ; Chang et al., 2023 ) , which produces a single static frame, video generation can extend beyond visual synthesis and has the potential to serve as a world modeling tool ( NVIDIA et al., 2025 ; Xiang et al., 2024 ; Wang et al., 2024a ; Kang et al., 2024 ) and a physical AI engine ( Authors, 2024 ) for simulating real-world dynamics. As world modeling expects video generation models to control and generate the motion of physical objects, enabling realistic interactions with the environment, existing T2V models—despite their creative and powerful generative capabilities—struggle to produce large motion magnitudes ( Zheng et al., 2024 ; Lab & etc., 2024 ; Pika, 2024 ; Wang et al., 2023a ) or adaptable motion intensities (i.e., motion speed and magnitude) that align with user intent ( NVIDIA et al., 2025 ; Chen et al., 2024a ) , limiting their applicability. Towards this, in this work, we aim to address a new research question: Can text-to-video models be trained efficiently to control both motion direction and intensity in alignment with user intent? The primary limitations of current models stem from several key challenges. First , existing diffusion-based T2V models—whether U-Net or DiT-based ( Peebles & Xie, 2022 ) —rely on diffusion processes in either pixel space or latent visual feature space during training, but lack dedicated motion modeling mechanisms. Consequently, they inherently struggle to adjust object motion in generated videos solely through text prompts and often fail to maintain temporal consistency in moving objects, resulting in flickering, trajectory inconsistencies, and visual artifacts across frames; Second , capturing relative motion data is inherently complex in real-world videos due to the simultaneous movements of both cameras and objects, leading to lack of video data in training. There is also a lack of large-scale, annotated datasets specifically for motion direction and intensity. Existing video datasets rarely include detailed labels for motion dynamics, and obtaining human annotations is labor-intensive; Third , training T2V models with such detailed annotations would also bring substantial computational resources. To tackle these challenges, we present Mojito, the first diffusion model for text-to-video generation that simultaneously integrates text-based prompts with flexible motion controls, allowing for the precise modulation of both motion direction and intensity. Figure 1 showcases how Mojito enables precise control over both motion trajectory and intensity. To achieve this, Mojito introduces two novel modules apart of our trained video generation backbone: (1) the Motion Intensity Modulator (MIM), which encodes input motion intensities as features and integrates them seamlessly into the diffusion framework during training. (2) the Directional Motion Control (DMC) module, which adjusts object motion direction through cross-attention guidance without training , aligning object trajectories with specified paths. Our contributions are summarized as follows: • We propose Mojito, a novel diffusion-based transformer framework for text-to-video generation, integrating both trajectory/direction-aware motion direction control and motion intensity modulation within a unified architecture. • We introduce a training-free, test-time Directional Motion Control (DMC) method that grants text-to-video generation with user-specified motion trajectories or directions. Our study explores how to effectively guide attention during video generation by analyzing attention maps and identifying the optimal strategy for steering motion using user-defined inputs, such as bounding boxes, during the forward diffusion process. Furthermore, we show that DMC is a model-agnostic approach that generalizes across different video generation architectures, including U-Net-based and DiT-based ( Peebles & Xie, 2022 ) models. • To tackle object inconsistency in generated videos under directional motion control, we introduce a novel temporal smoothness function, optimizing it as it updates latent features during the forward diffusion process to enhance coherence across frames. • We develop a training-based Motion Intensity Modulator (MIM) that seeks to achieve precise control over motion intensity by leveraging optical flow ( Horn & Schunck, 1981 ) , ensuring generated videos align with user-defined motion intensity levels. Additionally, we explore three alternative designs for MIM. We find that combining motion intensity embeddings with text embeddings yields the best performance • We demonstrate that Mojito not only generates high-quality videos comparable to state-of-the-art models but also provides effective and efficient motion control. Through ablations, we provide insights into designing efficient, motion-controllable video generation techniques, shedding light on advancing future motion-augmented video generative models. 2 Related Works Figure 3: Overview of the Directional Motion Control module. The cross-attention map for the chosen word token in the given guidance step is marked with a red border. We compute the loss and perform backpropagation during inference time to update latents. 2.1 Diffusion-based Text to Video Generation With the rapid development of generative models, particularly diffusion models, numerous breakthroughs