Video Motion Graphs
Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou, Yang Zhou · 2025
Read Video Motion Graphs by Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou, Yang Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20218.txt.
Introduction
Haiyang Liu and co-authors build Video Motion Graphs, a system that assembles new human motion videos from old clips. Given a source video plus cues such as music or motion tags, the system hunts for clips whose moves fit the cues. It then has to stitch those clips into one smooth flow. Their 2025 paper centers on the stitch step.
Their stitch model, HMInterp, works in two branches. One branch fills in the skeleton path between clips. The other paints the final frames. Training feeds strong and weak cues in stages so both guide the result. Even hard cases like dance cuts came out smooth. The team reports wins over both pure generation and pure retrieval rivals.
Catalog record
| CREATOR | Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou, Yang Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Motion Graphs Haiyang Liu, Zhan Xu, Fa-Ting Hong, Hsin-Ping Huang, Yi Zhou, Yang Zhou Abstract We present Video Motion Graphs , a system designed to generate realistic human motion videos. Using a reference video and conditional signals such as music or motion tags, the system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating interpolation frames to seamlessly connect clip boundaries. The core of our approach is HMInterp , a robust Video Frame Interpolation (VFI) model that enables seamless interpolation of discontinuous frames, even for complex motion scenarios like dancing. HMInterp i) employs a dual-branch interpolation approach, combining a Motion Diffusion Model for human skeleton motion interpolation with a diffusion-based video frame interpolation model for final frame generation. ii) adopts condition progressive training to effectively leverage identity strong and weak conditions, such as images and pose. These designs ensure both high video texture quality and accurate motion trajectory. Results show that our Video Motion Graphs outperforms existing generative- and retrieval-based methods for multi-modal conditioned human motion video generation. Project page can be found here . Figure 1 : Video Motion Graphs is a system to generate human motion videos from a reference video and conditional signals such as music, action tags and sparse keyframes. The video is generated by first retrieving matched video clips from reference video and then generating interpolation frames between clips to smooth the transition boundaries. 1 Introduction Human motion videos play a vital role across numerous industries, including entertainment, virtual reality, and interactive media. However, capturing high-quality, realistic motion videos can be labour-intensive and costly. Recent advancements in video generation offer solutions in generating human motion videos based on inputs like skeletal animation, action labels, and speech audios, making production more efficient and customizable. Human motion video generation has two primary approaches: generative- and retrieval-based methods. Generative-based models [ 9 , 55 , 23 , 18 ] synthesize all frame pixels from conditional inputs, offering flexibility in generating diverse motions. However, they often produce artifacts for complex contents, such as human distorted limbs, fingers, etc [ 1 , 62 ] . Retrieval-based models utilize key frames from given reference videos, and generate interpolation frames to ensure smooth transitions [ 67 , 33 ] . While they require reference material, they typically deliver higher video quality and maintain the actor’s identity. Motivated by the video quality, we focus on the retrieval-based method for real-world applications. Existing retrieval-based methods, such as, Gesture Video Graph (GVR) [ 67 ] and TANGO [ 33 ] , are specifically designed for co-speech gesture video generation and are not applicable to general human motion animation, e.g. , dancing, kung fu, etc. In particular, these methods retrieve video frames based on input audio using a motion graph framework [ 26 ] , and then apply a Video Frame Interpolation (VFI) model to generate the interpolated frames between the retrieved frames. Extending these methods to a comprehensive system to accommodate diverse conditions beyond speech audio presents one main challenges: Their VFI model leverages the linear blended motion guidance and thus limits its ability to handle complex, dynamic motions like human dancing. In this paper, we proposed an improved diffusion-based VFI model, HMInterp , to complete the essential functionality of the general video motion graphs system. The motivation of HMInterp is to seamlessly connect retrieved frames. Unlike the original VFI module in GVR, which uses a simple linearly interpolated motion as guidance in a flow-warp-based VFI model, HMInterp addresses the limitations of linear interpolation for complex, dynamic motions. For instance, approximately 78% of mild speech gestures can be reasonably approximated through linear blending, while only 17% of dance gestures which complex motion can, highlighting the need for a more advanced approach for human dynamic motion sequences (see supplemental for details). Specifically, HMInterp starts from UNet-based pretrained text-to-video model, AnimateDiff [ 14 ] , and further consists of a diffusion-based VFI model with motion guidance from a dedicated Motion Diffusion Model (MDM). HMInterp generates smooth interpolated frames while preserving accurate human structure and motion, guided by the MDM module. Based on [ 48 ] , the MDM is trained within a validated human skeleton space, ensuring both structural integrity and continuous motion of body parts. Moreover, we observed straightforward multi-condition (image and pose) joint training yields identity inconsistent results. To solve this, we introduce condition progressive training. It adopts different training orders and iterations for strong and weak conditions such as pose and image, respectively. Finally, to enhance the performance, we incorporate the ReferenceNet from [ 18 , 60 ] and reference decoder from [ 58 ] . As a consequence, these components enable HMInterp to produce realistic interpolated frames. The quality of HMInterp enable us to implement the Video Motion Graphs for general human video representation (in Figure 1 ). From an engineering perspective, we complete the audio-only GVR baseline [ 67 ] into a more comprehensive system. In particular, we define a standard four-stage pipeline for graph initialization, searching, frame interpolation and background reorganization. Besides, we adopt task-specific rule-based matching for searching, and introduce keyframe-based editing. This allows the system to retrieve relevant video clips to align with conditions like music beats. It also enables users to replace or customize specific frames by manually editing the output frames. Overall, our contributions can be concluded as follows: • We propose Video Motion Graphs , a comprehensive retrieval + generation system for general human motion videos. This system enables both sequence and keyframe retrieval, supporting applications such as real-time video generation and keyframe editing. It achieves state-of-the-art performance in generating high-quality, customizable human motion videos. • We propose HMInterp , a high-quality motion-aware, video frame interpolation module with the proposal that i) utilizes Motion Diffusion Model for generative and controllable human structure guidance. ii) adopts condition progressive training to effectively leverage identity strong and weak conditions. Figure 2 : System Pipeline of Video Motion Graphs. Given input reference video(s) and a condition signal like music, Video Motion Graphs generates a new video in four steps: (i) representing the video as a directed graph, where nodes are RGB frames and edges indicate valid playback transitions, (ii) retrieving a frame playback path in the graph to match conditions based on task-specific rules, such as beat alignment, (iii) blending discontinuous frame transitions with a Motion-Aware Frame Interpolation Model, and (iv) optionally changing the video background through background removal and generation models. 2 Related Work Generative Human Motion Video Synthesis. Generative methods generate all frames directly from networks with conditions. To the best of our knowledge, there is no unified model to accept flexible conditions and output general human motions. There are task-specific models, such as generating motion video from text [ 23 ] , music [ 55 ] , speech [ 9 , 29 , 15 , 31 , 34 , 32 ] , and pose [ 64 , 18 , 20 , 4 , 5 , 68 ] . These methods are flexible to generate novel poses, but the video quality is sub-optimal. The video quality depends on the video generation backbone [ 46 ] . Even though the backbone has moved from UNet [ 46 , 6 , 2