Learning to Animate Images from A Few Videos to Portray Delicate Human Actions
Haoxin Li, Yingchen Yu, Qilong Wu, Hanwang Zhang, Song Bai, Boyang Li · 2025
Read Learning to Animate Images from A Few Videos to Portray Delicate Human Actions by Haoxin Li, Yingchen Yu, Qilong Wu, Hanwang Zhang, Song Bai, Boyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00276.txt.
Introduction
Haoxin Li and co-authors taught a system in 2025 to animate a still photo into a video of a delicate human action, learning from very few example videos. Making fine actions, such as subtle hand moves, is hard for video generators, above all for rare actions with little training data.
Their method, called FLASH, forces the model to rebuild one video using the motion of another video that shows the same action with a different look. That push makes the motion lessons transfer to new images and stops the model from memorizing the few examples. Extra decoder layers carry detail from the photo into each frame. Human judges preferred its videos in about two thirds of comparisons.
Catalog record
| CREATOR | Haoxin Li, Yingchen Yu, Qilong Wu, Hanwang Zhang, Song Bai, Boyang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Animate Images from A Few Videos to Portray Delicate Human Actions Haoxin Li, Yingchen Yu, Qilong Wu, Hanwang Zhang, Song Bai, Boyang Li Abstract Despite recent progress, video generative models still struggle to animate static images into videos that portray delicate human actions, particularly when handling uncommon or novel actions whose training data are limited. In this paper, we explore the task of learning to animate images to portray delicate human actions using a small number of videos—16 or fewer—which is highly valuable for real-world applications like video and movie production. Learning generalizable motion patterns that smoothly transition from user-provided reference images in a few-shot setting is highly challenging. We propose FLASH ( F ew-shot L earning to A nimate and S teer H umans), which learns generalizable motion patterns by forcing the model to reconstruct a video using the motion features and cross-frame correspondences of another video with the same motion but different appearance. This encourages transferable motion learning and mitigates overfitting to limited training data. Additionally, FLASH extends the decoder with additional layers to propagate details from the reference image to generated frames, improving transition smoothness. Human judges overwhelmingly favor FLASH, with 65.78% of 488 responses prefer FLASH over baselines. We strongly recommend watching the videos in the Webpage 1 1 1 https://lihaoxin05.github.io/human_action_animation/ , as motion artifacts are hard to notice from images. 1 Introduction Despite substantial progress [ 33 , 75 , 110 , 26 , 86 , 11 , 99 , 50 , 102 , 28 , 91 , 87 , 85 , 98 , 45 ] , video generative models still struggle to accurately portray delicate human actions, especially when they are required to start from a user-provided reference image. Even commercial AI video generators trained on large-scale datasets, such as KLING AI 2 2 2 https://www.klingai.com/image-to-video and Wanx AI 3 3 3 https://tongyi.aliyun.com/wanxiang/videoCreation , encounter difficulty with this task. As shown in Figure 1 , both fail to animate actions such as balance beam jump or shooting a soccer ball. We attribute the difficulty mainly to the interplay between action complexity and training data scarcity. Due to skeletal and joint structures, human actions are highly complex and constrained in unique ways. For example, the range of forearm pronation is typically 80-90°, whereas the neck can do 150° rotation and 125° flexion [ 5 ] . Inferring such precise specifications from videos alone is a difficult inverse problem. To make things worse, human action videos follow a long-tailed distribution [ 104 ] , which means that, for a wide range of human actions, only a small amount of data are available for learning. The action recognition community has long recognized this problem and created a body of literature on few-shot and open-vocabulary action recognition [ 101 , 65 , 37 , 51 ] . Interestingly, few-shot generation of human actions remains under-investigated. Figure 1 : Comparison of animated human action videos produced by KLING AI, Wanx AI and FLASH (our method). In the balance beam jump action, Wanx AI produces physics-defying movements, whereas KLING AI generates a jump but fails to portray the standard jump on the balance beam. For the soccer shooting action, both KLING AI and Wanx AI struggle to generate the correct shooting motion and the person never kicks the ball away. In contrast, FLASH successfully generates actions that resemble the real-world actions in the last row. We strongly recommend watching the animated videos in the Webpage, as motion artifacts can be hard to notice from static images. In this paper, we explore the task of animating an image to portray delicate human actions by learning from a small set of videos. The problem input includes a static reference image as well as a textual prompt describing the action. We learn from up to 16 videos for each action class, thereby reducing the need for extensive video data collection. The animation should begin exactly with the reference image, giving users precise control over the initial state of the video. This capability is particularly valuable for applications like video and movie production, which need to animate diverse actors performing uncommon or newly designed actions, often with only a few example videos available and requiring precise control over the initial states of the action, such as the actors’ orientations and spatial arrangement of the scenes. Controllable generation has been studied for both images [ 63 , 74 , 103 ] and videos [ 27 , 99 , 55 ] , but the additional constraint also renders the problem more difficult. Existing image animation methods encounter considerable difficulties with this task. These approaches typically rely on large video datasets for training and primarily focus on preserving the appearance of the reference images [ 94 , 23 , 40 , 18 , 84 , 25 , 56 , 68 , 18 , 106 , 24 ] or learning spatial-temporal conditioning controls ( e.g . , optical flows) to guide image animation [ 60 , 41 , 73 ] . However, with limited training data, these methods suffer from overfitting and struggle to learn generalizable motion patterns. Customized video generation methods [ 58 , 90 , 108 , 47 ] can learn target motion from a few examples but fall short in the ability to align the action with the user-provided reference image. We propose FLASH ( F ew-shot L earning to A nimate and S teer H umans), a framework for few-shot human action animation. To learn generalizable motion patterns from limited videos, FLASH introduces the Motion Alignment Module, which forces the model to reconstruct a video using the motion features of another video with the same motion but different appearance. This facilitate the learning of generalizable motion features and reduces overfitting. Additionally, FLASH employs a Detail Enhancement Decoder to propagate multi-scale details from the reference image to the generated frames, providing smooth transition from the first reference frame. Experiments on 16 delicate human actions demonstrate that FLASH accurately and plausibly animates diverse reference images into lifelike human action videos. Human judges overwhelmingly favor FLASH; 65.78% of 488 responses prefer FLASH over open-source baselines. FLASH also outperforms existing methods across automatic metrics, and generalizes to non-realistic figures like cartoon characters or humanoid aliens. 2 Related Work Video Generation. Video generation using diffusion models [ 31 , 78 , 77 ] have notably surpassed methods based on GANs [ 19 ] , VAEs [ 44 ] and flow techniques [ 6 ] . Diffusion models for video generation can be broadly classified into two groups. The first group generates videos purely from text descriptions. These methods extend advanced text-to-image generative models by integrating 3D convolutions, temporal attention layers, or 3D full attention layers to capture temporal dynamics in videos [ 33 , 32 , 75 , 110 , 2 , 26 , 86 , 98 , 45 ] . To mitigate concept forgetting when training on videos, some methods use both videos and images jointly for training [ 33 , 4 , 45 ] . Large Language Models (LLMs) contribute by generating frame descriptions [ 21 , 35 , 48 ] and scene graphs [ 13 ] to guide the video generation. Trained on large-scale