AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models
Muyao Niu, Mingdeng Cao, Yifan Zhan, Qingtian Zhu, Weihang Ran, Yanhong Zeng, Xiao Sun, Zhihang Zhong, Yinqiang Zheng · 2025
Read AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models by Muyao Niu, Mingdeng Cao, Yifan Zhan, Qingtian Zhu, Weihang Ran, Yanhong Zeng, Xiao Sun, Zhihang Zhong, Yinqiang Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20255.txt.
Introduction
Muyao Niu and eight co-authors bring flat portraits to life in fresh scenes in a 2025 paper. Video diffusion tools can move a drawn figure, but past tools leaned on stick poses or body meshes. Those guides fail when the backdrop moves or the figure must act with the scene in rich ways.
Their model, called AniCrafter, takes a portrait and a backdrop as joint guides, plus a set motion chain. It sews the figure into open, shifting scenes while the figure keeps its looks and follows the motion. Tests show more true-to-life clips than pose-led tools give.
Catalog record
| CREATOR | Muyao Niu, Mingdeng Cao, Yifan Zhan, Qingtian Zhu, Weihang Ran, Yanhong Zeng, Xiao Sun, Zhihang Zhong, Yinqiang Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20255
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models Muyao Niu, Mingdeng Cao, Yifan Zhan, Qingtian Zhu, Weihang Ran, Yanhong Zeng, Xiao Sun, Zhihang Zhong, Yinqiang Zheng Abstract . Recent advances in video diffusion models have substantially enhanced character animation techniques. However, existing methods primarily depend on structural conditions, such as DWPose or SMPL-X, to animate character images, which limits their effectiveness in open-domain scenarios involving dynamic backgrounds or complex character-scene interactions. This study presents AniCrafter, a diffusion-based human-centric animation model designed to seamlessly integrate and animate a given character within open-domain dynamic backgrounds while adhering to specified human motion sequences. Built upon advanced Image-to-Video (I2V) diffusion architectures, the model introduces an innovative “avatar-background” conditioning mechanism that reformulates open-domain human-centric animation as a restoration problem, thereby achieving versatile, occlusion-aware animation results. Experimental evaluations demonstrate that the proposed approach outperforms current state-of-the-art methods and exhibits an exceptional capability in handling challenging scenarios. Codes and model are available at: https://github.com/MyNiuuu/AniCrafter \correspondingauthor 1. Introduction Recent breakthroughs in generative modeling ( Ho et al., 2020 ; Rombach et al., 2022 ; Peebles and Xie, 2023 ) , particularly in Video Diffusion Models (VDMs) ( Yang et al., 2024b ; Blattmann et al., 2023 ; Xing et al., 2024 ; He et al., 2022 ; Chen et al., 2024b ) , have markedly advanced the field of dynamic video generation. Leveraging the strong generative capabilities of these models, numerous studies ( Hu, 2024 ; Niu et al., 2024 ; Zhang et al., 2024 ; Zhu et al., 2024 ; Yu et al., 2024 ; Shi et al., 2024 ; He et al., 2024 ) have explored controllable video generation guided by the prior knowledge embedded within VDMs. Figure 1. Within any open-domain dynamic scene, AniCrafter integrates and animates given characters with harmonized illumination and appearance, following specified motion sequences. In addition to SMPL-X (blue boxes), an “avatar-background” conditioning signal (yellow boxes) is introduced to enable robust and diverse human-centric animation. In addition, the model supports character-scene occlusion , character-object interaction , and novel motion . In the most closely related research domain, a range of approaches ( Hu et al., 2025 ; Wang et al., 2024c ; Wang et al., 2024b ; Xu et al., 2024b ; Chang et al., 2025 ; Cao et al., 2024 ; Lin et al., 2025 ; Niu et al., 2026 ; Zhan et al., 2025 ; Xu et al., 2025 ; Xu et al., 2025 ; Chen et al., 2024c ; Gong et al., 2025 ; Zhang et al., 2025b ) has been developed to animate static character images into motion sequences defined by representations such as DWPose ( Yang et al., 2023 ) , DensePose ( Güler et al., 2018 ) , SMPL-X ( Pavlakos et al., 2019 ) , or 3D skeletons ( Men et al., 2024 ) . Beyond animating human body parts, these methods effectively capture natural motion dynamics in non-rigid regions, such as hair or garments, which are difficult to model explicitly with conventional 3D techniques. Despite these strengths, existing approaches typically rely on basic structural conditioning and often exhibit structural distortions or temporal inconsistencies. Furthermore, most methods assume that the input character image and the generated video share the same static background, focusing solely on animating the character itself. Another technical paradigm ( Qiu et al., 2025a ; Qiu et al., 2024b ; Wen et al., 2025 ; Qiu et al., 2025b ) employs explicit 3D modeling techniques, particularly 3D Gaussian Splatting ( Kerbl et al., 2023 ) , to reconstruct an animatable 3D human avatar from a single reference image. In contrast to diffusion-based approaches that directly synthesize 2D videos, these methods adopt a two-stage pipeline: (1) constructing an explicit 3D human avatar using the segmented character image without the background, and (2) animating and rendering the 3D avatar into 2D image space according to 3D pose transformations such as SMPL-X. This pipeline produces multi-view-consistent outputs with strong structural integrity, owing to the geometric constraints of 3D warping and transformation. However, avatars reconstructed through these methods typically lack high-frequency appearance details and fail to reproduce realistic motion dynamics due to the inherent limitations of explicit 3D modeling, particularly for non-rigid components such as hair or garments. Moreover, the rendered characters often exhibit poor visual harmony when composited directly into background videos. Building upon the analysis of the two aforementioned paradigms, this study introduces AniCrafter , a human-centric video animation model capable of integrating and animating arbitrary characters within diverse backgrounds according to specified motion sequences. The key idea is to integrate the strengths of 3D human avatar-based modeling into diffusion-based frameworks, thereby enabling robust and realistic human-centric animation. Specifically, since 3DGS human avatars possess view-consistent geometry through explicit 3D representations, their 2D renderings provide 3D-consistent structural information and a pose-aligned coarse appearance that remains stable across frames. Given a reference character image, the corresponding human avatar is first reconstructed, then animated and rendered according to the specified motion sequences. The rendered avatar frames are subsequently integrated with the associated scene video to form an avatar-background conditioning sequence, which closely approximates the ground-truth video except for local degradations around the human region, such as the loss of high-frequency textures or temporal dynamics in elements like hair and loose garments. The proposed integration process also naturally considers the spatial relationships between the character and the scene ( e.g. , occlusions). With this effective conditioning signal, the diffusion process is reformulated as a restoration task, i.e. , 1) refining the degraded yet 3D-consistent appearance of human avatar renderings, and 2) synthesizing realistic non-rigid motion dynamics, such as the movement of hair and garments. A dedicated data processing pipeline and a multi-conditional architecture are designed, incorporating a relighting augmentation strategy and an occlusion-aware avatar–scene integration mechanism to fully exploit the avatar-background conditioning signal while leveraging the rich prior knowledge embedded in the pre-trained base model. Consequently, the proposed approach combines the advantages of both diffusion-based and 3D avatar-based paradigms, resulting in superior visual generation quality. The codes and data will be released to facilitate further exploration. 2. Related Work Figure 2. Data Processing Pipeline. The video DiT model generates the output video by conditioning on three distinct signals: the reference image, the SMPL-X sequence, and the composite “avatar–background” video. Video diffusion models. Driven by recent advances in generative modeling ( Ho et al., 2020 ; Song et al., 2020 ; Ho et al., 2022 ) , Video Diffusion Models (VDMs) have achieved remarkable progress in recent years ( Rombach et al., 2022 ; Kingma et al., 2013 ; Van Den Oord et al., 2017 ; Esser et al., 2021 ; HaCohen et al., 2024 ; Raffel et al., 2020 ; Radford et al., 2021 ; Liu et al., 2023 ; Li et al., 2024a ; Ronneberger et al., 2015 ; Shi et al., 2026 ; Zhan et al., 2026 ) . Early approaches ( He et al., 2022 ; Xing et al., 2024 ; Chen et al., 2023 ; Chen et al., 2024b ; Blattmann et al., 2023 ) predominantly adopted UNet ( Ronneberger et al., 2015 ) , whereas more recent efforts ( Hong et al., 2022 ; Yang et al., 2024b