Identity-Preserving Pose-Guided Character Animation via Facial Landmarks Transformation
Lianrui Mu, Xingze Zou, Wenjie Zheng, Jiangnan Ye, Haoji Hu · 2024
Read Identity-Preserving Pose-Guided Character Animation via Facial Landmarks Transformation by Lianrui Mu, Xingze Zou, Wenjie Zheng, Jiangnan Ye, Haoji Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08976.txt.
Introduction
Lianrui Mu, Xingze Zou, Wenjie Zheng, Jiangnan Ye, and Haoji Hu wrote a 2024 computer vision paper on animating a character's photo with someone else's movements without losing the face. Pose guided animation copies motion from a driving video onto a still portrait. In lively scenes like dancing, the face often drifts or warps, because the driver's facial landmarks do not fit the reference face's geometry.
Their fix passes the landmarks through three dimensions. A morphable face model lifts the flat points into a 3D face, reshapes that face to match the reference identity, then flattens it back to guide the animation. Experiments showed identity held up far better through complex motion. The method targets games, film, and virtual avatars that must stay recognizably themselves.
Catalog record
| CREATOR | Lianrui Mu, Xingze Zou, Wenjie Zheng, Jiangnan Ye, Haoji Hu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08976
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Identity-Preserving Pose-Guided Character Animation via Facial Landmarks Transformation Lianrui Mu, Xingze Zou, Wenjie Zheng, Jiangnan Ye, Haoji Hu Abstract Creating realistic pose-guided image-to-video character animations while preserving facial identity remains challenging, especially in complex and dynamic scenarios such as dancing, where precise identity consistency is crucial. Existing methods frequently encounter difficulties maintaining facial coherence due to misalignments between facial landmarks—extracted from driving videos that provide head pose and expression cues—and the facial geometry of the reference images. To address this limitation, we introduce the F acial L andmarks T ransformation (FLT) method, which leverages a 3D Morphable Model to address this limitation. FLT converts 2D landmarks into a 3D face model, adjusts the 3D face model to align with the reference identity, and then transforms them back into 2D landmarks to guide the image-to-video generation process. This approach ensures accurate alignment with the reference facial geometry, enhancing the consistency between generated videos and reference images. Experimental results demonstrate that FLT effectively preserves facial identity, significantly improving pose-guided character animation models. † † address: Zhejiang University 1 Introduction Pose-guided character animation generation has emerged as a pivotal research area, driven by its extensive applications in virtual characters, animation, and video production. By synthesizing videos from reference images, such methods enable the creation of highly personalized content tailored to diverse user requirements. Early approaches [ 1 ] leveraging Generative Adversarial Networks (GANs), and utilizing UV coordinates to map appearance and labels to a unified representation, achieved notable progress in producing realistic and diverse visual content. However, these methods still exhibit shortcomings in synthesis quality. Figure 1: We propose a facial landmark transformation approach using 3D face reconstruction. Our method aligns the driving image’s landmarks with a reference face, significantly improving identity consistency in pose-guided generation, even under large facial geometry differences. Recent diffusion-based video generation techniques [ 2 ] have surpassed typical GAN-based solutions in visual quality. Researchers have leveraged the robust generative capabilities of diffusion models. This has enabled fine-grained control over character movements and scene layouts in pose-guided image-to-video tasks. Notable examples include MagicPose [ 3 ] , which employs a two-stage training strategy to disentangle appearance and pose, thereby enhancing identity consistency. AnimateAnyone [ 4 ] improves temporal coherence through a spatial attention module (ReferenceNet) and a pose guider. ControlNeXt [ 5 ] introduces a streamlined control architecture, reducing computational overhead and enabling flexible content manipulation. Despite these advancements, identity preservation remains problematic. This issue arises when driving facial landmarks, often extracted from real human videos, deviate significantly from the facial geometry of the reference image. Such mismatches often cause the generated face to inherit features from the driving identity, deviating from the intended reference. To address this issue, we propose our F acial L andmarks T ransformation (FLT) method, a training-free , plug-and-play solution that seamlessly integrates into frameworks using facial landmarks as generation conditions. FLT fuses face identity information from a reference image with the expressions and poses from a driving image . This is achieved by aligning the driving landmarks with the reference identity before feeding them into the video generation model. Leveraging a 3D Morphable Model by Huber et al. [ 6 ] , FLT parametrizes and adjusts facial shape and expression. It then re-renders the modified 3D face back to 2D, producing transformed landmarks that better capture the reference identity. As illustrated in Fig. 1 , FLT excels in scenarios where the driving and reference identities exhibit large facial geometry differences, effectively preserving facial contours and enhancing overall identity retention. We evaluate FLT on two pose-guided animation models, AnimateAnyone [ 4 ] and ControlNeXt [ 5 ] , using the TikTok [ 7 ] and UBC Fashion [ 8 ] datasets. Our experiments confirm that FLT significantly improves identity preservation in challenging motion settings. The main contributions of this work are summarized as follows: • We propose the F acial L andmarks T ransformation (FLT), providing accurate facial guidance to maintain the consistency of the reference character’s identity. • We develop a training-free and plug-and-play tool applicable to various pose-guided character animation generation models, effectively enhancing the performance of identity preservation in our experiments. To contribute to the community, we have open-sourced our approach . 2 Related Work 2.1 3D Morphable Models 3D Morphable Models (3DMMs) [ 9 ] were originally proposed by Blanz and Vetter to represent and manipulate 3D facial structures by parameterizing both shape and texture. This representation has proven effective for various face-related tasks, such as recognition, expression manipulation, and facial animation, as it allows 3D face reconstruction from single or multiple 2D images. Subsequent works have combined 3DMMs with deep learning to enhance accuracy and robustness in 3D face reconstruction [ 10 , 11 ] . In video generation and editing, 3DMM-based strategies ensure facial coherence across frames by aligning model parameters with target features [ 12 ] . This alignment is pivotal in preserving identity, especially when a subject undergoes complex movements or diverse expressions. In our work, we build upon a 3DMM framework developed by Huber et al. [ 6 ] , which incorporates a large-scale 3D scan dataset covering diverse facial geometries. It supports convenient PCA-based shape manipulation and includes an expression blendshape model for linear expression blending. By leveraging these properties, our method transforms 2D landmarks into a 3D face mesh and adapts both shape and expression parameters, ultimately generate transformed landmarks to guide identity-preserving video synthesis. 2.2 Pose-guided Character Generation Recent advancements in pose-guided character generation have made notable progress in addressing facial consistency and identity preservation challenges. Approaches like DreamPose [ 13 ] incorporate character features and pose embeddings, allowing direct concatenation to guide the overall image structure. DisCo [ 14 ] focuses on spliting foreground and background for stable video synthesis but relies solely on pose skeletons, overlooking detailed facial cues. Recent methods that incorporate 68-point facial landmarks—such as MagicPose [ 3 ] , AnimateAnyone [ 4 ] , and ControlNeXt [ 5 ] have significantly advanced facial detail retention and identity preservation in pose-guided generation. MagicPose [ 3 ] adopts a two-stage training strategy that disentangles appearance and pose, thereby enabling high-fidelity animations without the need for fine-tuning. By learning separate representations for identity and motion, MagicPose ensures that changes in pose and facial expressions do not degrade identity consistency. AnimateAnyone [ 4 ] introduces a spatial attention module (ReferenceNet) and a pose guider, which work in tandem to maintain detailed appearance features while enforcing accurate pose control across consecutive frames. This design not only improves temporal coherence but also upholds a subject’s unique facial traits, thereby strengthening identity preservation throughout the animation sequence. ControlNeXt [ 5 ] provides a lightweight control mechanism that streamlines the generative architecture. Although primarily focused on computational efficiency and flexible content manipulation, ControlNeXt also enhances facial identity fidelity by injecting pose and motion signals in a way that minimally disrupts the subject’s intrinsic facial characteristics. Although existing methods have made progress, they still face the issue where, when driving landmarks differ from the reference geometry, the generated face often inherits unwanted driving face features. Our Facial Landmarks Transformation (FLT) merges the reference shape with the driving expressions in 3D, thereby boosting identity fidelity in pose-guided