Text-based Animatable 3D Avatars with Morphable Model Alignment
Yiqian Wu, Malte Prinzler, Xiaogang Jin, Siyu Tang · 2025
Read Text-based Animatable 3D Avatars with Morphable Model Alignment by Yiqian Wu, Malte Prinzler, Xiaogang Jin, Siyu Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15835.txt.
Introduction
Making a moving 3D head from a line of text is a prized goal for games, film, and virtual helpers. Yiqian Wu, Malte Prinzler, Xiaogang Jin, and Siyu Tang present AnimPortrait3D in this 2025 paper, a method built around a standard morphable face model. Older methods often made heads that looked off or moved in odd ways.
The authors trace those flaws to loose guidance from the text and weak alignment with the face model. They start from a pretrained text-to-3D model to set a strong base shape and rig, then refine expressions with a control network fed by maps from the face model. Tests show gains in image quality, alignment, and motion faithfulness over prior methods.
Catalog record
| CREATOR | Yiqian Wu, Malte Prinzler, Xiaogang Jin, Siyu Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15835
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text-based Animatable 3D Avatars with Morphable Model Alignment Yiqian Wu, Malte Prinzler, Xiaogang Jin, Siyu Tang Abstract . The generation of high-quality, animatable 3D head avatars from text has enormous potential in content creation applications such as games, movies, and embodied virtual assistants. Current text-to-3D generation methods typically combine parametric head models with 2D diffusion models using score distillation sampling to produce 3D-consistent results. However, they struggle to synthesize realistic details and suffer from misalignments between the appearance and the driving parametric model, resulting in unnatural animation results. We discovered that these limitations stem from ambiguities in the 2D diffusion predictions during 3D avatar distillation, specifically: i) the avatar’s appearance and geometry is underconstrained by the text input, and ii) the semantic alignment between the predictions and the parametric head model is insufficient because the diffusion model alone cannot incorporate information from the parametric model. In this work, we propose a novel framework, AnimPortrait3D , for text-based realistic animatable 3DGS avatar generation with morphable model alignment, and introduce two key strategies to address these challenges. First, we tackle appearance and geometry ambiguities by utilizing prior information from a pretrained text-to-3D model to initialize a 3D avatar with robust appearance, geometry, and rigging relationships to the morphable model. Second, we refine the initial 3D avatar for dynamic expressions using a ControlNet that is conditioned on semantic and normal maps of the morphable model to ensure accurate alignment. As a result, our method outperforms existing approaches in terms of synthesis quality, alignment, and animation fidelity. Our experiments show that the proposed method advances the state of the art in text-based, animatable 3D head avatar generation. Code and model for this paper are at AnimPortrait3D . PAPERS_666 Journal: TOG CCS: Computing methodologies Computer vision Figure 1. Our method generates high-quality, realistic, and animatable 3D avatars from text descriptions which can be driven with morphable model parameters. We show the rendered results of our generated avatars from various camera angles, with expressions and body poses sampled from the NeRSemble dataset ( Kirschstein et al., 2023 ) . 1. Introduction The creation of 3D human avatars is an extensively researched topic with applications in gaming, filmmaking, and social media. Text-driven methods have demonstrated impressive results in generating 3D avatars using 2D diffusion models ( Prinzler et al., 2024 ; Wang et al., 2023 ; Massague et al., 2024 ; Zhang et al., 2023b ; Shi et al., 2024 ; Huang et al., 2024 ; Kolotouros et al., 2023 ) . Furthermore, parametric models ( Pavlakos et al., 2019 ; Li et al., 2017 ; Paysan et al., 2009 ) offer a viable approach for creating animatable avatars ( Zhou et al., 2024 ; Liao et al., 2024 ; Liu et al., 2024 ) for subsequent applications. However, existing diffusion-model-based generation methods primarily rely on algorithms such as score distillation sampling (SDS) ( Poole et al., 2023 ) to integrate the 2D information from diffusion models into a 3D avatar, introducing ambiguities that reduce the quality of the results. These ambiguities stem from two major factors: i) First, a single text prompt in a diffusion model maps to a wide range of images, resulting in underconstrained appearance and geometry guidance. This leads to blurred results and the “Janus” problem ( Poole et al., 2023 ; Wang et al., 2023 ) . Portrait3D ( Wu et al., 2024b ) addresses this by utilizing an appearance-geometry joint prior. However, the results exhibit noisy geometry and are restricted to static reconstructions without animation capabilities. ii) Second, the diffusion model’s predictions typically align poorly with the underlying parametric model which causes artifacts during animation. HeadStudio ( Zhou et al., 2024 ) aims to solve this by using a landmarks-based ControlNet for guidance, but the 3D information provided by landmarks alone remains insufficient for robust alignment. The primary challenge is to incorporate robust guidance that is aware of both geometry and semantic information from the parametric model into the optimization process. In this paper, we present a novel framework, AnimPortrait3D , for creating animatable 3D avatars from text descriptions, achieving realistic appearance and geometry while maintaining accurate alignment with the underlying parametric mesh. We choose 3DGS ( Kerbl et al., 2023 ) for our 3D representation due to its efficient rendering capabilities and animation flexibility. As shown in Figure 2 , our method consists of two stages: an initialization stage that eliminates appearance ambiguities by creating a well-defined initial avatar, and a dynamic optimization stage that produces highly detailed results and eliminates animation artifacts. During the first stage, we initialize our avatar from the static text-to-3D model Portrait3D ( Wu et al., 2024b ) which provides high-quality appearance and geometry information. Specifically, we fit the SMPL-X model ( Pavlakos et al., 2019 ) to the initialized avatar- to enable animation in the second stage of our optimization procedure later on. To handle hair and clothing, which are not modeled by SMPL-X, we extract a noisy mesh from the static avatar and refine it using estimated normal maps. The refined mesh is then segmented into hair and clothing components using pretrained semantic segmentation networks. Then we texturize the SMPL-X, hair, and clothing meshes with 3D Gaussians and optimize their appearance features using multi-view images rendered from the static Portrait3D avatar. We adopt the rigging of SMPL-X to animate the avatar. However, animating the avatar with SMPL-X rigging introduces artifacts due to its optimization in a static setting. To address this, we introduce the second stage in which we optimize the avatar for dynamic poses and expressions using a 2D diffusion model. Specifically, we train a ControlNet ( Zhang et al., 2023a ) to provide geometry- and semantics-aware guidance based on the SMPL-X model’s normal and segmentation maps as conditions. For challenging areas with complex geometry and frequent occlusions, such as the eyelids and mouth interiors, we introduce a pre-training strategy to refine color details. For the eye region, we leverage the diffusion model and ControlNet to generate a refined eye image, which is used to optimize the eye area of the 3D avatar. For the mouth region, we calculate the Interval Score Matching (ISM) loss ( Liang et al., 2024 ) using the diffusion model and ControlNet to refine the mouth area. The full 3D avatar is then optimized with the ISM loss. To eliminate minor artifacts and enhance the final quality, we further apply a refinement strategy by optimizing the 3D avatar with images refined by the diffusion model. We demonstrate that our method can generate high-quality, animatable 3D avatars from text, achieving superior results in challenging areas such as the mouth interiors and eyes, and establishing robust rigging relationships with the corresponding parametric model. In summary, our work makes the following key contributions: • A novel framework, AnimPortrait3D, for generating animatable, text-based 3D avatars with superior synthesis quality and animation fidelity. • A novel initialization strategy for animatable 3D avatar generation that integrates geometry and appearance initialization, while establishing robust rigging relationships with the corresponding parametric model. • A new geometry- and semantics-aware 3D avatar optimization method for dynamic poses and expressions, leveraging a ControlNet for robust guidance and improved alignment with the driving parametric model. Figure 2. Overview of