StyleMorpheus: A Style-Based 3D-Aware Morphable Face Model
Peizhi Yan, Rabab K. Ward, Dan Wang, Qiang Tang, and Shan Du · 2025
Read StyleMorpheus: A Style-Based 3D-Aware Morphable Face Model by Peizhi Yan, Rabab K. Ward, Dan Wang, Qiang Tang, and Shan Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11792.txt.
Introduction
Peizhi Yan and co-authors present a style based three dimensional face model in 2025. New rendering methods can show faces from any angle with photo like quality. Controllable models still needed lab gathered data sets for training. Their model, StyleMorpheus, learns from wild photos found in daily sources instead.
The model splits control over identity, expression, and look into separate codes. An encoder learns those codes while a decoder sharpens the split with style inputs. No exact three dimensional shapes are needed at training time. The result is a face model that artists can steer while keeping views consistent.
Catalog record
| CREATOR | Peizhi Yan, Rabab K. Ward, Dan Wang, Qiang Tang, and Shan Du |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11792
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StyleMorpheus: A Style-Based 3D-Aware Morphable Face Model Peizhi Yan, Rabab K. Ward, Dan Wang, Qiang Tang, and Shan Du Abstract For 3D face modeling, the recently developed 3D-aware neural rendering methods are able to render photorealistic face images with arbitrary viewing directions. The training of the parametric controllable 3D-aware face models, however, still relies on a large-scale dataset that is lab-collected. To address this issue, this paper introduces “StyleMorpheus”, the first style-based neural 3D Morphable Face Model (3DMM) that is trained on in-the-wild images. It inherits 3DMM’s disentangled controllability (over face identity, expression, and appearance) but without the need for accurately reconstructed explicit 3D shapes. StyleMorpheus employs an auto-encoder structure. The encoder aims at learning a representative disentangled parametric code space and the decoder improves the disentanglement using shape and appearance-related style codes in the different sub-modules of the network. Furthermore, we fine-tune the decoder through style-based generative adversarial learning to achieve photorealistic 3D rendering quality. The proposed style-based design enables StyleMorpheus to achieve state-of-the-art 3D-aware face reconstruction results, while also allowing disentangled control of the reconstructed face. Our model achieves real-time rendering speed, allowing its use in virtual reality applications. We also demonstrate the capability of the proposed style-based design in face editing applications such as style mixing and color editing. Project homepage: https://github.com/ubc-3d-vision-lab/StyleMorpheus . I Introduction The field of 3D-aware face modeling has witnessed rapid advancements in recent years, due to its promise in many applications such as virtual reality avatars [ 1 , 2 ] , face editing [ 3 , 4 ] , and facial animation [ 5 , 6 ] . Different from traditional 3D face models which are mesh-based, 3D-aware methods do not explicitly represent the 3D face geometry (shape). This ability is enabled by employing neural radiance fields (NeRF) models [ 7 , 8 , 9 ] for learning an implicit 3D face representation. The key benefits of most 3D-aware face models over traditional 3D face models are (1) they are differentiable in nature and can learn from 2D images, (2) they can represent complex structures such as hair and eyeglasses, and (3) they can produce photorealistic appearance without the need for costly 3D modeling. Despite these benefits, the training of a 3D-aware parametric face model using in-the-wild face images (captured in uncontrolled, everyday environments, rather than in a structured studio setting) has remained a challenge. Amongst the traditional 3D face models, the most notable is the 3D Morphable Face Model (3DMM) [ 10 ] as it has the ability to independently and parametrically control the identity, expression, and appearance (color and lighting) of the reconstructed 3D face. In this work, we propose a 3D-aware face model that is parametrically controllable like 3DMM but only uses in-the-wild 2D face images for training. By leveraging a large amount of in-the-wild training data, our method can achieve better generalization ability and handle a wide range of diversity than using lab-collected data. StyleGAN has achieved great success in 2D photorealistic face image synthesis using in-the-wild face images as training data [ 11 , 12 ] . It provides style-based control over generative image synthesis: StyleGAN leverages a high-dimensional latent space to manipulate the style aspects (such as age, gender, and other high-level facial attributes) of generated face images, emphasizing the separation of style from content to enable a more controllable image generation. To address the lack of 3D view consistency in 2D StyleGANs, 3D-aware style-based face modeling methods have recently been proposed. Some of these methods focus on random face image generation as in the original StyleGAN [ 13 , 14 , 15 , 16 ] . Other methods focus on obtaining better controllability and view consistency in face image generation while allowing an explicit reconstructed 3D face geometry as the 3D prior [ 17 , 18 , 19 ] . CGoF proposes the mesh-guided volume sampler to impose the shape information in volume rendering [ 17 ] . OmniAvatar trains a Signed Distance Function (SDF) network to generate the head shape [ 18 ] . This network is subsequently utilized to construct a volumetric correspondence map that serves as a guide for the process of volume rendering [ 20 , 21 ] . Next3D extends the idea of neural textures [ 22 ] by synthesizing a neural texture through the StyleGAN generator [ 19 ] . The synthesized neural texture and a reconstructed head mesh [ 23 ] are then used in the rasterization process to derive a texture-rasterized field for 3D-aware rendering. Overall, these methods still have the problem of using explicit 3D face geometry. This dependency on an explicitly reconstructed 3D face shape in face reconstruction leads to high complexity in the model’s architecture. For face generation models to be parametrically controllable and to not employ explicit 3D face shapes, recent 3D-aware parametric face models [ 24 , 25 , 26 ] have drawn inspiration from the development of 3DMM [ 10 ] . MoFaNeRF is the first parametric 3D-aware face model trained on large-scale datasets of multi-view face images [ 24 ] . Similar to the original NeRF, MoFaNeRF adopts volume rendering at the end to directly render the RGB image, which is computationally expensive and hinders its real-time application. HeadNeRF achieves real-time rendering by using NeRF to render low-resolution feature maps and having 2D neural rendering layers [ 27 , 13 ] to generate high-resolution face images [ 25 ] . The parameter space of HeadNeRF is based on non-linear 3DMM [ 28 ] and is adjusted through learning to capture more information such as hair, teeth, and detailed textures. Although both MoFaNeRF and HeadNeRF deliver impressive results, they heavily depend on large-scale lab-collected data, which is expensive to form and presents a challenge for generalizing to the in-the-wild images. 3DMM-RF explores the use of a 3DMM to synthesize images as training data and avoids the use of real images [ 26 ] . However, 3DMM-RF does not fully take advantage of NeRF’s benefits as it only mimics the 3DMM and thus cannot render hair and teeth. Importantly, due to the nature of NeRF which learns an implicit 3D representation, accurate camera pose is one of the key factors in training a NeRF to accurately render the details. When using in-the-wild images as training data, we can only roughly estimate the camera pose, which usually leads to blurry rendered images. Recently, the 3D Gaussian Splatting (3DGS) [ 29 ] based face avatars have shown success in rendering photo-realistic faces with controllability and real-time efficiency [ 30 , 31 , 32 , 33 , 34 , 35 ] . The 3DGS-based methods are somewhere between explicit and implicit 3D representations. However, these methods are not general 3D face models and focus on training a new face avatar model for each person, which is expensive. For example, GaussianAvatars needs to be trained for at least 6 hours for a single person to achieve satisfactory results [ 32 ] . Moreover, the training data used in these methods are either multi-view videos [ 36 ] or monocular videos [ 37 ] of the person with varying head poses and expressions. Our work has a different focus from the 3DGS-based face avatars as our method