PERSE : Personalized 3D Generative Avatars from A Single Portrait
Hyunsoo Cha Inhee Lee Hanbyul Joo Seoul National University {243stephen, ininin0516, hbjoo}@snu.ac.kr https://hyunsoocha.github.io/perse/ · 2024
Read PERSE : Personalized 3D Generative Avatars from A Single Portrait by Hyunsoo Cha Inhee Lee Hanbyul Joo Seoul National University {243stephen, ininin0516, hbjoo}@snu.ac.kr https://hyunsoocha.github.io/perse/ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-21206.txt.
Description
Abstract We present PERSE, a method for building an animatable personalized generative avatar from a reference portrait. Our avatar model enables facial attribute editing in a continuous and disentangled latent space to control each facial attribute, while preserving the individual’s identity.
Catalog record
| CREATOR | Hyunsoo Cha Inhee Lee Hanbyul Joo Seoul National University {243stephen, ininin0516, hbjoo}@snu.ac.kr https://hyunsoocha.github.io/perse/ |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-21206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PERSE : Personalized 3D Generative Avatars from A Single Portrait Hyunsoo Cha Inhee Lee Hanbyul Joo Seoul National University {243stephen, ininin0516, hbjoo}@snu.ac.kr https://hyunsoocha.github.io/perse/ Abstract We present PERSE, a method for building an animatable personalized generative avatar from a reference portrait. Our avatar model enables facial attribute editing in a continuous and disentangled latent space to control each facial attribute, while preserving the individual’s identity. To achieve this, our method begins by synthesizing large-scale synthetic 2D video datasets, where each video contains consistent changes in the facial expression and viewpoint, combined with a variation in a specific facial attribute from the original input. We propose a novel pipeline to produce high-quality, photorealistic 2D videos with facial attribute editing. Leveraging this synthetic attribute dataset, we present a personalized avatar creation method based on the 3D Gaussian Splatting, learning a continuous and disentangled latent space for intuitive facial attribute manipulation. To enforce smooth transitions in this latent space, we introduce a latent space regularization technique by using interpolated 2D faces as supervision. Compared to previous approaches, we demonstrate that PERSE generates high-quality avatars with interpolated attributes while preserving identity of reference person. 1 Introduction A personalized 3D face avatar can represent each individual in VR/AR environments, replicating the user’s appearance and facial expressions. However, the exact replication of the appearance does not fully reflect real-world humans. In reality, people often change the attributes of their appearance, like hairstyles, or start growing a beard or mustache. Users may also wish to adjust their facial features in the virtual world, like the shape of their nose, eyes, or mouth, enhancing their desired look while preserving their core identity. While most prior avatar creation methods focus on building an exact digital twin of the person from images or video data Zheng et al. 2022 ; Zheng et al. 2023 ; Chen et al. 2024 ; Zielonka et al. 2023 ; Gafni et al. 2021 ; Saito et al. 2024 ; Shao et al. 2024 ; Xu et al. 2023 , the personalized avatar model with the generative ability to control and edit facial attributes remains underexplored. In this work, we present PERSE, a method to build a 3D personalized generative avatar from a single reference portrait image. Our method goes beyond merely creating an exact twin from video inputs, introducing a novel approach that emphasizes flexibility and control over facial attributes, such as changing hairstyles or beards shown in Fig. . To build PERSE from a single reference portrait image, we generate a large-scale 2D synthetic video dataset of the reference identity, where each video has a variation in a specific facial attribute from the original input (e.g., a different hairstyle) driven by the face motion guidance, as shown in Fig. 5 . Each video is also paired with a text prompt description addressing the changed attribute. To build this high-quality and photorealistic synthetic video dataset, we introduce a new pipeline that begins with synthesizing 2D images with attribute editing, followed by a portrait animation process leveraging the combination of existing pre-trained 2D portrait animation method Guo et al. 2024b and our newly trained image-to-video model extending a prior work Zhu et al. 2024 . Notably, our synthetic video generation process is efficient, scalable, and provides significantly more attribute diversity by effectively synthesizing a thousand attribute videos compared to tens in prior work Cha et al. 2024 . Using this synthetic video dataset, we train an avatar model with the continuous and disentangled attribute latent space. To enhance the generative ability of our avatar model for unseen or interpolated attribute appearances, we also present a novel technique to enforce continuous and smooth latent space. To achieve this, we present a latent space regularization technique by using interpolated 2D face images from an image morphing technique Zhang et al. 2024a (e.g., synthesizing medium-length hair from short hair and long hair attributes), providing pseudo supervisions for the interpolated latent spaces. We show the efficacy of our regularization technique by producing novel and unseen attributes from interpolated latent spaces, as shown in Fig. 7 . Furthermore, we present an efficient fine-tuning technique via Low-Rank Adaptation (LoRA) Hu et al. 2021 , to integrate any new facial attributes from in-the-wild images into our avatar model. Our contributions are summarized as follows: (1) the first method to generate a personalized 3D avatar from a reference portrait image with controllable facial attributes; (2) a method to generate high-quality synthetic 2D video datasets with diverse attribute editing from a reference portrait image; (3) latent space regularization by using face morphing supervision for continuous and smooth latent space to enhance the generative ability for unseen or interpolated attribute appearances; (4) an efficient fine-tuning technique via Low-Rank Adaptation (LoRA) Hu et al. 2021 to integrate any new facial attribute into the avatar model. 2 Related Work 3D Facial Avatar Reconstruction. Since the introduction of foundational 3D Morphable Model Blanz and Vetter 1999 (3DMM), parametric 3D face models Blanz and Vetter 1999 ; Li et al. 2017 ; Cao et al. 2013 have evolved to capture the diverse and dynamic nature of human faces, representing variations in shape, head pose, and facial expression through a set of parameters. Building on these models, various methods reconstruct 3D face avatars from single portrait images by estimating 3DMM parameters Sanyal et al. 2019 ; Feng et al. 2021 ; Daněček et al. 2022 . Recently, monocular 3D avatar reconstruction methods Gafni et al. 2021 ; Zheng et al. 2022 ; Zheng et al. 2023 ; Chen et al. 2024 ; Kirschstein et al. 2024 ; Lee et al. 2024 generate morphable photorealistic avatars leveraging advancements in 3D representation Mildenhall et al. 2021 ; Kerbl et al. 2023 . To move beyond single-subject avatars, a PEGASUS Cha et al. 2024 reconstructs a personalized generative avatar enabling control over facial attributes while preserving the reference identity, using synthetic DB. Similarly, HeadGAP Zheng et al. 2024b trains a generalizable prior model for 3D head avatar leveraging a large-scale multiview dataset and an avatar model with part-specific and point-specific feature codes. Despite advancements, constructing a unified 3D representation that can precisely capture and control all facial attributes remains challenging. To address this, disentangled or hybrid representations have been proposed, enabling selective modification of facial features or garments Feng et al. 2023 ; Feng et al. 2022 ; Kim et al. 2024 . However, these approaches are limited by discrete 3D structures, restricting continuous interpolation capabilities. Recently, latent-conditioned generative models Kim et al. 2023 ; Li et al. 2023 ; Ho et al. 2023 ; Cha et al. 2024 have been introduced to mitigate these constraints, yet they often lack the capacity for fine-grained editing and are confined to specific categories. Smooth Image Morphing and Interpolation. Generating a plausible intermediate image between two pivot images has been widely studied within the context of image generative field Karras et al. 2021 ; Song et al. 2020 ; Ho et al. 2020 . The recent breakthroughs of diffusion model Song and Ermon 2019 ; Ho et al. 2020 ; Rombach et al. 2022 improved the image interpolation methods to generate more plausible and better quality interpolated images with less limitation on categories Wang and Golland 2023