GraphAvatar: Compact Head Avatars with GNN-Generated 3D Gaussians
Xiaobao Wei, Peng Chen, Ming Lu, Hui Chen, Feng Tian · 2024
Read GraphAvatar: Compact Head Avatars with GNN-Generated 3D Gaussians by Xiaobao Wei, Peng Chen, Ming Lu, Hui Chen, Feng Tian (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13983.txt.
Introduction
Xiaobao Wei and co-authors present GraphAvatar in this 2024 paper, a compact way to build lifelike digital heads. Such avatars serve in virtual reality, where they must look real from any angle and render fast. Older methods reached good looks but were slow or bulky. A recent method using 3D Gaussians renders fast yet takes much storage.
GraphAvatar instead trains two graph neural networks, one for head shape and one for look. The networks produce the Gaussian values from a tracked face mesh, so only the networks need storing. That cuts the file size to about ten megabytes. Extra steps correct face-tracking errors and polish the final image. Tests show better looks and far lower storage than rival methods.
Catalog record
| CREATOR | Xiaobao Wei, Peng Chen, Ming Lu, Hui Chen, Feng Tian |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GraphAvatar: Compact Head Avatars with GNN-Generated 3D Gaussians Xiaobao Wei, Peng Chen, Ming Lu, Hui Chen, Feng Tian Abstract Rendering photorealistic head avatars from arbitrary viewpoints is crucial for various applications like virtual reality. Although previous methods based on Neural Radiance Fields (NeRF) can achieve impressive results, they lack fidelity and efficiency. Recent methods using 3D Gaussian Splatting (3DGS) have improved rendering quality and real-time performance but still require significant storage overhead. In this paper, we introduce a method called GraphAvatar that utilizes Graph Neural Networks (GNN) to generate 3D Gaussians for the head avatar. Specifically, GraphAvatar trains a geometric GNN and an appearance GNN to generate the attributes of the 3D Gaussians from the tracked mesh. Therefore, our method can store the GNN models instead of the 3D Gaussians, significantly reducing the storage overhead to just 10MB. To reduce the impact of face-tracking errors, we also present a novel graph-guided optimization module to refine face-tracking parameters during training. Finally, we introduce a 3D-aware enhancer for post-processing to enhance the rendering quality. We conduct comprehensive experiments to demonstrate the advantages of GraphAvatar, surpassing existing methods in visual fidelity and storage consumption. The ablation study sheds light on the trade-offs between rendering quality and model size. The code will be released at: https://github.com/ucwxb/GraphAvatar Code — https://github.com/ucwxb/GraphAvatar Introduction Rendering photorealistic head avatars from any viewpoint is essential for virtual reality and augmented reality applications. Key aspects such as visual fidelity, rendering speed, and storage overhead are crucial. With the advancement of deep learning, methods based on neural fields have become prevailing due to their advantages in these aspects. The seminal neural field work Neural Radiance Fields (NeRF) ( Mildenhall et al. 2021 ) and its variants ( Wang et al. 2021 ; Wei et al. 2023 ) have achieved impressive results for neural rendering and reconstruction. Regarding head avatars, NeRF-based methods ( Gao et al. 2022 ; Zielonka, Bolkart, and Thies 2022a ; Zheng et al. 2023 ; Lombardi et al. 2019 ; Lombardi et al. 2021 ) have focused on improving the generation from short RGB video inputs. Based on 3D morphable models (3DMM) ( Li et al. 2017 ; Paysan et al. 2009 ) , these methods learn neural fields to create 3D-consistent and interpretable digital head avatars. These approaches enable high-quality rendering and diverse applications, such as facial retargeting, expression editing, and rapid avatar generation. Although point ( Zheng et al. 2023 ) and hash table ( Zielonka, Bolkart, and Thies 2022a ; Gao et al. 2022 ) representations have been explored for acceleration, they still struggle with the excessive sampling in the volume rendering of NeRF, which prevent them from achieving real-time rendering easily. Moreover, due to the implicit characteristics of NeRF, these methods lack controllability and do not generalize well to the novel poses and expressions. Recently, 3D Gaussian Splatting ( Kerbl et al. 2023 ) successfully represents a static scene as 3D Gaussians and renders with a differentiable rasterizer, significantly accelerating novel view synthesis. Due to the unstructured nature of 3D Gaussian representation, 3DGS also excels in controllability and generalization compared to NeRF ( Fang et al. 2023 ) . Many recent methods ( Qian et al. 2024 ; Xiang et al. 2024 ; Xu et al. 2024 ; Dhamo et al. 2023 ; Wei et al. 2024 ; Chen et al. 2024 ) have been proposed to apply 3DGS to head avatars. Based on face tracking parameters, these methods initialize or bind 3D Gaussians to the geometry prior using neutral mesh ( Xu et al. 2024 ) , tracked mesh ( Qian et al. 2024 ) , or UV mapping ( Xiang et al. 2024 ) . Subsequently, the 3D Gaussians are learned from short RGB video inputs involving densification, such as cloning and pruning, and deformations from canonical space. These methods leverage 3D head meshes to incorporate prior knowledge, facilitating better convergence of 3D Gaussians, which capture human heads’ appearance, geometry, and dynamics. Figure 1: GraphAvatar leverages graph neural networks to generate 3D Gaussians, which are then rasterized into high-fidelity images based on tracked meshes. Compared to contemporary approaches, GraphAvatar not only delivers superior rendering performance but also features the most compact model size, substantially minimizing storage requirements. However, existing 3DGS-based approaches still face two obvious limitations: 1) Significant and fluctuating storage overhead. Since the 3D head mesh contains many triangles, assigning multiple 3D Gaussians to each triangle results in excessive storage overhead due to the large number of triangles in the 3D head mesh. Furthermore, the number of 3D Gaussians also changes during the densification process. This leads to a fluctuating storage overhead, posing challenges for practical applications. 2) Heavy reliance on face tracking. Since these methods initialize or bind 3D Gaussians to the tracked head mesh, they heavily rely on the accuracy of face tracking. The accumulated errors of face tracking will impact the training of 3D Gaussians. To address these issues, we propose a novel method named GraphAvatar, which utilizes Graph Neural Networks (GNN) ( Kipf and Welling 2016 ; Ranjan et al. 2018 ) to generate 3D Gaussians for photorealistic head avatars. As shown in Fig. 1 , GraphAvatar optimizes a geometric GNN and an appearance GNN to generate 3D Gaussians using tracked meshes as input. These 3D Gaussians serve as anchors and are fed into a view-dependent MLP to learn 3D Gaussian offsets related to different viewpoints. The predicted offsets adjust the anchor 3D Gaussians, breaking free from the constraints imposed by the tracked meshes, allowing for learning better details. Subsequently, rasterization is used to render the adjusted 3D Gaussians into photorealistic head avatars. Thus, our method significantly reduces the storage overhead by storing the GNN models instead of many 3D Gaussians, making our method more compact. To reduce the impact of face-tracking errors, we also introduce an advanced graph-guided optimization module to optimize face-tracking parameters during the training. Finally, to reduce the over-smoothing induced by GNN, we incorporate a lightweight 3D-aware enhancer for post-processing, which utilizes rendered depth maps to improve rendering quality. Our main contributions are summarized as follows: • We propose GraphAvatar, a novel and compact method that utilizes GNN to generate 3D Gaussians. Our method only stores the GNN models, eliminating the need to store 3D Gaussians directly. • To reduce reliance on tracked meshes, we introduce a graph-guided optimization module to refine face-tracking parameters during training. • To alleviate the over-smoothing induced by GNN, we incorporate a 3D-aware enhancer to enhance details for the rendered images. • We conduct comprehensive experiments on challenging datasets to demonstrate that GraphAvatar achieves state-of-the-art rendering quality while requiring the least storage overhead. Related work Head Avatar Animation. The animation generation of head avatars is divided into 3D and 2D scenes. In 3D animations, the FLAME model ( Li et al. 2017 ) is commonly used for generation and editing tasks. For instance, studies such as VOCA ( Cudeiro et al. 2019 ) , COMA ( Ranjan et al. 2018 ) , FaceFormer ( Fan et al. 2022 ) , and SelfTalk ( Peng et al. 2023a ) utilize the FLAME model for speech-driven head avatar animation generation. Meanwhile, methods like EmoTalk ( Peng et al. 2023b ) and