MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussians

Peng Chen, Xiaobao Wei, Qingpo Wuwu, Xinyi Wang, Xingyu Xiao, Ming Lu · 2024

Read MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussians by Peng Chen, Xiaobao Wei, Qingpo Wuwu, Xinyi Wang, Xingyu Xiao, Ming Lu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04955.txt.

Introduction

Peng Chen and co-authors blend two rendering styles for digital heads in a 2024 paper. Virtual reality needs head avatars that look real and measure right. Volumetric Gaussian blobs render fast with fine appearance but wobbly surfaces. Flat two dimensional surfels give crisp geometry at some cost to looks. Their method, named in the title, mixes the two so each covers the other's weakness.

The blend targets avatars that are realistic and geometrically accurate at once. Speed matters too: the older neural field approach trained and rendered slowly. A mixed representation could serve telepresence and games, where a head must hold up from every angle at frame rate. The paper builds and tests that compromise.

Catalog record

CREATORPeng Chen, Xiaobao Wei, Qingpo Wuwu, Xinyi Wang, Xingyu Xiao, Ming Lu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussians Peng Chen, Xiaobao Wei, Qingpo Wuwu, Xinyi Wang, Xingyu Xiao, Ming Lu Abstract . Reconstructing high-fidelity 3D head avatars is crucial in various applications such as virtual reality. The pioneering methods reconstruct realistic head avatars with Neural Radiance Fields (NeRF), which have been limited by training and rendering speed. Recent methods based on 3D Gaussian Splatting (3DGS) significantly improve the efficiency of training and rendering. However, the surface inconsistency of 3DGS results in subpar geometric accuracy; later, 2DGS uses 2D surfels to enhance geometric accuracy at the expense of rendering fidelity. To leverage the benefits of both 2DGS and 3DGS, we propose a novel method named MixedGaussianAvatar for realistically and geometrically accurate head avatar reconstruction. Our main idea is to utilize 2D Gaussians to reconstruct the surface of the 3D head, ensuring geometric accuracy. We attach the 2D Gaussians to the triangular mesh of the FLAME model and connect additional 3D Gaussians to those 2D Gaussians where the rendering quality of 2DGS is inadequate, creating a mixed 2D-3D Gaussian representation. These 2D-3D Gaussians can then be animated using FLAME parameters. We further introduce a progressive training strategy that first trains the 2D Gaussians and then fine-tunes the mixed 2D-3D Gaussians. We use a unified mixed Gaussian representation to integrate the two modalities of 2D image and 3D mesh. Furthermore, the comprehensive experiments demonstrate the superiority of MixedGaussianAvatar. The code will be released. 1. Introduction Reconstructing high-quality 3D head avatars from multiple 2D images captured from different viewpoints is essential for various applications ( Wohlgenannt et al., 2020 ; Chollet et al., 2009 ; Bai et al., 2024 ; Wei et al., 2024a ; Chen et al., 2025 ; Shao et al., 2024 ; Wei et al., 2025 ) . The key high-fidelity aspects include the geometric and realistic accuracy of 3D head avatars and rendered images. With the advancement of deep learning, techniques based on Neural Radiance Fields (NeRF) ( Mildenhall et al., 2021 ) and 3D Gaussian Splatting (3DGS) ( Kerbl et al., 2023 ) have gained popularity, providing significant benefits for creating high-fidelity 3D head avatars. Figure 1. MixedGaussianAvatar uses a mixed 2D-3D Gaussian Splatting method to reconstruct a realistically and geometrically accurate 3D head avatar mesh. NeRF introduces the idea of radiance fields, utilizing neural networks to store 3D information, which allows for reconstructing high-quality static scenes. When it comes to 3D head avatar reconstruction, the proposed methods primarily focus on reconstructing dynamic head avatars ( Gao et al., 2022 ; Zielonka et al., 2022 ) . For example, Neural Head Avatars (NHA) ( Grassal et al., 2022 ) process RGB video inputs that display various expressions and perspectives. NHA involves estimating and refining the low-dimensional shape, expression, and pose parameters of the FLAME head model from an RGB input frame, which allows for creating a neural head avatar that can be animated and rendered from novel viewpoints. However, the head avatars generated using NeRF-based methods have lower accuracy due to the implicit representation characteristics of NeRF, and the training and rendering processes are slow. Recently, 3DGS effectively models a static scene using 3D Gaussians and rendered it through a differentiable rasterizer, greatly speeding up the 3D reconstruction process from multi-view RGB images. Because of 3DGS’s explicit representation and efficient rasterization properties, it achieves high rendering quality and fast training and inference speeds ( Chen et al., 2024 ; Wang et al., 2025 ; Hu et al., 2024a ; Hu et al., 2024b ) . In 3D head avatar reconstruction, most methods attach 3D Gaussians to the FLAME triangular mesh to create dynamic facial expressions ( Yu et al., 2024 ) . For example, GaussianAvatars ( Qian et al., 2024a ) employ a rigging approach to align the local positions, rotations, and scales of 3D Gaussians with the global parameters. FlashAvatar ( Xiang et al., 2024 ) and Gaussian Head Avatar ( Xu et al., 2024 ) utilize neural networks to predict 3D Gaussian parameters. The advantage of these methods lies in their ability to dynamically represent 3D head avatars while producing high-quality images. However, due to the inherent multi-view inconsistency of 3DGS, it cannot accurately reconstruct geometric surfaces. 2D Gaussian Splatting ( Huang et al., 2024b ) replaces 3D Gaussians with 2D Gaussians and employs a ray-splat intersection method for the splatting process, ensuring consistency across multiple views. The advantage lies in the high quality of reconstructed geometric surfaces, but it compromises the quality of rendered images. To summarize 3DGS and 2DGS, 1) 3DGS has achieved realistic results in novel view synthesis, but it struggles to capture accurate geometric structures. 2) 2DGS uses 2D surfels to accurately reconstruct geometric surfaces, but it cannot render realistic images as effectively as 3DGS. To unify the two modalities of 2D image rendering and 3D mesh reconstruction, we harness the advantages of both 3DGS and 2DGS and introduce an innovative method called MixedGaussianAvatar. This method uses a mixed 2D-3D Gaussians representation, combining the color rendering advantages of 3DGS with the geometric reconstruction strengths of 2DGS to achieve a realistically and geometrically accurate 3D head avatar reconstruction. As shown in Fig. 1 , our primary goal is to use 2DGS to preserve the geometric accuracy of the 3D head surface. We attach 2D Gaussians to the triangular mesh of the FLAME model and then connect additional 3D Gaussians to the corresponding 2D Gaussians in areas where the rendering quality is insufficient. The mixed 2D-3D Gaussians can be driven using FLAME parameters to form a dynamic 3D representation. To train the mixed 2D-3D Gaussians, we further propose a progressive training strategy that first trains the 2D Gaussians and then finetunes the mixed 2D-3D Gaussians. Our method unifies image rendering and mesh reconstruction through a mixed Gaussian representation, with contributions summarized as follows: • We introduce MixedGaussianAvatar, an innovative approach that combines the rendering benefits of 3DGS with the reconstruction strengths of 2DGS, enabling the realistically and geometrically accurate reconstruction of 3D head avatars. • We present a progressive training strategy to train mixed 2D-3D Gaussians for dynamic 3D head avatars. • We conduct extensive experiments that show MixedGaussianAvatar achieves state-of-the-art color rendering and geometric reconstruction results. 2. Related Work Figure 2. Pipeline of MixedGaussianAvatar. We propose a differentiable mixed 2D-3D Gaussian Splatting method for reconstructing realistically and geometrically accurate head avatars from multi-view 2D images. This approach uses 2D Gaussians for geometric precision and 3D Gaussians to correct color errors, resulting in high-quality 3D head mesh and realistic mesh textures or RGB images. The method is integrated with the FLAME model, enabling dynamic effects and animation driven by parameters through local-to-global transformation and the progressive training strategy. Dynamic Neural Fields The representation of dynamic scenes has attracted considerable attention. Previous methods, like NeRF ( Mildenhall et al., 2021 ) and its variants ( Müller et al., 2022 ; Chen et al., 2022 ; Wei et al., 2024c ; Wei et al., 2024b ) , focus on modeling static scenes, storing them in learnable neural networks to achieve competitive novel view synthesis quality. Recently, 3D Gaussian Splatting (3DGS) ( Kerbl et al., 2023 ) and its variants ( Lu et al., 2024

Read the complete text