Instant Expressive Gaussian Head Avatar via 3D-Aware Expression Distillation

Kaiwen Jiang, Xueting Li, Seonwook Park, Ravi Ramamoorthi, Shalini De Mello, Koki Nagano, University of California, San Diego · 2025

Read Instant Expressive Gaussian Head Avatar via 3D-Aware Expression Distillation by Kaiwen Jiang, Xueting Li, Seonwook Park, Ravi Ramamoorthi, Shalini De Mello, Koki Nagano, University of California, San Diego (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16893.txt.

Introduction

Kaiwen Jiang and coauthors tackle fast, expressive 3D head avatars from a single photo. Video diffusion methods animate portraits well but break 3D consistency and run slow, while 3D-aware feed-forward methods are fast yet lose facial detail.

Their pipeline turns one wild photo into an animatable 3D form in one pass. A light local fusion step joins structure and motion data, replacing heavy global attention. Motion is learned from data in a form split from the face shape, so no fixed parametric face model caps expression. The system runs at 107.31 frames per second for animation and pose control, a three to four order speed gain over the state of the art at like quality.

Catalog record

CREATORKaiwen Jiang, Xueting Li, Seonwook Park, Ravi Ramamoorthi, Shalini De Mello, Koki Nagano, University of California, San Diego
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Instant Expressive Gaussian Head Avatar via 3D-Aware Expression Distillation Kaiwen Jiang, Xueting Li, Seonwook Park, Ravi Ramamoorthi, Shalini De Mello, Koki Nagano, University of California, San Diego Abstract Portrait animation has witnessed tremendous quality improvements thanks to recent advances in video diffusion models. However, these 2D methods often compromise 3D consistency and speed, limiting their applicability in real-world scenarios, such as digital twins or telepresence. In contrast, 3D-aware facial animation feedforward methods – built upon explicit 3D representations, such as neural radiance fields or Gaussian splatting – ensure 3D consistency and achieve faster inference speed, but come with inferior expression details. In this paper, we aim to combine their strengths by distilling knowledge from a 2D diffusion-based method into a feed-forward encoder, which instantly converts an in-the-wild single image into a 3D-consistent, fast yet expressive animatable representation. Our animation representation is decoupled from the face’s 3D representation and learns motion implicitly from data, eliminating the dependency on pre-defined parametric models that often constrain animation capabilities. Unlike previous computationally intensive global fusion mechanisms (e.g., multiple attention layers) for fusing 3D structural and animation information, our design employs an efficient lightweight local fusion strategy to achieve high animation expressivity. As a result, our method runs at 107.31 FPS for animation and pose control while achieving comparable animation quality to the state-of-the-art, surpassing alternative designs that trade speed for quality or vice versa. Figure 1 : We present an instant feedforward encoder that transforms an in-the-wild source image into an animatable 3D avatar by distilling knowledge from a pre-trained 2D diffusion model. Our method introduces a fast, consistent yet expressive 3D animation representation. Given a driving image, we evaluate both the expression transfer quality and the animation speed (measured as “FPS” on an NVIDIA 6000 Ada GPU) against (a) 2D diffusion- or GAN-based methods and (b) 3D-aware methods. In the first row, Portrait4D-v2 16 , GAGAvatar 11 and VOODOO-XP 78 fail to faithfully transfer expressions, particularly around the nasal wrinkles. LivePortrait 26 is inaccurate at eyes. In the second row, the baby wears a fake mustache as a decoration. X-NeMo distorts identity and adds a hallucinated mustache. Other methods cannot deal with the asymmetric expression in the driving image well. The FPS marked with * reports inference time excluding time-consuming morphable model fitting optimization required for the method. In contrast, ours not only accurately transfers expressions but also achieves high animation speed and consistent pose control. Insets show our rendered results under different poses. † † footnotetext: This project was initiated and substantially carried out during an internship at NVIDIA. 1 Introduction Creating a digital twin from a single facial image that supports both 3D viewpoint control and animation (4D) is a long-standing goal in computer vision and graphics. Interactive synthesis and animation control of photorealistic digital humans is essential for developing AR/VR, video conferencing, and agentic AI applications. Achieving such comprehensive 4D control has been historically challenging. With the advent of radiance fields including neural radiance fields (NeRFs) 51 and 3D Gaussians 32 , previous 3D-aware face animation work 43 ; 12 ; 112 ; 11 ; 71 has achieved interactive and consistent animation with photo-realistic view synthesis by using parametric models 4 ; 20 ; 41 . However, parametric models inherently limit the animation capability. Follow-up methods 36 ; 78 ; 16 ; 15 ; 79 therefore resort to learning the animation purely from data. Nevertheless, their representations entangle 3D structure and animation (e.g., global residual triplanes 8 or feature maps), requiring computationally expensive attention mechanisms to repeatedly fuse 3D structure and motion at every animation step. Meanwhile, the introduction of 2D diffusion models 98 ; 113 ; 13 ; 92 ; 49 ; 59 ; 26 ; 7 ; 10 ; 63 ; 60 ; 38 into portrait animation has brought the expressivity of achieved facial animation to a whole new level 113 ; 98 . However, these methods often suffer from 3D inconsistency and remain slow due to the expensive denoising process, preventing them from being used in a real-time system. Fig. 2 shows a quantitative comparison where existing methods fail to excel at all three criteria : speed, 3D consistency and expression transfer accuracy. Different from other slower optimization-based methods 75 ; 24 ; 72 ; 1 ; 74 , we seek to design a 3D-aware animation framework that instantly encodes a facial image into an animatable 4D avatar, which supports fast, consistent and detailed animation. Our key insights to solving this problem are twofold. First, we argue that high-quality facial animation has already been effectively learned by 2D diffusion models, and this knowledge can be distilled into 3D-aware methods rather than learning from scratch. Our second insight is that achieving efficiency and 3D consistency without sacrificing expressiveness requires a novel animation representation. Specifically, we build upon 3D Gaussians 32 that makes dynamic deformation more efficient than NeRFs’ volumetric fields to propose an expressive yet efficient animation representation. We first encode an input image into triplanes 80 as an intermediate representation, from which we sample feature vectors to decode the Gaussian attributes. For each Gaussian, we encode its motion information in an auxiliary vector which is analogous to learned personalized “PCA bases” in traditional blendshapes animation. This auxiliary vector is then combined with the driving signal to deform each Gaussian on top of the existing 3D structure, updating the 3D representation while keeping the animation representation both efficient and decoupled from the underlying 3D representation. While previous works 46 ; 47 ; 90 ; 45 deform the 3D Gaussians in spatial space to model motion, we find it unable to capture expressive facial details. Instead, we propose to deform the Gaussians individually in the high-dimensional feature vector space, rather than in 3D spatial space, which offers better expressivity and is capable of capturing asymmetric expressions, details such as shadow changes and wrinkles (Fig. 1 ). Typically, portrait animation methods, including ours, are trained with a self-reenactment objective using datasets that contain multiple expression-varying images per identity. Instead of training on real datasets, we construct a synthetic facial expression dataset by a state-of-the-art facial animation diffusion model 113 for distilling its expressive motion priors in a 3D consistent manner. To mitigate the potential 3D inconsistency issue, we synthesize facial expressions by the diffusion model on real face portraits frontalized by a pre-trained 3D lifting encoder 80 and again use the same 3D lifting encoder to estimate its multi-view synthetic images on the fly during training. In summary, our main contributions include: • We design an expressive yet computationally-efficient animation representation for 3D Gaussians that achieves detailed animation for human faces (Sec. 4.1 ). • We propose practical strategies to train such an animation representation by distilling the knowledge of existing diffusion methods into it (Sec. 4.2 ). • Our method is the first to simultaneously achieve best 3D consistency, fast inference speed and detailed expressions such as wrinkles, and run orders of magnitude faster than 2D diffusion models during inference (Fig. 1 , 2 ). Figure 2 : We provide a visualization of the quantitative comparison in terms of 3D inconsistency (measured by MEt3R \downarrow ), expression transfer inaccuracy (measured by AED \downarrow ) and animation speed (measured by

Read the complete text