TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

Jianchuan Chen ,Jingchuan Hu ,Gaige Wang ,Zhonghua JiangTiansong Zhou, Zhiwen Chen, Chengfei LvAlibaba Group, Hangzhou, China · 2025

Read TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting by Jianchuan Chen ,Jingchuan Hu ,Gaige Wang ,Zhonghua JiangTiansong Zhou, Zhiwen Chen, Chengfei LvAlibaba Group, Hangzhou, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17032.txt.

Introduction

Jianchuan Chen and co-authors present TaoAvatar, a system for full-body talking avatars in augmented reality, in a 2025 paper. The avatars are built with 3D Gaussian Splatting, a method that paints a scene from many soft blobs. Past systems struggled to control face and body motion together, missed fine detail, or ran too slowly on mobile gear. The team aim for a lifelike avatar that still runs in real time.

Their method first builds a personal template of the clothed body and ties the blobs to it. A large network learns how cloth and skin deform with pose, and its skill is then distilled into a small, fast network, with blend shapes restoring lost detail. Tests show top rendering quality at 90 frames per second on stereo headsets such as Apple Vision Pro.

Catalog record

CREATORJianchuan Chen ,Jingchuan Hu ,Gaige Wang ,Zhonghua JiangTiansong Zhou, Zhiwen Chen, Chengfei LvAlibaba Group, Hangzhou, China
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17032

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting Jianchuan Chen ,Jingchuan Hu ,Gaige Wang ,Zhonghua JiangTiansong Zhou, Zhiwen Chen, Chengfei LvAlibaba Group, Hangzhou, China Abstract Realistic 3D full-body talking avatars hold great potential in AR, with applications ranging from e-commerce live streaming to holographic communication. Despite advances in 3D Gaussian Splatting (3DGS) for lifelike avatar creation, existing methods struggle with fine-grained control of facial expressions and body movements in full-body talking tasks. Additionally, they often lack sufficient details and cannot run in real-time on mobile devices. We present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking avatar driven by various signals. Our approach starts by creating a personalized clothed human parametric template that binds Gaussians to represent appearances. We then pre-train a StyleUnet-based network to handle complex pose-dependent non-rigid deformation, which can capture high-frequency appearance details but is too resource-intensive for mobile devices. To overcome this, we ”bake” the non-rigid deformations into a lightweight MLP-based network using a distillation technique and develop blend shapes to compensate for details. Extensive experiments show that TaoAvatar achieves state-of-the-art rendering quality while running in real-time across various devices, maintaining 90 FPS on high-definition stereo devices such as the Apple Vision Pro. Figure 1 : TaoAvatar generates photorealistic, topology-consistent 3D full-body avatars from multi-view sequences. It provides high-quality, real-time rendering with low storage requirements, compatible across various mobile and AR devices like the Apple Vision Pro. 1 Introduction Creating lifelike 3D human avatars is a dynamic and rapidly advancing research area in computer vision and graphics, essential for applications in AR/VR, 3D entertainment, and virtual communication. Despite decades of research, achieving realistic and immersive avatars on mobile devices—particularly AR—remains challenging due to computational limitations. Current industry solutions, such as MetaHuman, often rely on high-precision scans and extensive manual effort by artists for 3D modeling, rigging, and motion capture. In recent years, academic researchers have adopted parametric models [ 36 , 42 ] for human representation, which are constructed from massive body scans and can capture a performer’s shape, expression, and pose from images or videos. These methods [ 5 , 4 , 6 , 26 ] extend the naked model [ 36 , 42 ] by adding offsets to represent tailored clothing, but are still struggling with complex geometries and high-frequency details, such as loose skirts and fine hair, due to limitations of topology and texture resolution. With the rise of neural radiance fields (NeRF) [ 39 ] , many researchers tend to define implicit neural human representations in canonical space, animated by backward linear blend skinning [ 64 , 43 ] of these parametric models [ 36 , 42 ] . While these methods can handle arbitrary topology and deliver higher rendering quality, they are highly time-consuming due to the large amount of sampling required during volume rendering. Although many efforts [ 23 , 9 ] have significantly improved rendering efficiency by introducing iNGP [ 41 ] , the animated results still fail to escape the Uncanny Valley effect when driven. Recently, 3D Gaussian Splatting (3DGS) [ 28 ] , an explicit and efficient point-based representation, has gained tremendous attention from researchers for its ability to deliver both high-quality rendering and real-time performance. Compared to implicit representations [ 39 , 53 ] , the explicit point-based representation is more compatible with being integrated with the mesh-based parametric models [ 36 , 42 ] , which can be directly driven by forward linear blend skinning. Many recent methods [ 19 , 45 , 47 , 33 ] , combining 3DGS and the parametric models [ 36 , 42 ] , have made significant strides toward realizing lifelike 3D human avatars. However, achieving real-time rendering on mobile devices remains challenging, especially for AR glasses, which require high resolution, stereo rendering, and seamless interaction. We introduce TaoAvatar , a high-fidelity, lightweight, 3DGS-based full-body talking avatar designed for running in real-time on Augmented Reality (AR) devices. Unlike previous approaches [ 8 , 33 , 10 ] , which develop implicit parameterized models from scratch, we construct a personalized, clothed parameterized template that preserves the facial expressions and gesture controls inherent to the native SMPLX [ 42 ] . At the same time, we bind the Gaussians to the triangles as texture to create a hybrid avatar representation. Inspired by [ 33 ] , we pre-train a large StyleUnet-based teacher network to learn pose-dependent dynamic deformation maps of Gaussians in 2D space by front and back orthogonal projection. Although the teacher network is powerful enough to capture high-frequency appearance details in various poses, its large number of parameters makes it difficult to run in real-time on AR devices. To achieve high-performance rendering, we bake the dynamic deformation of Gaussians into the non-rigid deformation field of the mesh using a distillation technique, employing a compact MLP-based student network. Meanwhile, we propose two lightweight, learnable blend shapes to compensate for the non-rigid deformation of the Gaussians. After fine-tuning, our student model achieves high-performance rendering without compromising quality. To evaluate our approach, we propose a multi-view dataset named TalkBody4D , focusing on prevalent full-body talking scenarios encountered in daily life, which includes rich facial expressions and gestures with synchronous audios different from existing full-body motion datasets [ 21 , 11 , 68 ] . Experimental results demonstrate that our method effectively captures high-frequency appearance details of human avatars while maintaining a lightweight architecture capable of rendering stereo images at 2K resolution on common mobile and AR devices, including the Apple Vision Pro. The resulting full-body avatar is highly expressive, enabling users to animate it with facial expressions, hand gestures, and body poses, thereby highlighting the potential applications of our method. Our contributions can be summarized as follows: • We introduce TaoAvatar , a novel teacher-student framework which yields topologically consistent, well-aligned geometry, further creating a high-fidelity, lightweight, 3DGS-based drivable full-body talking avatar. • We propose two innovative strategies including non-rigid deformation baking and two lightweight blend shapes compensations to ensure high-quality rendering and efficient performance on mobile and AR devices, achieving 2K resolution rendering at 90 FPS on high-definition stereo devices like the Apple Vision Pro. • We contribute TalkBody4D , a multi-view dataset to be released, designed for full-body talking scenarios, featuring diverse facial expressions and gestures with synchronous audios. Extensive experiments demonstrate that our approach outperforms other state-of-the-art methods in both quality and performance. 2 Related work 3D Avatar Representation. Traditional computer graphics techniques [ 49 , 50 , 12 , 18 ] use 3D scanning or multi-view stereo (MVS) to create full-body meshes, which are then combined with a 3D skeleton to produce driveable templates. However, this approach is both expensive and inefficient. As parametric body models, SMPL [ 36 ] and SMPLX [ 42 ] are widely used for their versatility but lack detail in simulating loose clothing. Some works [ 5 , 4 , 26 ] have attempted to add clothing offsets to SMPLX, effectively handling tight clothing but struggling with loose garments. Due to the limited expressiveness of traditional meshes, researchers have combined SMPL with advanced rendering techniques. Some methods [ 7 , 56 , 22 , 35 ] use NeRF [ 39 ] to implicitly express human bodies, mapping points from

Read the complete text