DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation
Peng Chen, Xiaobao Wei, Ming Lu, Hui Chen, Feng Tian · 2025
Read DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation by Peng Chen, Xiaobao Wei, Ming Lu, Hui Chen, Feng Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18159.txt.
Introduction
Peng Chen and co-authors compress speech-driven talking heads with a distillation guided by personal style. Their 2025 paper builds three-dimensional faces that speak from audio alone. Diffusion models give varied, lifelike motion, but they are bulky and slow, and they miss a speaker's personal style: lip habits and emotional tone.
A contrastive personalizer learns identity and emotion codes from the voice itself. Those codes steer the teaching of a smaller, faster student model. The aim is a talking head that still sounds and moves like a particular person, on hardware that can run it live, for telepresence and virtual characters.
Catalog record
| CREATOR | Peng Chen, Xiaobao Wei, Ming Lu, Hui Chen, Feng Tian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation Peng Chen, Xiaobao Wei, Ming Lu, Hui Chen, Feng Tian Abstract Real-time speech-driven 3D facial animation has been attractive in academia and industry. Traditional methods mainly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the nondeterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. Existing diffusion-based methods can improve the diversity of facial animation. However, personalized speaking styles conveying accurate lip language is still lacking, besides, efficiency and compactness still need to be improved. In this work, we propose DiffusionTalker to address the above limitations via personalizer-guided distillation. In terms of personalization, we introduce a contrastive personalizer that learns identity and emotion embeddings to capture speaking styles from audio. We further propose a personalizer enhancer during distillation to enhance the influence of embeddings on facial animation. For efficiency, we use iterative distillation to reduce the steps required for animation generation and achieve more than 8x speedup in inference. To achieve compactness, we distill the large teacher model into a smaller student model, reducing our model’s storage by 86.4% while minimizing performance loss. After distillation, users can derive their identity and emotion embeddings from audio to quickly create personalized animations that reflect specific speaking styles. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released at: https://github.com/ChenVoid/DiffusionTalker . 2 2 footnotetext: Corresponding Author. I Introduction Speech-driven 3D facial animation is pivotal in applications such as virtual reality [ 1 , 2 , 3 , 4 , 5 ] , augmented reality [ 6 ] , and computer games [ 7 ] . This technology involves predicting facial parameters from audio sequences to drive a 3D facial model for animation. Real-time speech-driven 3D animation of special interests in communication anywhere, such as on mobile devices or VR glasses, which places high demands on the accuracy of lip movement, efficiency of inference speed and the compactness of model parameters. With the advent of deep learning, data-driven techniques for speech-driven 3D animation have gained popularity [ 8 , 9 , 10 , 11 ] , offering better performance. VOCA [ 12 ] utilized CNNs for speech-driven 3D face generation with 12 FLAME-based facial models [ 13 ] , but ignored emotional speech impacts on expressions. EmoTalk [ 14 ] , using Transformers [ 15 ] , introduced an emotion disentangling encoder, but was limited by its reliance on user-defined one-hot identity encoding, restricting personalized speaking style capture. SelfTalk [ 16 ] uses a self-supervised approach, focusing on facial action units for better lip movement accuracy, but adds redundant encoders and decoders, increasing storage needs. Most methods employ deterministic networks, while speech-driven 3D facial animation is inherently nondeterministic [ 17 ] . FaceDiffuser [ 17 ] addresses this with a diffusion architecture, yielding impressive results, but its inference speed is slowed by a denoising process that requires thousands of steps. Fig. 1: The illustration of DiffusionTalker. We reduce the steps of the diffusion model for faster inference and compress the model size for compactness by personalizer-guided distillation. Our distilled 2-step model surpasses the state-of-the-art methods in terms of emotional expression and lip accuracy, while also achieving the fastest inference speed and the fewest model parameters. Overall, current methods face three primary limitations. First, the slow inference speed of existing approaches is a significant challenge, particularly for real-time applications, with long-step diffusion-based methods being especially impacted. Second, many methods rely on large pre-trained audio encoders, such as HuBERT [ 18 ] and Wav2Vec [ 19 ] , which introduces redundancy in model parameters and leads to excessive storage requirements. Third, most methods struggle to effectively integrate identity and emotion in 3D face animation. In practice, even when different people say the same words with the same emotions, their facial movements remain unique. And a person’s expressions can vary greatly with their emotional state, even when repeating the same phrases. To address existing limitations, we propose DiffusionTalker, an efficient and compact model that combines identity and emotion information to generate personalized 3D facial animations from speech, as illustrated in Fig. 1 . A contrastive personalizer module is proposed, in which identity and emotion are extracted from the audio input through contrastive learning. Using a cross-attention mechanism, these embeddings are fused into a personalized embedding, which serves as a key conditioning factor for the denoising process. This personalized embedding guides the motion decoder in generating customized facial animations. To boost efficiency, a personalizer-guided distillation approach is used to iteratively distill a teacher model with N steps into a student model with n steps (N>n) , thereby accelerating the inference speed. For model compression, we distill the large pre-trained audio encoder from the teacher model into a smaller encoder in the student model, significantly reducing both model parameters and storage requirements. Our main contributions are summarized as follows. • We introduce DiffusionTalker, an efficient and compact 3D face diffuser that generates personalized 3D facial animations based on the diffusion model. • A personalizer-guided distillation method is employed to iteratively reduce the denoising steps, significantly accelerating inference and achieving a speed-up of more than 8x. Furthermore, we distill a large model into a smaller one, resulting in 86.4% reduction in model size. • We introduce a contrastive personalizer that learns identity and emotion embeddings using contrastive learning, which are then integrated into a personalized embedding through a cross-attention mechanism. • A personalizer enhancer is applied during distillation to bring personalized embeddings of the same identity-emotion closer together and to push those of different identity-emotions farther apart, further enhancing the model’s ability to generate personalized facial animations. Fig. 2: Pipeline of DiffusionTalker. DiffusionTalker employs a contrastive personalizer to extract audio features and personalized embeddings from the input speech. These representations serve as conditioning inputs to guide the motion decoder in denoising noisy facial animations effectively. In personalizer-guided distillation process, the number of steps in the student model is iteratively reduced to half of the original, significantly accelerating inference. Simultaneously, the model parameters are compressed to create a more compact and efficient model. The personalizer enhancer integrates the personalized embedding with the facial areas of the predicted results, leveraging contrastive learning to strengthen the embedding’s representational capacity. II Related Work Speech-driven 3D Facial Animation. Methods for speech-driven 3D facial animation can be divided into two branches: phoneme-based methods [ 20 ] and data-driven based methods [ 21 , 22 , 23 ] . Phoneme-based methods, such as JALI [ 7 , 24 ] , utilize the mapping from phonemes to visemes and come with the advantage of simplified animation control [ 17 ] . However, they require intermediary representations of phonemes for co-articulation. Unlike phoneme-based methods, data-based methods can learn the mapping between audio and facial animation automatically. VOCA [ 12 ] uses CNN to map voice to 3D mesh. While it considers the influence of different individuals’ identities on animation generation, it does not account for emotions. MeshTalk [ 25 ] attempts to disentangle audio and 3D animation in a categorical latent space but suffers from long inference times. FaceFormer [ 8 ] employs a transformer as its model backbone