Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control

Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang Yuan Zhang Pengfei Wan Di Zhang Shuai Li · 2025

Read Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control by Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang Yuan Zhang Pengfei Wan Di Zhang Shuai Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14517.txt.

Introduction

Hejia Chen and co-authors animate 3D talking faces with both broad and fine control over expression, in a 2025 paper. Older systems set one emotion label for a whole clip and cannot tweak a brow or a cheek mid sentence. Their diffusion transformer takes speech plus layered controls, from coarse emotion down to single facial action units.

Training runs in two stages so fine controls do not wreck lip sync. A label swapping trick lets fine instructions win when they conflict, and a mask trick dials their strength. Users can even type the expression in words. Tests topped rivals on lip sync and expressiveness.

Catalog record

CREATORHejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang Yuan Zhang Pengfei Wan Di Zhang Shuai Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14517

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang Yuan Zhang Pengfei Wan Di Zhang Shuai Li Abstract Speech-driven 3D talking face method should offer both accurate lip synchronization and controllable expressions. Previous methods solely adopt discrete emotion labels to globally control expressions throughout sequences while limiting flexible fine-grained facial control within the spatiotemporal domain. We propose a diffusion-transformer-based 3D talking face generation model, Cafe-Talk , which simultaneously incorporates c o a rse- and f in e -grained multimodal control conditions. Nevertheless, the entanglement of multiple conditions challenges achieving satisfying performance. To disentangle speech audio and fine-grained conditions, we employ a two-stage training pipeline. Specifically, Cafe-Talk is initially trained using only speech audio and coarse-grained conditions. Then, a proposed fine-grained control adapter gradually adds fine-grained instructions represented by action units (AUs), preventing unfavorable speech-lip synchronization. To disentangle coarse- and fine-grained conditions, we design a swap-label training mechanism, which enables the dominance of the fine-grained conditions. We also devise a mask-based CFG technique to regulate the occurrence and intensity of fine-grained control. In addition, a text-based detector is introduced with text-AU alignment to enable natural language user input and further support multimodal control. Extensive experimental results prove that Cafe-Talk achieves state-of-the-art lip synchronization and expressiveness performance and receives wide acceptance in fine-grained control in user studies. Project page: https://harryxd2018.github.io/cafe-talk/ Figure 1: Adding multimodal coarse- and fine-grained control enables more flexible animations: Scenario : A senior actor is arguing with the director about how to smile. Action : The actor responds with anger and concludes with a sudden sarcastic laugh . 1 Introduction Generating realistic speech-driven 3D facial animation holds significant application value in the traditional film industries and advanced AI applications. It is a challenging task that necessitates synchronized lip motions and the conveyance of vivid, compelling facial expressions. Moreover, an animator-friendly and agent-driven application, as illustrated in Fig. 1 , requires fine-grained and multimodal control capabilities. Although existing methods have achieved impressive progress in facial expression control by using emotion labels ( Daněček et al., 2023 ; Haque & Yumak, 2023 ) or multimodal emotional references ( Xu et al., 2023 ; Peng et al., 2023b ) for entire sequences, modeling and flexibly controlling fine-grained facial expressions during the speech is still unexplored. Our motivation is to present a 3D talking face method with effective, precise, and flexible controllability. To achieve this, the controlling condition is spatiotemporally modeled as coarse- and fine-grained conditions, where the coarse-grained condition establishes a foundation for the produced facial movements and the fine-grained condition enriches the details. Spatially, the coarse-grained conditions abstractly depict overall facial movements through talking style and emotion. Conversely, the fine-grained condition emphasizes specific and localized muscle movements, such as blinking and raising the eyebrows. Temporally, the coarse-grained condition remains constant during speaking, while the fine-grained condition describes the instant facial expressions. Existing methods ( Daněček et al., 2023 ; Peng et al., 2023b ) focus on modeling the talking style and emotion temporal-consistently, neglecting fine-grained facial movements in spatiotemporal domains, which results in an inability to control intricate local motions and flexible expression changes. To tackle the aforementioned challenge, in this paper, we present Cafe-Talk as the first 3D talking face generation method with c o a rse- and f in e -grained multimodal control. In order to achieve detailed and flexible control within the current diffusion-based framework, an intuitive approach is to directly implement additional fine-grained conditions and jointly train the network alongside speech audio and coarse-grained conditions. However, the entanglement of the three conditions results in unsynchronized lip movements and an inability to fine-grained control based on our preliminary experimental findings. Specifically, (1) the entanglement of the audio and fine-grained condition provides a shortcut to producing lip movements and weakening speech audio guidance, as the fine-grained conditions can also describe local muscle motions around the mouth. To this end, Cafe-Talk is designed as a coarse-to-fine structure with a two-stage pipeline. In the first stage, a diffusion-based transformer ( Ng et al., 2024 ; Zhao et al., 2024 ) generates the facial movements with encoded speech audio and incorporates the multimodal coarse-grained condition following Peebles & Xie (2023) to achieve efficient global control. In the second stage, the fine-grained conditions represent a sequence of action units (AUs), which flexibly guide the corresponding muscles to stay excited. Although existing methods ( Ma et al., 2023 ; Wang et al., 2024 ; Sun et al., 2024 ) utilize AUs to control facial motions, they merely refine the coarse-grained control instead of disentangle the detail controllability. With the frozen base model, we design a fine-grained control adapter to gradually add the fine-grained condition into the pipeline without jeopardizing lip movements benefiting from a zero convolution layer ( Zhang et al., 2023a ) . Moreover, (2) to achieve effective, precise, and flexible fine-grained control, we first build a swap-label training mechanism upon the two-stage pipeline to introduce conflict coarse- and fine-grained conditions, enabling predominant fine-grained conditional control. We then design a mask-based classifier-free guidance (CFG) technique for inference, which ensures the occurrence of fine-grained conditions and allows for effective and accurate intensity control. We finally introduce a text-based detector for language-based emotion and expression description by aligning it with AUs on a projected CLIP space ( Radford et al., 2021 ) , extending flexible multimodal fine-grained control. We thoroughly validate the effectiveness of our proposed controllable talking face generation method through extensive experiments. To the best of our knowledge, we are the first to address the fine-grained 3D talking face control task. Thus, we compare the lip synchronization and coarse-grained expression control capabilities with existing methods and achieve state-of-the-art performances. Additionally, we conduct ablation experiments to demonstrate the importance of each proposed module in achieving fine-grained control. Finally, we execute a user study and received wide acceptance of the effectiveness of our control. The contributions of this paper can be summarized as follows: 1. We propose Cafe-Talk, the first 3D talking face generation model enabling coarse- and fine-grained multimodal conditional control in an effective, precise, and flexible manner. 2. We design a two-stage training pipeline that explicitly disentangles speech audio and fine-grained conditions. Additionally, we devise a fine-grained control adapter that gradually introduces fine-grained conditions without jeopardizing lip synchronization. 3. We introduce a swap-label training mechanism and mask-based CFG technique in the inference process to disentangle coarse- and fine-grained conditions, which achieves dominant and precise localized facial control. 4. We devise a text-based AU detector to efficiently extract AU fine-grained conditions from natural language descriptions based on user intent, which enables our model to support multimodal control. 2 Related work Talking face animation has attracted considerable attention due to its significant application potential and market demand. This technology can be broadly divided into two categories based on the output format: video-based and 3D-based methods. Video-based methods ( Zhou et al., 2020 ; Wang et al., 2020 ; Liu et al., 2023 ; Guo et al., 2024 ) focus on generating realistic portrait videos, particularly benefiting from advanced neural rendering techniques ( Guo et al., 2021 ; Shen et al., 2022 ; Ye et al., 2023 ; Peng et al., 2024 )

Read the complete text