MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October

Chang Liu, Ye Pan, Chenyang Ding, Susanto Rahardja, Xiaokang Yang · 2025

Read MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October by Chang Liu, Ye Pan, Chenyang Ding, Susanto Rahardja, Xiaokang Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06071.txt.

Introduction

Chang Liu and four co-authors present a study from twenty twenty-five on three dimensional face animation driven by speech, with live emotion. Most past tools fix mood to a set label, which flattens the range of looks. Their frame, MEDTalk, splits the codes for what is said from the codes for how it feels. A cross rebuild step in training keeps the two apart.

Sound and text then drive lip moves and frame by frame shifts in mood strength. Users can steer the face with text prompts or a sample look. The output fits a known avatar pipeline used in film and games. The team shares its code in public. The aim is faces that shift mood as speech flows.

Catalog record

CREATORChang Liu, Ye Pan, Chenyang Ding, Susanto Rahardja, Xiaokang Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06071

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754933 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Animation Chang Liu, Ye Pan, Chenyang Ding, Susanto Rahardja, Xiaokang Yang Abstract . Audio-driven emotional 3D facial animation aims to generate synchronized lip movements and vivid facial expressions. However, most existing approaches focus on static and predefined emotion labels, limiting their diversity and naturalness. To address these challenges, we propose MEDTalk , a novel framework for fine-grained and dynamic emotional talking head generation. Our approach first disentangles content and emotion embedding spaces from motion sequences using a carefully designed cross-reconstruction process, enabling independent control over lip movements and facial expressions. Beyond conventional audio-driven lip synchronization, we integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions. Furthermore, to enhance control and personalization, we incorporate multimodal inputs—including text descriptions and reference expression images—to guide the generation of user-specified facial expressions. With MetaHuman as the priority, our generated results can be conveniently integrated into the industrial production pipeline. The code is available at: https://github.com/SJTU-Lucy/MEDTalk . 1. Introduction 3D talking head generation aims to synthesize realistic facial animations from speech or text inputs and has become a significant research area in computer graphics and artificial intelligence. This technology has broad applications in virtual assistants, digital avatars, gaming, and education ( Korban and Li 2022 ; Al Farsi et al. 2021 ) , enhancing human-computer interaction through lifelike digital characters. Compared to traditional manual animation, which is time-consuming and labor-intensive, data-driven approaches significantly improve efficiency while producing expressive and natural facial animations. Figure 1. Results of MEDTalk. Given an input speech, MEDTalk produces expressive facial movements based on emotion labels, reference images, or text descriptions, with natural emotional transitions. Early studies ( Edwards et al. 2016 ; Pham et al. 2017 ; Pham et al. 2018 ; Zhou et al. 2018 ; Cudeiro et al. 2019 ; Richard et al. 2021 ) primarily focused on lip synchronization while neglecting movements in other facial regions, leading to unnatural animations. Recent methods have introduced emotion control by incorporating emotion or intensity labels ( Fan et al. 2022 ; Pan et al. 2023 ; Liu et al. 2024 ) , enhancing the expressiveness of facial animations. However, label-based approaches impose a fixed emotional state throughout the animation, limiting diversity and expressiveness. Some studies ( Peng et al. 2023b ; Kim et al. 2024 ) try to extract emotion features from the input audio to improve the diversity of facial expressions. However, emotional features are not fully represented in audio, reflected in inaccurate predictions. We argue that high-quality facial animation relies on two key components: lip synchronization and emotional expressiveness. Lip movements must align precisely with speech audio, while emotional expressions should exhibit both diversity and dynamic micro-expressions to enhance realism. To address these issues, we propose MEDTalk : M ultimodal Controlled 3D Facial Animation with Dynamic E motions by D isentang-led Embedding, as shown in Fig. 1 . Our framework disentangles emotion and content features through self-supervised training on motion sequences, enabling independent control of lip synchronization and facial expressions. Apart from precise control over lip synchronization, MEDTalk also supports multiple input modalities, including conventional emotion labels, textual descriptions, and reference images, allowing for fully controllable expression generation. Additionally, we employ cross-attention to integrate features of audio and spoken text, predicting frame-wise emotion intensity to get dynamic emotion features. This approach enables the generation of highly accurate lip-synced facial animations with fine-grained, controllable, and dynamically evolving emotions. Our key contributions are as follows: • We introduce MEDTalk, an emotional 3D talking head framework that generates facial animation with fine-grained and multimodal-guided emotion states. • We propose a novel paradigm that disentangles content and emotion through self-supervised training, enabling independent control of lip and facial expressions. • We integrate audio and speech text to predict frame-wise emotional intensity without requiring frame-wise annotations, generating dynamic emotional facial animation. • We introduce multimodal guidance, overcoming the limitations of predefined labels in previous studies and enabling user control, improving diversity in facial animation. 2. Related Work 2.1. Audio-driven 3D Facial Animation The goal of audio-driven talking head generation is to animate a target character based on an audio clip while ensuring accurate lip synchronization. With the advancement of deep learning, various model architectures have been utilized to generate expressive and realistic 3D facial animations. Early works ( Pham et al. 2018 ; Cudeiro et al. 2019 ) primarily focused on obtaining realistic, speaker-independent lip shapes from mel spectrograms using convolutional neural networks (CNNs). Transformer-based approaches, such as FaceFormer ( Fan et al. 2022 ) , leveraged wav2vec2.0 ( Baevski et al. 2020 ) for audio feature extraction and synthesized continuous motion via autoregressive modeling, improving lip synchronization accuracy. CodeTalker ( Xing et al. 2023 ) employed a VQ-VAE framework, treating facial animation generation as a discrete code query task, enhancing robustness. VividTalker ( Zhao et al. 2023 ) further refined this approach by encoding head pose and mouth motion into separate discrete latent spaces, using audio features for autoregressive motion prediction. Meanwhile, SelfTalk ( Peng et al. 2023a ) introduced a self-supervised framework that aligns generated lip shapes with corresponding speech text, incorporating text-audio consistency as additional supervision. Recent methods ( Chen et al. 2023 ; Stan et al. 2023 ; Sun et al. 2024b ) have explored the diffusion model ( Ho et al. 2020 ) , leveraging its generative diversity to enhance realism and motion diversity in talking head generation Despite these advancements, most existing methods primarily focus on audio-synchronized lip movements while neglecting expressive variations in other facial regions. As a result, generated animations often appear rigid and emotionless. In contrast, our approach, MEDTalk, allows for expressive and natural facial animations with fine-grained emotional variations. 2.2. Emotional Talking Head Emotional talking head generation incorporates emotional information, either manually controlled or extracted from audio, to produce facial animations with expressive emotional features. Some works apply predefined emotion labels to control the generated facial expression, such as EMOTE ( Daněček et al. 2023 ) , EmoFace ( Liu et al. 2024 ) , and FlowVQTalker ( Tan et al. 2024a ) . Some works have explored extracting emotion features from audio to improve expressiveness ( Ji et al. 2021 ; Peng et al. 2023b ; Tan et al. 2023 ) . They applied cross-reconstruction to separate emotional cues from speech. DEEPTalk ( Kim et al. 2024 ) constructed a joint feature space for audio and expressions, connecting the emotional features between them. AVI-Talking ( Sun et al. 2024a ) utilized a large language model (LLM) to analyze emotion features in speech. Despite these advancements, most existing methods rely on static emotion features, resulting in facial animations that lack variation. To address this, DEITalk ( Shen et al. 2024 ) combined emotion labels with dynamically estimated emotion intensities derived from a speech-expression joint space. However, the predicted intensity lacked supervision and was heavily correlated with input audio. In contrast, our approach

Read the complete text