SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

Farzaneh Jafari, Stefano Berretti, Anup Basu · 2026

Read SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization by Farzaneh Jafari, Stefano Berretti, Anup Basu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13335.txt.

Introduction

Jafari, Berretti, and Basu introduce SEDTalker, a system for speech-driven 3D facial animation that tracks emotion frame by frame. Earlier systems rely on whole-sentence or hand-set emotion labels, but theirs predicts dense emotion types and strengths from speech at each frame. Those signals steer a hybrid Transformer-Mamba animation model, keeping word content apart from emotional style while preserving identity and timing. Tests show strong frame-level emotion recognition, low shape and timing errors, and smooth shifts between expressions.

Catalog record

CREATORFarzaneh Jafari, Stefano Berretti, Anup Basu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35611
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization Farzaneh Jafari, Stefano Berretti, Anup Basu Abstract We introduce SEDTalker, an emotion-aware framework for speech-driven 3D facial animation that leverages frame-level speech emotion diarization to achieve fine-grained expressive control. Unlike prior approaches that rely on utterance-level or manually specified emotion labels, our method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. The diarized emotion signals are encoded as learned embeddings and used to condition a speech-driven 3D animation model based on a hybrid Transformer-Mamba architecture. This design allows effective disentanglement of linguistic content and emotional style while preserving identity and temporal coherence. We evaluate our approach on a large-scale multi-corpus dataset for speech emotion diarization and on the EmoVOCA [ 15 ] dataset for emotional 3D facial animation. Quantitative results demonstrate strong frame-level emotion recognition performance and low geometric and temporal reconstruction errors, while qualitative results show smooth emotion transitions and consistent expression control. These findings highlight the effectiveness of frame-level emotion diarization for expressive and controllable 3D talking head generation. Our code and videos are available at https://SEDTalker.github.io/ . Figure 1: SEDTalker is a speech-driven framework for emotion-aware 3D facial animation. By leveraging frame-level speech emotion diarization, the model captures fine-grained emotional cues and directly translates them into 3D facial parameters for precise expressive control. 1 Introduction Speech-driven 3D facial animation has made significant progress in recent years, enabling realistic lip synchronization and identity-preserving motion from raw audio signals. Modern applications span virtual assistants, digital avatars, film production, and telepresence systems, where generating emotionally expressive and temporally coherent facial animations is essential for natural human-computer interaction. However, while existing methods excel at speech-to-motion mapping, they struggle to capture the dynamic and nuanced nature of emotional expression in speech. A fundamental challenge lies in the temporal granularity of emotion modeling. Human emotional expression is inherently dynamic; emotions shift, blend, and vary in intensity throughout an utterance. Yet most existing methods condition animation models on a single global emotion category for the entire utterance, limiting expressiveness and failing to capture natural emotion transitions. This coarse-grained approach results in facial animations that appear emotionally flat or exhibit abrupt, unnatural expression changes. Furthermore, the scarcity of high-quality emotional speech datasets exacerbates this problem. Datasets like EmoVOCA [ 15 ] provide richly expressive facial motion but only neutral speech, preventing models from learning direct correlations between emotional acoustic features and facial expressions. This data limitation has been a persistent bottleneck in developing truly emotion-aware talking head systems. In this work, we address these limitations by introducing Speech Emotion Diarization (SED) as a core component for emotional facial animation. Unlike utterance-level classification, SED predicts emotion categories and intensities at fine temporal resolution directly from speech, providing dense, time-varying affective signals that capture emotion transitions, intensity fluctuations, and mixed emotional states within utterances, capabilities impossible with coarse labels. To overcome data scarcity, we propose a decoupled training strategy: the animation model trains on neutral speech paired with emotional facial sequences, while a pre-trained SED model extracts emotion signals at inference time. This approach enables effective emotion transfer without requiring aligned emotional speech-face training pairs and makes the framework adaptable to new emotions without retraining. The main contributions of this work are the following: 1) We formulate speech-driven 3D facial animation as a frame-level emotion-conditioned generation task, enabling continuous, intra-utterance emotion modulation that utterance-level approaches cannot capture by design. 2) By separating SED from animation training, we eliminate the requirement for paired emotional speech-3D mesh data, a bottleneck that has blocked prior work. 3) We construct and release a frame-level emotion diarization corpus of 58,834 utterances aggregated from nine publicly available datasets, covering 149 speakers, four English accents, and diverse recording conditions, achieving a 78.9% weighted F1 at 20 ms resolution. 4) We extend the JambaTalk [ 14 ] backbone with an additive emotion–intensity conditioning module that separates linguistic content from emotional style while maintaining temporal coherence and reduces GPU memory usage to 13 GB for 18 emotion–intensity states plus a neutral state. 2 Related Work Speech-Driven 3D Facial Animation. Recent deep learning approaches have achieved impressive results in generating realistic talking heads from speech. Early works such as VOCA [ 15 ] introduced neural networks for voice-operated character animation, laying the foundation for learning-based audio-to-mesh synthesis. Transformer-based architectures have since dominated the field: FaceFormer [ 18 ] applies cross-modal attention between audio and motion sequences to capture long-range dependencies; CodeTalker [ 19 ] introduces discrete motion priors through VQ-VAE to improve motion quality; SelfTalk [ 21 ] proposes self-supervised commutative training for better disentanglement of content and style. More recently, diffusion-based approaches like FaceDiffuser [ 20 ] leverage denoising probabilistic models for diverse motion generation. While these methods achieve high-fidelity lip synchronization and identity preservation, they typically operate on neutral or weakly emotional speech and lack explicit mechanisms for fine-grained emotional control. Emotion-Aware Facial Animation. Several works have attempted to incorporate emotion into speech-driven animation. EmoTalk [ 22 ] introduces emotional disentanglement by conditioning on utterance-level emotion labels and style vectors, achieving better emotional expressiveness compared to neutral baselines. However, EmoTalk [ 22 ] relies on a single emotion label per utterance and requires manually annotated or pre-classified emotion categories at training time. These approaches use global emotion embeddings or style codes, which cannot capture intra-utterance emotion dynamics. More critically, these approaches require paired emotional speech and facial motion data, which severely limits their applicability given dataset scarcity. In contrast, our method predicts frame-level emotion signals directly from speech at inference time and supports continuous emotion intensity control without requiring emotional speech during training. Speech Emotion Recognition (SER). Traditional SER systems classify entire utterances into discrete emotion categories. Recent advances leverage self-supervised speech representations: Wav2Vec 2.0 [ 13 ] and WavLM [ 2 ] pre-trained on large-scale unlabeled speech provide robust acoustic features that transfer well to emotion recognition tasks. However, most SER works operate at utterance-level granularity, producing a single emotion label per audio segment. Frame-level emotion recognition, predicting emotion at fine temporal resolution, has received less attention despite its importance for capturing temporal dynamics. Our work bridges this gap by training a frame-level emotion diarization model on a multi-corpus dataset of 58,834 utterances and demonstrating its effectiveness for downstream animation conditioning. Table 1: Dataset composition and characteristics. Dataset Utterances Style Speakers Contrib.% MELD [ 3 ] 12,070 Conv. Multi 20.5 IEMOCAP [ 4 ] 11,032 Script+Improv 10 18.7 JL-Corpus [ 5 ] 10,661 Acted 4 18.1 ESD [ 6 ] 10,500 Professional 10 17.8 CREMA-D [ 7 ] 7,438 Acted 91 12.6 EmoV-DB [ 8 ] 3,353 Varied 4 5.7 TESS [ 9 ] 2,400 Acted 2 4.1 RAVDESS [ 10 ] 960 Acted 24 1.6 SAVEE [ 11 ] 420 Acted 4 0.7 Total 58,834 Diverse 149 100 Table 2: Emotion distribution and normalized class weights after frame-level propagation (20 ms). Emotion Utt. Utt.% Frames (M) Frame% Weight Happy 16,133 27.4 4.00 32.1 0.31 Angry 15,115 25.7 4.05 32.4 0.34 Sad 10,558 17.9 2.30 18.5 0.48 Neutral 8,041 13.7 0.20 1.6 0.63 Upset 3,573 6.1 1.12 9.0 1.42 Disgust 3,054 5.2 0.53 4.3 1.66 Fear 2,360 4.0 0.55 2.0 2.15 Total 58,834 100

Read the complete text