RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation
Peng Chen Xiaobao Wei Yi Yang Naiming Yao Hui Chen Feng Tian · 2026
Read RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation by Peng Chen Xiaobao Wei Yi Yang Naiming Yao Hui Chen Feng Tian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10606.txt.
Introduction
Peng Chen, Xiaobao Wei, Yi Yang, Naiming Yao, Hui Chen, and Feng Tian build RSATalker, a system that animates talking heads for conversation. Such avatars matter more as virtual reality grows, above all in social scenes with turns of talk. Older methods split into two camps. Mesh-based 3D methods handle two-person dialogue but look fake.
Large 2D models look natural yet cost too much compute to run smoothly. RSATalker aims for realistic, socially aware faces across multi-turn exchanges. A good avatar must react to a partner, not just recite lines. The work speaks to teams building virtual meeting and social spaces where faces carry the interaction.
Catalog record
| CREATOR | Peng Chen Xiaobao Wei Yi Yang Naiming Yao Hui Chen Feng Tian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10606
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation Peng Chen Xiaobao Wei Yi Yang Naiming Yao Hui Chen Feng Tian Abstract Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS)-based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation, with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and un-equal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech–mesh–image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and dataset will be released. 2 2 footnotetext: Corresponding author. 1 Introduction Figure 1 : Pipeline of RSATalker. We introduce RSATalker, a novel framework that leverages 3D Gaussian Splatting (3DGS) to generate highly realistic and socially-aware talking heads, specifically designed for dynamic speaker–listener interaction scenarios. In recent years, the advancement of deep learning techniques and computer graphics technologies has significantly promoted the development of talking head generation 1 ; 3 ; 6 . This capability has gained considerable importance in applications within virtual reality (VR) 21 ; 43 , especially in immersive social interaction scenarios that involve multi-turn conversation between two participants 26 ; 23 . In such settings, the roles of speaker and listener alternate dynamically over the course of dialogue. The outputs of talking head generation systems can generally be categorized into two representational paradigms: those based on 3D meshes and those relying on 2D imagery. For 3D mesh-based methods, most approaches generate vertices or blendshapes 15 using Transformer- or Diffusion-based models to drive head mesh models such as FLAME 20 , thereby enabling talking head animations. Representative works include FaceFormer 7 , EmoTalk 29 , DiffusionTalker 3 , and Learning to Listen. However, these methods are typically “speaker-only” or “listener-only.” For multi-turn conversation, DualTalk 26 made an initial attempt by modeling dual-person dialogues, which marked an important step forward. Nevertheless, these approaches struggle to achieve photorealistic textures, accurate color reproduction, and naturalistic dynamic behaviors (e.g., facial expressions or micro-movements), frequently inducing the ‘uncanny valley’ 33 ; 13 effect in VR environments due to perceptible inconsistencies that trigger visceral discomfort. In contrast, 2D image-based methods address this limitation by driving RGB human faces to generate photorealistic talking heads. Current approaches can be broadly grouped into two paradigms. First, large-scale pretrained models based on Stable Diffusion 32 or Diffusion Transformer (DIT) 24 , such as Wan 35 and HunyuanVideo-Avatar 4 , exhibit strong generalization capabilities, but their massive parameter sizes lead to extremely high computational costs, rendering them impractical for VR deployment. Second, Neural Radiance Field 22 (NeRF)- and 3D Gaussian Splatting 14 (3DGS)-based methods, such as ER-NeRF 17 , SyncTalk 27 , and GaussianTalker 5 , achieve high rendering efficiency and accuracy at relatively low computational costs, making them well-suited for VR applications. However, existing approaches remain fundamentally “speaker-only,” without modeling the dynamics of multi-turn conversation. Moreover, in the context of social interactions during multi-turn conversation, the social relationship between two speakers plays a decisive role in shaping facial expressions, head movements, and other nonverbal behaviors, a crucial factor that previous works have not taken into account 8 ; 25 . In real multi-turn conversation, these relationships strongly influence both expressiveness and responsiveness: for instance, the conversation between close friends often involve relaxed expressions, frequent smiling, and dynamic gestures, whereas professional dialogues between a superior and a subordinate tend to be more restrained, with controlled gaze shifts and formal expressions 31 ; 34 . Such interpersonal dynamics accumulate over multiple dialogue rounds, directly affecting how smoothly roles transition between speaker and listener, and determining the overall naturalness of the interaction. Ignoring this relationship leads to talking heads that may appear visually plausible but socially inconsistent, ultimately limiting their effectiveness in VR-based social communication. In summary, as shown in Fig. , current methods for talking head generation in multi-turn conversation face the following limitations: (1) Existing realistic talking head generation methods only focus on the speaker-only setting. (2) Mesh-based 3D approaches can achieve dynamic multi-turn interaction between speakers and listeners, but the results are unrealistic. (3) Existing talking head methods overlook the importance of social relationships in multi-turn conversation. Figure 2 : Pipeline of RSATalker. We introduce RSATalker, a novel framework that leverages 3D Gaussian Splatting (3DGS) to generate highly realistic and socially-aware talking heads, specifically designed for dynamic speaker–listener interaction scenarios. To address the above issues, we propose RSATalker, the first method that leverages 3DGS to achieve R ealistic and S ocially- A ware talking head generation between speakers and listeners. Our approach combines 3D mesh with 3DGS and consists of three modules: a speaker–listener motion generator, a realistic head avatar renderer, and a socially-aware module. The speaker–listener motion generator is built upon 3D mesh and integrates multiple modalities such as visual and audio cues to infer smooth facial motion. Subsequently, the realistic head avatar renderer associates 3D Gaussian primitives with the mesh’s triangular facets and exploits mesh deformations to produce high-fidelity 2D avatar videos. To incorporate the social relationship between two people during the conversation, we introduce a socially-aware module. Specifically, we categorize social relationships along two dimensions— blood vs. non-blood and equal vs. non-equal —and employ a learnable query mechanism to abstract these relationships into high-level embeddings, which are integrated into the model features. We adopt a three-stage training paradigm, consisting of two cold-start stages followed by an end-to-end training stage. Moreover, we construct the RSATalker dataset, which contains speech–mesh–image triplets of different social relationships. Extensive experiments demonstrate that our method achieves state-of-the-art performance in this field. In summary, the main contributions of this work are as follows: • We propose RSATalker, which leverages 3DGS to achieve realistic and socially-aware talking head generation for speaker–listener interaction. • We introduce a socially-aware module that incorporates the crucial interpersonal relationship between conversational partners to investigate its impact on talking head generation. • We construct the RSATalker dataset, a realistic multi-turn conversation dataset focusing on social relationships, and design a three-stage training strategy to optimize the performance of talking head generation. 2 Related Work 2.1 Realistic Speech-Driven Talking Head Generation Recent advances in talking head generation have been largely driven by large-scale pretrained models, which achieve impressive photorealism. MEMO 19 introduces a memory-guided diffusion framework that enhances long-term identity consistency and emotion-aware expression alignment through temporal modeling and audio-driven emotion modulation. Sonic 11 improves photorealism and expressiveness by incorporating audio-aligned identity units and semantic motion decomposition into a coarse-to-fine denoising pipeline, achieving accurate lip-sync and strong identity preservation across diverse speakers. EmotiveTalk 36 disentangles emotion and content in both audio and visual domains, allowing fine-grained control