From Flat to Feeling: A Feasibility and Impact Study on Dynamic Facial Emotions in AI-Generated Avatars

Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Pål Halvorsen · 2025

Read From Flat to Feeling: A Feasibility and Impact Study on Dynamic Facial Emotions in AI-Generated Avatars by Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Pål Halvorsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13477.txt.

Introduction

Pegah Salehi and co-authors study lifelike child avatars with moving facial emotions in a 2025 paper. On the head's account, the work tackles the challenge of blending sound, face, and voice in expressive avatars for training. Such avatars could train professionals who interview abused children, a task where practice cases are sensitive.

Their real-time system pairs a game engine's digital humans with a tool that drives faces from the voice's tone. Seventy people rated the avatars with sound and face together, or face alone. A voice-age mismatch, since adult female voices dubbed the children, may have skewed the fit. The study tests what helps users believe the avatar's feelings.

Catalog record

CREATORPegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Pål Halvorsen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13477

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Flat to Feeling: A Feasibility and Impact Study on Dynamic Facial Emotions in AI-Generated Avatars Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Pål Halvorsen Abstract Dynamic facial emotion is essential for believable AI-generated avatars; however, most systems remain visually inert, limiting their utility in high-stakes simulations such as virtual training for investigative interviews with abused children. We introduce and evaluate a real-time architecture fusing Unreal Engine 5 MetaHuman rendering with NVIDIA Omniverse Audio2Face to translate vocal prosody into high-fidelity facial expressions on photorealistic child avatars. We implemented a distributed two-PC setup that decouples language processing and speech synthesis from GPU-intensive rendering, designed to support low-latency interaction in desktop and VR environments. A between-subjects study ( N = 70 N=70 ) using audio+visual and visual-only conditions assessed perceptual impacts as participants rated emotional clarity, facial realism, and empathy for two avatars expressing joy, sadness, and anger. Results demonstrate that avatars could express emotions recognizably, with sadness and joy achieving high identification rates. However, anger recognition significantly dropped without audio, highlighting the importance of congruent vocal cues for high-arousal emotions. Interestingly, removing audio boosted perceived facial realism, suggesting that audiovisual desynchrony remains a key design challenge. These findings confirm the technical feasibility of generating emotionally expressive avatars and provide guidance for improving non-verbal communication in sensitive training simulations. K eywords Human-Computer Interaction; Interactive Avatars; Emotionally Expressive Avatars; Perceived Realism, Unreal Engine MetaHuman; NVIDIA Audio2Face 1 Introduction AI-generated avatars are rapidly becoming integral components of digital interaction across diverse domains, with applications spanning virtual, augmented, and mixed reality (VR/AR/XR) platforms, virtual telepresence, embodied digital assistants, e-commerce, entertainment, education, and potential therapeutic interventions. The efficacy and adoption of these avatars critically depend on their perceived believability and their capacity to engage users in naturalistic interactions. For an avatar to "feel alive" and effectively bridge the human-computer interaction gap, it must convincingly replicate key facets of human social signaling [ 1 ] . Among these, the dynamic expression of facial emotions stands out as particularly vital. Facial expressions serve as a primary, often subconscious, channel for non-verbal communication, conveying a wealth of affective information that shapes interpersonal understanding, relationship dynamics, and the interpretation of intent [ 2 ] . Consequently, an avatar’s ability to display appropriate, recognizable, and dynamically evolving facial emotions is paramount for creating engaging, trustworthy, and effective virtual agents. This paper investigates the technical feasibility of creating such avatars and evaluates their perceptual impact, particularly in training scenarios. Figure 1: Example frames of the two avatars used in the user study: "Emory" (male-presenting) and "Amelia" (female-presenting). The top row displays their neutral facial expressions. The bottom rows show expressions of joy (left) and sadness (right) for each avatar, as used in the study. While the importance of such affective realism is broadly recognized, its consistent and effective implementation is foundational and challenging in high-stakes domains. One such domain is simulation-based training for professionals who conduct forensic interviews with children, often concerning experiences of abuse or trauma. In this context, previous virtual-child systems have primarily focused on dialogue flow and question typology. However, empirical evaluations have repeatedly highlighted two significant shortcomings [ 3 ] : firstly, the avatars themselves are frequently emotionally inert, their faces remaining neutral regardless of the conversational context, thereby offering no micro-expressions that might signal anxiety, relief, or distress. Secondly, practitioners often find the overall visual fidelity of these avatars inadequate, with subtle inconsistencies in proportions, shading, or motion disrupting immersion and undermining trust in the simulation. Collectively, these deficits impede core learning objectives; interviewers are unable to accurately judge whether a question soothes or unsettles the virtual child, cannot adequately fine-tune their empathic responses, and consequently struggle to transfer refined interviewing skills to live cases where non-verbal cues are indispensable. Furthermore, the capacity to render these digital characters in real time is essential for such interactive applications, facilitating immediate responsiveness and a heightened sense of presence. Addressing these limitations requires a focused research endeavor into developing AI avatar systems capable of rich, dynamic facial emotion expression with real-time responsiveness. This research directly tackles this objective by introducing and assessing a novel, distributed architecture engineered for high-fidelity emotional expressiveness. This system represents a significant advancement by integrating the sophisticated rendering capabilities of Unreal Engine 5 (UE5) MetaHuman software [ 4 ] with the intelligent, audio-driven facial animation provided by NVIDIA’s Omniverse Audio2Face [ 5 ] . The system’s distributed architecture employs two synchronized computational nodes to effectively balance processing demands—decoupling language understanding and speech generation from GPU-intensive rendering—thereby enabling low-latency interaction. Figure 1 displays representative screenshots of the two photorealistic MetaHuman avatars, "Amelia" and "Emory," utilized in this study. In summary, our main contributions are as follows: • Designing and implementing a modular pipeline for generating emotionally expressive avatars, emphasizing real-time facial animation using NVIDIA Audio2Face and Unreal Engine MetaHuman. • Conducting a user study ( N=70 ) comparing audio+visual and visual-only conditions, highlighting trade-offs between cross-modal interference and perceived realism. • Demonstrating that high-arousal emotions (e.g., anger) rely heavily on audio-visual congruence, while low-arousal emotions (e.g., sadness) achieve robust recognition through visual cues alone. • Quantifying how facial structure (angular vs. soft) shapes emotion perception, guiding context-specific avatar design. • Identifying key challenges in avatar design and implementation, such as synchronization, the uncanny valley, and expression subtlety, through qualitative feedback analysis. 2 Related Work The pursuit of emotionally expressive AI avatars lies at the confluence of affective computing, human-computer interaction (HCI), and advancements in multimodal systems, all striving to endow machines with a semblance of human emotional intelligence [ 6 , 7 , 8 , 9 ] . While foundational work in affective computing centered on the detection and modeling of emotional cues, the emphasis has progressively shifted towards the dynamic and adaptive expression of these emotions in avatars to foster user engagement and perceived believability. This aligns with broader HCI principles where anthropomorphism in digital agents consistently correlates with enhanced user engagement, perceived credibility, and overall satisfaction [ 10 , 11 , 12 , 13 ] . The significance of expressive capabilities is underscored by research in applied domains like virtual therapy and education, where emotionally inert or insufficiently expressive avatars can lead to misunderstandings, diminished rapport, and ultimately, reduced system effectiveness [ 14 , 15 , 16 , 17 ] . This need for dynamic emotional expressiveness becomes particularly acute in high-stakes interaction scenarios, such as the virtual training environments for child forensic interviews. In these contexts, an avatar’s ability to display contextually appropriate and recognizable emotions can profoundly influence user perception and training outcomes. However, existing virtual-child systems tailored for such training have predominantly concentrated on dialogue management and question typologies. Empirical evaluations of these systems have consistently identified critical shortcomings: specifically, avatars often lack dynamic facial affect, remaining emotionally static, and their overall visual fidelity is frequently judged as inadequate [ 3 ] . These deficiencies impair training efficacy, as trainees cannot accurately gauge a child’s non-verbal reactions to their questioning or refine their empathic responses, which are skills paramount in real-world interviews. The technological evolution towards more expressive avatars has been considerable. Initial efforts often involved 2D video-based avatars that, while capable of displaying emotions, tended to produce rigid outputs through direct

Read the complete text