Evaluating multimodal emotion recognition in proactive conversational agents: A user study Publication type: ORIGINAL ARTICLE Journal: arXiv Volume: 0
Adnana Dragut, Raquel Lacuesta, F. Xavier Gaya-Morey, Jose M. Buades-Rubio · 2026
Read Evaluating multimodal emotion recognition in proactive conversational agents: A user study Publication type: ORIGINAL ARTICLE Journal: arXiv Volume: 0 by Adnana Dragut, Raquel Lacuesta, F. Xavier Gaya-Morey, Jose M. Buades-Rubio (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20200.txt.
Introduction
Adnana Dragut and three co-workers test an emotion recognition module in a proactive chat agent. The system reads users' feelings in real time from faces and words. In a study with 20 users, visual cues diverged sharply from actual feelings: users kept a poker face, serious and focused expressions even when feeling positive. Word analysis therefore proved more reliable, by placing verbal expressions in context. The agent could also stir emotions by adapting themes and using empathetic or humorous language, though uncalibrated proactivity sometimes caused disengagement and artificiality.
Catalog record
| CREATOR | Adnana Dragut, Raquel Lacuesta, F. Xavier Gaya-Morey, Jose M. Buades-Rubio |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55727 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating multimodal emotion recognition in proactive conversational agents: A user study Publication type: ORIGINAL ARTICLE Journal: arXiv Volume: 0 Adnana Dragut, Raquel Lacuesta, F. Xavier Gaya-Morey, Jose M. Buades-Rubio Abstract This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence. The system evaluates real-time affective states through two distinct channels: a computer vision-based facial recognition module and a semantic linguistic analysis engine. To validate the framework, an empirical study was conducted with 20 users who engaged in dynamic, unscripted dialogues with the conversational agent. The findings reveal a significant discrepancy between automated visual cues and actual internal emotional states. When interacting with the AI, users consistently exhibited a “poker face” effect, displaying serious, concentrated facial expressions even when experiencing positive emotions. Consequently, the generative AI linguistic analysis proved significantly more reliable, by contextualizing the users’ verbal expressions. Furthermore, an analysis of the interaction dynamics demonstrated that SIAs can effectively elicit specific emotions by adapting conversational themes and employing structured linguistic patterns, such as empathetic or humorous language. However, the study also noted that instances of uncalibrated proactivity occasionally led to user disengagement and a perception of artificiality. Ultimately, this research highlights the necessity of refining SIAs to dynamically adapt to users’ emotional evolution, relying on deep linguistic context to foster more natural, human-like interactions. \transabstract [transABSTRACT] keywords facial recognition | linguistic analysis | socially interactive agents | affective computing \startpage 1 \articledoi \transtitle \subtranstitle [https://orcid.org/0000-0002-4773-4904] [https://orcid.org/0000-0003-1231-7235] [https://orcid.org/0000-0002-6137-9558] \corres F. Xavier Gaya-Morey () \transkeywords 1 Introduction Understanding and responding to emotions is a fundamental component of human communication. For Socially Interactive Agents (SIAs) to establish meaningful connections with users, they must be capable of recognizing and adapting to the user’s emotional state in real time 15 . Integrating this affective awareness is essential for humanizing conversational agents, fostering natural dialogues, and ultimately improving their overall user acceptance 17 . Currently, emotion detection systems rely on a variety of modalities, ranging from facial expression recognition and voice processing to gaze tracking and the analysis of biological signals, such as electroencephalograms (EEG) and electrocardiograms (ECG) 46 , 18 . Recent research has demonstrated significant progress in multimodal emotion recognition applied to social agents. For example, 34 proposed a novel cross-representation model combining speech and text that effectively outperforms traditional speech-only or text-only unimodal approaches 8 . Similarly, 27 evaluated the fusion of physiological signals with other modalities to estimate different types of sentiment during naturalistic human-agent interactions. Despite these technical advancements, most existing studies rely on pre-recorded datasets, theoretical simulations, or passive observation. There is a notable lack of research integrating multimodal emotion detection within dynamic, real-time conversations driven by generative AI. Specifically, current literature does not sufficiently address how proactive, AI-generated dialogues attempt to connect with specific user emotions, nor how these interactions directly influence the user’s facial expressions and internal affective states during a live exchange. To bridge this gap, this study presents the design and empirical evaluation of a proactive conversational SIA. The proposed system integrates two core emotion detection techniques: a computer vision module for facial expression recognition and a generative AI-based linguistic analysis module. Through practical, real-time evaluations with users interacting with the SIA, this research investigates how human emotions actually manifest during unscripted interactions. By comparing automated visual data, semantic linguistic analysis, and subjective user self-reports, this study highlights critical interaction patterns and evaluates the real-world accuracy of these emotion detection methods. The remainder of this article is organized as follows: Section 2 reviews related work in emotion recognition and SIAs. Section 3 outlines the research objectives and specific research questions. Section 4 details the methodology, including the system design and evaluation procedure. Section 5 presents the objective results of the multimodal evaluation. Section 6 discusses these findings and answers the core research questions. Section 7 acknowledges the study’s limitations and outlines future work, and Section 8 provides the final conclusions. 2 Related Work The success of interactions between humans and artificial agents relies heavily on emotional identity and the understanding of cultural affect. These elements allow agents to maintain emotional coherence, transforming their perception from mere machines into entities capable of establishing meaningful connections 23 , 35 . As argued by 25 , the ability of agents to engage in emotional contagion, the transmission of affective states between participants, is vital for enhancing trust, empathy, and prosocial orientation during social exchanges. 2.1 Emotional Intelligence in Socially Interactive Agents SIAs are increasingly integrated into diverse domains, necessitating the incorporation of emotional intelligence to improve human-machine dynamics. Emotions are not merely aesthetic; they are essential for creating realistic behaviors in social simulations by integrating cognition and social relationships. For instance, 1 focused on simulating emotions to foster empathy, thereby enhancing agent autonomy and adaptability. Similarly, 45 explored emotionally intelligent virtual agents through biologically inspired cognitive architectures to replicate human social behavior. Further research by 14 emphasizes the importance of recognizing human emotional states to improve bonding, proposing perception architectures for human-robot interaction. In parallel, 50 investigated multimodal deep neural networks to identify opportunities for empathetic responses. Despite these advances, some authors argue that a significant gap remains: most existing models focus on theoretical simulations and Affect Control Theory without addressing the challenges of real-time processing in dynamic interactions 35 , 23 . Many studies, such as those by 9 and 24 , primarily review how humans perceive emotions in agents but do not delve into how these perceptions can be processed and responded to by the agents themselves in live settings. 2.2 Facial Expression Recognition and its Limitations Facial Expression Recognition (FER) is a fundamental pillar for detecting user states. Early deep learning approaches, such as those discussed by 26 , highlighted the potential of analyzing expressions alongside speech and text. 52 demonstrated that bimodal approaches, combining facial features with speech, consistently outperform unimodal systems. Modern FER systems use sophisticated models like cGANs 12 , semantic-rich frameworks 7 , and efficient architectures like SwishNet 11 to facilitate natural interaction. However, the transition from laboratory settings to “in-the-wild” applications presents severe challenges. 42 and 10 point out that variations in lighting, facial angles, and cultural diversity require more robust models. While recent developments like LiteFer 56 and video representation learning 48 aim to improve efficiency on limited-resource devices, a deeper technical unreliability persists. 6 and 29 have criticized the inconsistency of automated FER when dealing with diverse observer samples. Crucially, 53 and 44 suggest that serious or neutral facial expressions, common in social interactions, often lead to unreliable detection. This “ambiguity of expression” suggests that relying solely on visual data may be insufficient for SIAs in real-world social contexts. 2.3 Text-Based Emotion Detection and Generative AI Text-Based Emotion Detection (TBED) has become a critical component for developing empathetic agents 29 , 30 . While it has proven valuable for big data analytics in social media 31 , applying TBED to real-time conversation introduces complexities such as handling short texts, synonyms, and reversed word order 36 , 54 . Traditional machine learning approaches have made significant progress 33 , yet full automation remains a challenge 36 . The emergence of Generative AI offers a paradigm shift. 4 proposed real-time