Modelling the Interplay of Eye-Tracking Temporal Dynamics and Personality for Emotion Detection in Face-to-Face Settings
Meisam J. Seikavandi, Jostein Fimland, Fabricio Batista Narcizo, Maria Barrett, Ted Vucurevich, Jesper Bünsow Boldt, Andrew Burke Dittberner, Paolo Burelli · 2025
Read Modelling the Interplay of Eye-Tracking Temporal Dynamics and Personality for Emotion Detection in Face-to-Face Settings by Meisam J. Seikavandi, Jostein Fimland, Fabricio Batista Narcizo, Maria Barrett, Ted Vucurevich, Jesper Bünsow Boldt, Andrew Burke Dittberner, Paolo Burelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24720.txt.
Introduction
Meisam J Seikavandi and seven co-authors build a way to read feelings from gaze in face to face scenes. Their frame joins eye track runs with Big Five trait scores and cues from the clip shown. The goal is to guess both the feeling a viewer saw and the one they felt.
Seventy three adults watched speech clips while their gaze was logged, then gave trait and feeling scores. Nets that used the full gaze run plus traits and clip cues beat old base lines. Clip cues helped most for seen feelings, while traits helped most for felt ones. The split shows why feeling reads need more than the face alone.
Catalog record
| CREATOR | Meisam J. Seikavandi, Jostein Fimland, Fabricio Batista Narcizo, Maria Barrett, Ted Vucurevich, Jesper Bünsow Boldt, Andrew Burke Dittberner, Paolo Burelli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24720
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modelling the Interplay of Eye-Tracking Temporal Dynamics and Personality for Emotion Detection in Face-to-Face Settings Meisam J. Seikavandi, Jostein Fimland, Fabricio Batista Narcizo, Maria Barrett, Ted Vucurevich, Jesper Bünsow Boldt, Andrew Burke Dittberner, Paolo Burelli Abstract Accurate recognition of human emotions is critical for adaptive human–computer interaction, yet remains challenging in dynamic, conversation-like settings. This work presents a personality-aware multimodal framework that integrates eye-tracking sequences , Big Five personality traits , and contextual stimulus cues to predict both perceived and felt emotions. Seventy-three participants viewed speech-containing clips from the CREMA-D dataset while providing eye-tracking signals, personality assessments, and emotion ratings. Our neural models captured temporal gaze dynamics and fused them with trait and stimulus information, yielding consistent gains over SVM and literature baselines. Results show that (i) stimulus cues strongly enhance perceived-emotion predictions (macro F1 up to 0.77 ), while (ii) personality traits provide the largest improvements for felt emotion recognition (macro F1 up to 0.58 ). These findings highlight the benefit of combining physiological, trait-level, and contextual information to address the inherent subjectivity of emotion. By distinguishing between perceived and felt responses, our approach advances multimodal affective computing and points toward more personalized and ecologically valid emotion-aware systems. 1 Introduction Emotion recognition is a central challenge in affective computing, with wide-ranging applications in areas such as adaptive human–computer interaction, virtual agents, education, and teleconferencing. Accurate recognition enables systems to interpret and respond to human states in a more personalized and context-sensitive manner. Yet, despite decades of progress, most computational models still rely on simplified representations—for example, static snapshots of facial expressions or coarse emotion labels—that fall short of capturing the richness of real human interactions. Genuine emotion perception unfolds dynamically, is shaped by subtle attentional cues, and is further modulated by stable individual traits such as personality Hughes et al. 2020 ; Kaspar and König 2012 ; Rauthmann et al. 2012 . At the theoretical level, these complexities lie at the intersection of two perspectives. Basic Emotion Theory (BET) Ekman 1992 views emotions as discrete, biologically hard-wired categories, while constructionist accounts such as the Theory of Constructed Emotion (TCE) Barrett 2017 emphasize that emotions emerge from interactions between core affect and conceptual knowledge. A layered view suggests that both levels matter: an observer can recognize expressed cues in another person (BET), yet their own internal affective experience may be constructed differently (TCE). This distinction is particularly salient in what we term the listener’s perspective . In everyday scenarios—watching a speaker in a meeting, following a video call, or interacting with a virtual agent—people interpret a talking face without necessarily engaging in full turn-taking dialogue. Such contexts approximate a dialogue-like setting while retaining one-sided attentional dynamics. Our study focuses on this listener scenario , where participants watched speech-containing video clips from the CREMA-D dataset Cao et al. 2014 , a setup that allows us to probe how perceived and felt emotions may converge or diverge. Key challenges include neglected temporal dynamics, under-modeled individual differences, and divergence between perceived and felt emotions Barrett 2017 . To systematically address these issues, we adopt a framework distinguishing Expressed Emotions ( E_{e} ), Perceived Emotions ( E_{p} ), and Felt Emotions ( E_{f} ). Encountering emotional stimuli involves recognizing E_{e} , forming a subjective perception E_{p} , and potentially experiencing a distinct internal state E_{f} . Modeling both E_{p} and E_{f} in parallel is therefore essential for realistic emotion recognition. In this paper, we introduce a multimodal approach that integrates eye-tracking data , temporal modeling , and personality traits to predict E_{p} and E_{f} in a dynamic, speech-based setting. Seventy-three non-actor participants contributed (1) detailed eye movements (fixations, pupil size), (2) self-reported Big Five personality profiles John et al. 1991 ; Fossati et al. 2011 , and (3) perceived and felt emotional responses on each trial. Our contributions are threefold: 1. Integration of multimodal data: Demonstrating that combining eye-tracking signals, personality traits, and temporal dynamics significantly improves emotion recognition in talking-face scenarios. 2. Insights into personality’s role: Providing empirical evidence that personality traits modulate both how participants perceive others’ emotions and how they feel in response. 3. Advancement in modeling: Proposing a multimodal neural architecture that achieves high predictive performance for both perceived and felt emotions, with direct relevance for adaptive, user-centered affective computing. By jointly modeling dynamic cues, stable personality traits, and the divergence between perceived and felt states, our work moves beyond static or purely acted benchmarks. Although the study does not yet capture fully interactive dialogues, it reflects a critical real-world interaction pattern: emotional decoding by the listener. This perspective, we argue, is essential for developing next-generation emotion-recognition systems that respect individual differences and capture the subtle, layered nature of human emotions. 2 Background A wide range of studies has explored different modalities and methodologies, often achieving impressive accuracy levels. However, many of these works simplify emotion detection by focusing on limited arousal and valence scores or by relying on highly controlled datasets. In practice, real-world emotion perception is shaped by multiple, often interdependent factors such as personality traits, gaze patterns, and contextual cues. Consequently, several important challenges remain insufficiently addressed, especially regarding the use of non-actor participants, individual differences, and temporal dynamics. These unresolved issues echo long-standing theoretical debates, e.g. whether emotions are discrete biological kinds or context-dependent constructions, highlighting the need for frameworks that link physiology, attention, and conceptual labeling in a single pipeline. 2.1 Emotion Models Two primary frameworks characterize how emotions are commonly modeled. Discrete models rooted in Basic Emotion Theory (BET) Ekman 1992 group emotions into basic categories such as anger, disgust, fear, joy, sadness, and surprise. These labels are intuitive and map neatly onto facial action patterns, but they often struggle to capture subtle or mixed states in realistic settings Siegert et al. 2011 . Dimensional models , by contrast, represent affect along continuous axes such as arousal and valence Mehrabian 1996 ; Russell 1980 , occasionally adding dominance as a third dimension. They allow nuanced representations, yet many engineering studies discretize them into low/medium/high bins, obscuring fine-grained shifts. Trade-offs and hybrid views. Discrete categories can over-simplify overlapping expressions, whereas purely dimensional schemes cannot easily separate qualitatively distinct emotions that share similar core affect (e.g. anger vs. fear). Recent evidence shows that subjective reports cluster into at least 27 categories connected by smooth gradients Cowen and Keltner 2017 , motivating hybrid pipelines that first estimate core affect and then map it onto discrete concepts, an approach compatible with Theory of Constructed Emotion (TCE) Barrett 2017 . Following Van Heijst et al. Van Heijst et al. 2025 , we view BET and TCE as complementary layers: BET explains why evolution endowed us with affect programs; TCE explains how any given episode is constructed from core affect plus conceptual knowledge. This layered stance underpins our decision to model both continuous ( E_{f} ) and categorical ( E_{p} ) labels in later sections. 2.2 Multimodal Emotion Recognition Approaches Leveraging multiple modalities facial expressions, vocal prosody, and physiological signals has yielded robust gains. For example, Kollias et al. Kollias et al. 2022 used multi-task audio-visual learning to detect valence, arousal, expressions, and action units. Reviews by Li et al. Li and Chen 2024 and Zhang et al. Zhang and