GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations

Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller · 2025

Read GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations by Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20919.txt.

Introduction

Yupei Li and co-authors present a model that recognizes emotions in conversations by listening to voices and reading words together. Emotions shift as people interact, so judging each utterance alone misses the story. Their system considers the speech and transcript of both the speaker and their partners, hunting for the sentences that drive emotional turns.

A contrastive audio-language model aligns the sound and text features. A gating unit then emphasizes the utterances that matter most emotionally. A dialog-aware decoder refines the final labels using context. On a standard benchmark, the system reached the best scores among open methods for four-way emotion classification. The authors also analyze its decisions from a psychological angle.

Catalog record

CREATORYupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller Abstract Affective Computing (AC) is essential for advancing Artificial General Intelligence (AGI), with emotion recognition serving as a key component. However, human emotions are inherently dynamic, influenced not only by an individual’s expressions but also by interactions with others, and single-modality approaches often fail to capture their full dynamics. Multimodal Emotion Recognition (MER) leverages multiple signals but traditionally relies on utterance-level analysis, overlooking the dynamic nature of emotions in conversations. Emotion Recognition in Conversation (ERC) addresses this limitation, yet existing methods struggle to align multimodal features and explain why emotions evolve within dialogues. To bridge this gap, we propose GatedxLSTM, a novel speech-text multimodal ERC model that explicitly considers voice and transcripts of both the speaker and their conversational partner(s) to identify the most influential sentences driving emotional shifts. By integrating Contrastive Language-Audio Pretraining (CLAP) for improved cross-modal alignment and employing a gating mechanism to emphasise emotionally impactful utterances, GatedxLSTM enhances both interpretability and performance. Additionally, the Dialogical Emotion Decoder (DED) refines emotion predictions by modelling contextual dependencies. Experiments on the IEMOCAP dataset demonstrate that GatedxLSTM achieves state-of-the-art (SOTA) performance among open-source methods in four-class emotion classification. These results validate its effectiveness for ERC applications and provide an interpretability analysis from a psychological perspective. I Introduction Artificial General Intelligence (AGI) represents a key future direction in AI development, with Affective Computing (AC) playing a crucial role in enhancing AGI’s ability to interact effectively with humans. Sun et al. [ 1 ] advocate this perspective, emphasising that emotional recognition is a fundamental and essential technique within AC. Emotional recognition has been extensively explored in the literature. For instance, Venkataramanan et al. [ 2 ] examine various audio features to predict emotions, while text-based emotion recognition has also been developed [ 3 ] . However, relying on a single modality is insufficient for capturing emotions, as they are expressed through audio, text, and visual cues from different perspectives [ 4 ] . Consequently, Multimodal Emotion Recognition (MER) has gained attention for its ability to incorporate diverse emotional expressions, for example audio and text, in order to mitigate misinterpretation. MER encompasses various model types, which differ based on the granularity of the corpus. Word-level MER models, for instance, focus on short phrases. Local attention mechanisms have been applied to frame-level acoustic features in Recurrent Neural Networks (RNNs) [ 5 ] , and word clustering models have also been utilised [ 6 ] . These approaches often enhance recognition speed by prioritising key emotional words, which is particularly advantageous for real-time MER systems requiring immediate responses. However, such models may sacrifice a comprehensive understanding of contextual or situational factors. Utterance-level models address this limitation by leveraging entire sentences for emotion prediction [ 7 ] . Nevertheless, human emotions are inherently dynamic, exhibiting momentary and situational variations [ 8 ] . Relying solely on utterance-level representations may result in the loss of critical contextual information, thereby failing to capture the evolving nature of emotions. To mitigate this issue, Emotion Recognition in Conversation (ERC) has been explored through AC models [ 9 ] , which incorporate broader contextual background information alongside the current utterance, thereby enhancing the accuracy of emotion recognition. Despite advancements in MER within ERC, several challenges remain. First, multimodal fusion methodologies have not been universally effective. Extensive research has explored early, late, and model fusion strategies; however, the integration of features across modalities remains suboptimal, particularly due to alignment issues [ 10 ] . Although cross-attention mechanisms [ 11 ] have been employed for alignment, they primarily adapt one modality to another rather than achieving comprehensive fusion. Additionally, the incorporation of contextual information in conversations remains limited. Many approaches simply concatenate features from previous utterances without explicitly determining their contributions to emotion recognition [ 12 ] . Furthermore, ERC is inherently a sequential classification problem, typically addressed using time-series models. While Long Short-Term Memory (LSTM) and RNN architectures [ 13 ] are commonly employed, they struggle to retain information over long sequences, as their processing tends to prioritise recent states. Transformer-based architectures [ 14 ] mitigate this limitation by leveraging self-attention mechanisms, yet, they do not inherently emphasise the sequential order of data to the same extent as traditional time-series models. To address these challenges, Contrastive Language-Audio Pretraining (CLAP) has been introduced for fusing text and audio modalities by utilising a shared embedding space, which we adopt in this study [ 15 ] . Additionally, xLSTM has been proposed as a hybrid approach that integrates the advantages of both LSTMs and Transformers; however, it has not yet been applied to this domain [ 16 ] . Although large language models (LLMs) have been explored for ERC [ 17 ] , their computational cost remains prohibitively high. Consequently, this study focuses on deep learning models that offer a more practical balance between performance and efficiency. Furthermore, to address the issue of contextual dependency, we propose a novel gate-based architecture that explicitly determines the contributions of different utterances within a conversation. This mechanism enhances interpretability and provides structured guidance for the model to refine its focus. Additionally, we find the Dialogical Emotion Decoder (DED) [ 18 ] to be a useful approach for addressing this challenge. Our contributions are as follows: • Utilise an xLSTM-based model in the ERC task firstly to the best of our knowledge. Propose a model-level fusion GatedxLSTM for ERC within AC, demonstrating state-of-the-art (SOTA) performance among open-source models for four-class emotion classification on the IEMOCAP dataset. • Enhance the interpretability of the ERC task by analysing the importance of the current speaker and the interlocutor in emotion prediction. • Outline future research directions for ERC, highlighting potential advancements and challenges in the field. II Related Work Fig. 1: The pipeline of GatedxLSTM. First, each audio sample and its corresponding transcription are processed using the CLAP model, which maps them into a shared embedding space and aligns their embeddings. Next, for a given utterance at time T , we identify its preceding utterance spoken by the interlocutor. Both the audio and text representations are then passed through four distinct xLSTM blocks to extract features. To incorporate contextual information, we retrieve relevant features from several preceding utterances (e. g., at T-1 ). These extracted features are then processed through a gating mechanism to determine their contribution to the final emotion recognition task, assigning the audio of the current utterance a weight of 1. After applying the gating mechanism, the features are passed through a fully connected layer. Finally, the prediction is refined using DED in the last stage of the pipeline. II-A Emotion Recognition in Conversation (ERC) Research shows that emotions can be conveyed through multiple channels, including vocal prosody, linguistic content, visual expressions, and physiological signals [ 19 , 20 ] . MER integrates information from different modalities to improve recognition accuracy. Compared to single-modal approaches, multimodal interaction captures emotional characteristics more comprehensively and enhances model robustness [ 10 ] . Existing studies adopt different modality combinations to optimise feature representation. Wang et al. [ 21 ] developed the MED4 multimodal database integrating electroencephalogram (EEG), speech, video,

Read the complete text