Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts
Aref Farhadipour, Hossein Ranjbar, Masoumeh Chapariniya, Teodora Vukovic, Sarah Ebling, Volker Dellwo · 2025
Read Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts by Aref Farhadipour, Hossein Ranjbar, Masoumeh Chapariniya, Teodora Vukovic, Sarah Ebling, Volker Dellwo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06805.txt.
Introduction
Aref Farhadipour, Hossein Ranjbar, and four colleagues combine voice, face, video, and text to read emotions in group conversations, in a paper published in two thousand twenty-five. Real talk among several people is messy, and any single cue can mislead an emotion judge.
Their system runs four channels side by side. A text model reads the words, a speech model listens to the audio, and two vision models watch faces and full video frames. The features from all four are merged into one combined vector. That vector predicts both the emotion and the sentiment of each moment. On a well-known dataset, the joint system beats any single channel alone. It reached about two-thirds accuracy on emotion labels and a higher mark on sentiment. The work shows the value of many cues at once for machines that must follow human conversation.
Catalog record
| CREATOR | Aref Farhadipour, Hossein Ranjbar, Masoumeh Chapariniya, Teodora Vukovic, Sarah Ebling, Volker Dellwo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts Aref Farhadipour, Hossein Ranjbar, Masoumeh Chapariniya, Teodora Vukovic, Sarah Ebling, Volker Dellwo Abstract Emotion recognition and sentiment analysis are pivotal tasks in speech and language processing, particularly in real-world scenarios involving multi-party, conversational data. This paper presents a multimodal approach to tackle these challenges on a well-known dataset. We propose a system that integrates four key modalities/channels using pre-trained models: RoBERTa for text, Wav2Vec2 for speech, a proposed FacialNet for facial expressions, and a CNN+Transformer architecture trained from scratch for video analysis. Feature embeddings from each modality are concatenated to form a multimodal vector, which is then used to predict emotion and sentiment labels. The multimodal system demonstrates superior performance compared to unimodal approaches, achieving an accuracy of 66.36% for emotion recognition and 72.15% for sentiment analysis. † † address: Department of Computational Linguistics, University of Zurich, Zurich, Switzerland 1 Introduction With the rapid growth of technology, the role of machines in various sectors, such as resource management, entertainment, and human assistance, has become increasingly prominent. In these applications, machines are typically provided with data from the real world, and they are expected to respond appropriately. The input data can be diverse, including text, audio, video, or multimodal information. Each type of input requires specific processing techniques, and each modality has distinct challenges. In multimodal data processing, selecting and implementing an effective method for each modality is critical to achieving optimal performance. One significant challenge that has gained considerable attention in recent years is enhancing the intelligence of Human-Machine Interaction (HMI). Users interacting with machines often wish to communicate their emotional states, making it imperative for the system to accurately perceive and respond to these emotions to facilitate a high level of intelligent interaction. Therefore, developing emotion and general sentiment recognition systems has become a crucial area of research. Based on the theory of emotions, the emotion recognition task identifies the inner sense based on the seven basic emotions. However, sentiment analysis deals with labelling the general opinion expressed as positive, negative, or neutral, for example, in customer feedback analysis. Emotion recognition in HMI involves multiple modalities, such as text, facial expressions, voice, and body movements, representing different aspects of human emotions. These modalities complement each other; for example, voice tone, facial expressions, and gestures can collectively modify or amplify the emotional content of spoken words. However, designing multimodal emotion recognition and sentiment analysis systems poses challenges due to the scarcity of large, high-quality multimodal datasets. Most of the current research is based on the IEMOCAP dataset [ 1 ] , which is recorded under controlled studio conditions. While this dataset provides a solid foundation, real-world emotion recognition presents more significant challenges, especially in multi-party conversational settings. For this reason, we employ the MELD dataset [ 2 ] in this work, as it offers a more realistic setting with varied background conditions, multiple speakers, and environmental noise, providing a more challenging yet practical scenario for emotion recognition. In this work, we propose a multimodal system designed for emotion recognition and sentiment analysis. This system is built upon four core components, each addressing a distinct modality. Specifically, we utilize Wav2Vec2 [ 3 ] for voice, RoBERTa [ 4 ] for text, a proposed FacialNet for facial expressions, and a CNN+Transformer architecture trained from scratch for video analysis. The remainder of this paper is organized as follows: Section 2 reviews related work in emotion recognition and sentiment analysis. Section 3 describes the dataset and the proposed system architecture. Experimental results are presented in Section 4 , followed by conclusions in Section 5 . 2 Previous Work While some approaches focus on unimodal emotion recognition [ 5 ] , recent advancements have favoured the integration of multiple data modalities and machine learning techniques. These multimodal methods aim to address the limitations of unimodal analysis by combining information from diverse sources to enhance emotion recognition performance. Zheng et al. [ 6 ] proposed a two-stage framework focusing on facial sequences. Their approach employs multi-task learning to generate emotion-informed visual representations. Similarly, Hu et al. [ 7 ] developed the supervised adversarial contrastive learning framework, which incorporates contextual adversarial training alongside a Dual-LSTM structure to enhance emotion recognition from textual data. Yun et al. [ 8 ] introduced the Teacher-leading Multimodal Fusion Network for emotion recognition, which employs cross-modal knowledge distillation. In this method, a language model is used to improve the performance of non-verbal modalities. Fu et al. [ 9 ] presented a bi-model system for emotion that uses prompts to generate commonsense knowledge for interlocutors based on historical utterances. This model incorporates an interlocutor commonsense identification task to fine-tune the system for identifying implicit speaker cues. Chudasama et al. [ 10 ] introduced a multimodal fusion network, which extracts emotion-relevant features from visual, audio, and textual modalities. Their model uses a multi-head attention-based fusion mechanism and introduces a novel feature extractor trained with adaptive margin-based triplet loss for the audio and visual modalities. In another novel approach, Wu et al. [ 11 ] developed a system that applies large language models to speech emotion recognition. This method converts speech characteristics into natural language descriptions and incorporates them into text prompts, allowing for multimodal emotion analysis without modifying the model’s architecture. Multimodal sentiment analysis in conversations has also gained traction as researchers seek to overcome the limitations of unimodal approaches. Poria et al. [ 2 ] compared various models, including Text-CNN, LSTM, and DialogueRNN, finding that multimodal fusion outperformed unimodal methods, with DialogueRNN excelling in multi-party situations. Shah et al. [ 12 ] took an ensemble learning approach to sentiment analysis, integrating RoBERTa for text and Wav2Vec2 for audio. Their study explored various ensemble techniques to improve sentiment classification accuracy through multimodal fusion. 3 Methodology In the proposed system, we used several pre-trained models. The use of pre-trained models is driven by the fact that available multimodal emotion recognition and sentiment analysis datasets are often limited in size, making transfer learning an effective strategy to overcome this data scarcity challenge. Various fusion techniques are commonly employed in multimodal systems, including sensory, feature, and score fusion and based on prior work, feature fusion has consistently demonstrated superior results [ 13 ] . Figure 1: Proposed system with unimodal and multimodal feature fusion strategies. Figure 1 depicts the proposed system’s architecture for emotion recognition and sentiment analysis tasks. Two approaches are explored: unimodal and multimodal fusion strategies. In the multimodal mode, the feature vectors extracted from each system are concatenated to make a multimodal vector. For the voice modality, Wav2Vec2 is applied in an audio classification framework. The model is initially trained on three emotion recognition datasets and later fine-tuned on the MELD dataset. A similar approach is adopted for RoBERTa, pre-trained on 58 million tweets for emotion recognition using the TweetEval benchmark before fine-tuning on MELD [ 14 ] . The combined representation is then used to make final decisions regarding emotion and sentiment via an additional classifier. The multimodal feature vector is classified using a multi-layer perceptron. The rest of this section outlines the materials and methods used in this study. We introduce the MELD dataset and briefly overview the four models employed for