Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations
Tao Meng, Weilun Tang, Yuntao Shou, Yilong Tan, Jun Zhou, Wei Ai, Keqin Li · 2026
Read Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations by Tao Meng, Weilun Tang, Yuntao Shou, Yilong Tan, Jun Zhou, Wei Ai, Keqin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22345.txt.
Introduction
Meng and co-authors recognize emotions in conversations from text, audio, and images at once. Graph networks help by modeling how speakers depend on each other. Most methods fuse the modalities with fixed settings. That forces one balance across all emotion types. Their model, DF-GCN, lets the fusion shift for each utterance. Ordinary differential equations inside the graph net capture how emotional links evolve. Prompts from a global utterance vector guide the fusion. Tests on two public datasets show gains over prior methods.
Catalog record
| CREATOR | Tao Meng, Weilun Tang, Yuntao Shou, Yilong Tan, Jun Zhou, Wei Ai, Keqin Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68267 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations Tao Meng, Weilun Tang, Yuntao Shou, Yilong Tan, Jun Zhou, Wei Ai, Keqin Li Abstract Multimodal emotion recognition in conversations (MERC) aims to identify and understand the emotions expressed by speakers during utterance interaction from multiple modalities (e.g., text, audio, images, etc.). Existing studies have shown that GCN can improve the performance of MERC by modeling dependencies between speakers. However, existing methods usually use fixed parameters to process multimodal features for different emotion types, ignoring the dynamics of fusion between different modalities, which forces the model to balance performance between multiple emotion categories, thus limiting the model’s performance on some specific emotions. To this end, we propose a dynamic fusion-aware graph convolutional neural network (DF-GCN) for robust recognition of multimodal emotion features in conversations. Specifically, DF-GCN integrates ordinary differential equations into graph convolutional networks (GCNs) to capture the dynamic nature of emotional dependencies within utterance interaction networks and leverages the prompts generated by the global information vector (GIV) of the utterance to guide the dynamic fusion of multimodal features. This allows our model to dynamically change parameters when processing each utterance feature, so that different network parameters can be equipped for different emotion categories in the inference stage, thereby achieving more flexible emotion classification and enhancing the generalization ability of the model. Comprehensive experiments conducted on two public multimodal conversational datasets confirm that the proposed DF-GCN model delivers superior performance, benefiting significantly from the dynamic fusion mechanism introduced. To the best of our knowledge, this is the first framework that adaptively assigns different fusion weights to different emotion categories during inference, leading to more effective multimodal information integration. Our code is available at https://github.com/yuntaoshou/DFGCN . keywords ,Multimodal Emotion Recognition ,Dynamic Fusion ,Graph Convolutional Neural Network ,Prompt Learning † † corresponding: Corresponding author: [email protected] 1 INTRODUCTION Multimodal emotion recognition in conversations (MERC) aims to exploit multimodal information (e.g., text, audio, video, etc.) in the dialogues to understand and identify the emotional state of the speaker [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . MERC has become an important research topic that has attracted much attention due to its wide application in opinion mining [ 8 , 9 , 10 , 11 ] , healthcare [ 12 , 13 , 14 ] , psychological counseling [ 15 , 16 , 17 ] , and building empathetic dialogue systems. MERC research not only helps us to deeply understand the emotional changes of users in multi-round dialogues but also improves the intelligence of dialogue systems, making them more capable of emotional perception and response. Unlike traditional non-conversational or unimodal emotion recognition (ER) methods, MERC focuses more on the fusion of semantic features from conversational context and multiple modalities [ 18 , 19 , 20 ] . The core challenge is how to effectively model and fuse the emotional dependencies between the context and multimodality in the conversation [ 21 , 22 , 23 ] . Specifically, MERC must fully capture the subtle changes in emotions in the conversation, understand the interaction patterns between multimodality, and reasonably fuse features to achieve more accurate emotion recognition [ 24 , 25 , 26 ] . Figure 1: Illustration of the difference between traditional static fusion and dynamic fusion processes. (a) Previous studies usually use fixed network parameters to fuse multimodal features. (b) Our proposed method generates modal parameters through GIV, thereby achieving dynamic multimodal feature fusion. The mainstream MERC methods mainly use a Transformer or GCN to model the emotional dependencies. and fuse the semantic features in the conversation [ 27 , 28 , 29 , 30 ] . For example, the Transformer-based self-distillation model (SDT) [ 1 ] adaptively learns weights related to different modalities through self-distillation technology and a hierarchical gated fusion strategy to perform feature fusion better. TBJE [ 31 ] leverages the Transformer architecture and a modular collaborative attention mechanism to effectively capture and integrate the key factors of emotion and sentiment expressed through multimodal language. DER-GCN [ 32 ] constructs multiple emotional relationship graphs through events to capture the emotional dependencies between multimodalities in dialogue, and uses self-supervised learning and multi-information Transformer to optimize feature fusion. M3Net [ 33 ] leverages multi-frequency information within a graph to enhance the representation of multivariate relationships and capture complex relationships across modalities and context. The above methods have achieved certain results in the MERC task through feature fusion. However, fixed parameters are often used in the inference stage to fuse the multimodal features across different emotion types. As shown in Figure 1 , although this unified parameter configuration simplifies the design of the model, the performance of the model on minority samples will drop significantly when faced with diverse emotion categories. In addition, due to the lack of specialized optimization for different emotion categories, the model has to balance performance between different categories, making it difficult to fully capture and represent the unique characteristics of each emotion. This not only affects the flexibility of the model but also limits its accuracy and sensitivity in certain emotion categories, making it difficult to achieve detailed distinction and recognition of complex emotions. Therefore, MERC needs to consider the dynamics of semantic dependencies in the conversation context and adaptively equip different network parameters for the fusion of utterance features of different emotion categories as the starting point of model design. A key aspect of capturing this context is modeling the sequential relationships between utterances. While pre-trained language models like RoBERTa provide powerful utterance-level semantics, they lack built-in mechanisms for inter-utterance context. To this end, we incorporate a Bidirectional GRU (Bi-GRU) to efficiently capture these contextual dependencies in the conversation flow. Inspired by the above analysis, we propose a DF-GCN for robust recognition of multimodal emotion features in conversations. The core of DF-GCN is the design of two serial graph convolution blocks, namely, the Static Graph Convolution (SGCODE) block and the Dynamic Graph Convolution (DGCODE) with ordinary differential equations block. Specifically, we first construct an emotional interaction graph based on multimodal utterance features and input it into SGCODE to capture the dynamic emotional dependencies between multimodal utterances. Then, the multimodal utterance features with emotional dependencies are input into the Transformer to obtain the GIV, and the multi-layer perceptron is used to generate prompts to obtain the dynamic weight of the utterance. Finally, the dynamic weight and multimodal utterance features are input into DGCODE for dynamic fusion to complete the emotional recognition of the utterance. It is worth noting that in the inference stage of the model, the parameters of the DGCODE we constructed are dynamically changed so that different network parameters can be equipped for utterances of different emotion categories in the inference stage, to achieve more flexible emotion classification and enhance the generalization ability of the model. The main contributions of this paper are summarized as follows: • We propose a DF-GCN for MERC. DF-GCN can adaptively equip utterances of different emotion categories with different network parameters to achieve more flexible emotion classification. • We construct a novel prompt generation network to obtain GIV and dynamic fusion weights with rich semantics to guide the network to dynamically fuse utterance