Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition
Yuntao Shou [email protected], Jun Zhou [email protected], Tao Meng [email protected], Wei Ai [email protected], Keqin Li [email protected] · 2026
Read Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition by Yuntao Shou [email protected], Jun Zhou [email protected], Tao Meng [email protected], Wei Ai [email protected], Keqin Li [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26840.txt.
Introduction
Shou and co-authors tackle emotion recognition in conversations that shift across settings and speakers. Models trained in one scenario often fail in another, and noisy labels add a second problem. Their DGDA framework builds a graph of emotional links between utterances. A dual-branch encoder captures both local groups and global paths. A domain discriminator pushes the features toward setting-invariant form. On IEMOCAP and MELD, the method beat strong baselines under cross-scenario tests.
Catalog record
| CREATOR | Yuntao Shou [email protected], Jun Zhou [email protected], Tao Meng [email protected], Wei Ai [email protected], Keqin Li [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78107 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition Yuntao Shou [email protected], Jun Zhou [email protected], Tao Meng [email protected], Wei Ai [email protected], Keqin Li [email protected] Abstract Multimodal Emotion Recognition in Conversations (MERC) aims to predict speakers’ emotional states in multi-turn dialogues through text, audio, and visual cues. In real-world settings, conversation scenarios differ significantly in speakers, topics, styles, and noise levels. Existing MERC methods generally neglect these cross-scenario variations, limiting their ability to transfer models trained on a source domain to unseen target domains. To address this issue, we propose a Dual-branch Graph Domain Adaptation framework (DGDA) for multimodal emotion recognition under cross-scenario conditions. We first construct an emotion interaction graph to characterize complex emotional dependencies among utterances. A dual-branch encoder, consisting of a hypergraph neural network (HGNN) and a path neural network (PathNN), is then designed to explicitly model multivariate relationships and implicitly capture global dependencies. To enable out-of-domain generalization, a domain adversarial discriminator is introduced to learn invariant representations across domains. Furthermore, a regularization loss is incorporated to suppress the negative influence of noisy labels. To the best of our knowledge, DGDA is the first MERC framework that jointly addresses domain shift and label noise. Theoretical analysis provides tighter generalization bounds, and extensive experiments on IEMOCAP and MELD demonstrate that DGDA consistently outperforms strong baselines and better adapts to cross-scenario conversations. Our code is available at https://github.com/Xudmm1239439/DGDA-Net . 1 Introduction Multimodal emotion recognition in conversations (MERC) Shou et al. 2026b ; Shou et al. 2022 ; Ai et al. 2025 ; Shou et al. 2024c ; Shou et al. 2024b ; Meng et al. 2024b ; Shou et al. 2025a aims to predict the emotional state of participants in a multi-round conversation through multimodal information (e.g., text, audio, and video) and has broad application prospects in dialogue generation Ghosh et al. 2017 ; Tu et al. 2024 ; Zhang et al. 2024a ; Tellamekala et al. 2023 ; Wen et al. 2024 ; Meng et al. 2024a ; Shou et al. 2026a ; Shou et al. 2024a , social media analysis Khare & Bajaj 2020 ; Peng et al. 2024 ; Xu et al. 2025 ; Huang et al. 2020 , intelligent systems like smart homes and chatbots Young et al. 2018 ; Lu et al. 2025 ; Kang & Cho 2025 ; Li et al. 2025 ; Shou et al. 2025b ; Ai et al. 2026 . Figure 1 : (a) Common MERC methods. A well-crafted encoder architecture is used to achieve multimodal emotion recognition without considering out-of-domain distribution differences. (b) Our Proposed Dual-branch Graph Domain Adaptation (DGDA) method. DGDA exploits a dual-branch encoder to explicitly and implicitly extract multimodal features, and constructs a domain adversarial alignment strategy and regularization loss to achieve out-of-domain distribution data generalization and resistance to noise label interference. In the MERC task, researchers mainly focus on learning the emotional feature representation of in-domain data. As shown in Fig. 1 (a), a well-crafted encoder-classifier architecture is used to achieve multimodal emotion recognition without considering out-of-domain distribution differences. The mainstream MERC method mainly uses Transformers Hazmoune & Bougamouza 2024 ; Lian et al. 2021 ; Ma et al. 2023 ; Shou et al. 2025d ; Shou et al. 2025g and graph neural networks (GNN) Yang et al. 2024 ; Chen et al. 2023 ; Shou et al. 2025e ; Shou et al. 2025i ; Shou et al. 2023 ; Shou et al. 2025c as the encoder to model contextual dependency information and speaker dependency. Although existing methods have achieved relatively good emotion recognition results, they ignore the impact of cross-domain distribution differences on emotion recognition performance. In other words, there may be significant differences in language styles, emotional expressions, and contextual environments in different domains, and these differences may lead to the limited generalization capabilities of the model. Furthermore, if some samples in the dataset are incorrectly labeled, e.g., the emotion of anger is incorrectly labeled as neutral, then the model may learn these wrong patterns during training. As a result, the model may misclassify the angry emotion as neutral in practical applications, thus affecting the accuracy and reliability of emotion recognition Lian et al. 2023 ; Wagner et al. 2023 ; Shou et al. 2025f . To address the above problems, we propose a Dual-branch Graph Domain Adaptation (DGDA) for multi-modal emotion recognition in cross-scenario conversations, as shown in Fig. 1 (b). Specifically, to capture the discriminative features of emotion in multimodal utterances, we first construct an emotion interaction graph to model the complex emotional dependencies between utterances. Then, we design a hypergraph aggregation and path aggregation dual-branch graph encoder to explicitly and implicitly capture the dynamic changes in emotion between utterances and explore multivariate relationships, respectively. To address the problem of out-of-domain distribution differences, we introduce a domain adversarial classifier to improve the representation ability of invariant features in the source domain. In addition, we construct a regularization loss to prevent the model from memorizing noise and improve the model’s ability to resist interference from noisy labels in source domains. Extensive experiments and evaluations demonstrate DGDA’s superiority. The main contributions of this paper are summarized as follows: • To the best of our knowledge, we make the first attempt to simultaneously mitigate domain shift and noisy label interference problems in MERC scenarios, thereby enhancing usability in real-world scenarios. • We improve the expressiveness of domain-invariant features of the original graph by introducing a domain adversarial classifier and solving the problem of out-of-domain distribution differences. • We added a regularization constraint loss on the basis of the cross-entropy loss term to effectively suppress the model’s over-learning of noisy labels and encourage the model to pay more attention to the real signals in the data. • We provide theoretical proof to ensure that the designed DGDA is more precisely tailored for cross-scenario conversations. Extensive experiments conducted on the IEMOCAP and MELD datasets showed that DGDA is significantly better than existing baseline methods. 2 Related work 2.1 Multimodal Emotion Recognition in Conversations Multimodal emotion recognition in conversations (MERC) has emerged as a key research area in artificial intelligence, especially at the intersection of natural language processing (NLP), computer vision (CV), and speech processing Li et al. 2024b ; Liu et al. 2024a ; Tao et al. 2025 ; Qin et al. 2025 ; Shou et al. 2025c ; Shou et al. 2025h . Its objective is to infer human emotional states by jointly analyzing textual content, acoustic cues, and visual expressions Sun et al. 2024 ; Chen et al. 2024 ; Guo et al. 2025 ; Tang et al. 2025 . RNN-based MERC methods primarily focus on extracting contextual semantic information by modeling sequential dependencies within multimodal inputs through recurrent memory units Majumder et al. 2019 ; Huddar et al. 2021 ; Ho et al. 2020 . Transformer-based methods leverage self-attention and multi-head attention mechanisms, often combined with pretrained language models, to capture long-range dependencies in conversations and achieve more effective multimodal fusion Zhao et al. 2023 ; Ma et al. 2023 . Meanwhile, GCN-based approaches utilize the structural flexibility of graph convolutional networks to model inter-utterance relations, multimodal associations, and latent interaction patterns within dialogues Ren et al. 2021 ; Yuan et al. 2023 ;