Bimodal Connection Attention Fusion for Speech Emotion Recognition
Jiachen Luo, Huy Phan, Lin Wang, Joshua D. Reiss* · 2025
Read Bimodal Connection Attention Fusion for Speech Emotion Recognition by Jiachen Luo, Huy Phan, Lin Wang, Joshua D. Reiss* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05858.txt.
Introduction
Jiachen Luo, Huy Phan, Lin Wang and Joshua Reiss wrote a 2025 paper on reading emotion from speech by fusing sound and text. Subtle feeling shifts are hard to catch from either channel alone. Their method stacks three modules. One models how the two channels connect, using an encoder-decoder shape.
A second module sharpens what each channel adds to the other. A third trims cross-channel noise while keeping true correlations. Two standard emotion data sets hosted the tests. The fused system beat the best prior baselines on both. The authors credit the layered attention design for the gain.
Catalog record
| CREATOR | Jiachen Luo, Huy Phan, Lin Wang, Joshua D. Reiss* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bimodal Connection Attention Fusion for Speech Emotion Recognition Jiachen Luo, Huy Phan, Lin Wang, Joshua D. Reiss* Abstract Multi-modal emotion recognition is challenging due to the difficulty of extracting features that capture subtle emotional differences. Understanding multi-modal interactions and connections is key to building effective bimodal speech emotion recognition systems. In this work, we propose Bimodal Connection Attention Fusion (BCAF) method, which includes three main modules: the interactive connection network, the bimodal attention network, and the correlative attention network. The interactive connection network uses an encoder-decoder architecture to model modality connections between audio and text while leveraging modality-specific features. The bimodal attention network enhances semantic complementation and exploits intra- and inter-modal interactions. The correlative attention network reduces cross-modal noise and captures correlations between audio and text. Experiments on the MELD and IEMOCAP datasets demonstrate that the proposed BCAF method outperforms existing state-of-the-art baselines. I Introduction Fig. 1: Architecture of the proposed Bimodal Connection Attention Fusion (BCAF) method. The method consists of three modules: the unimodal representation module, the connection attention fusion module, and the classification module. The core connection attention fusion module includes the interactive connection network, the bimodal attention network, and the correlative attention network, with details depicted in Figs. 2, 3, 4 and 5. The unimodal audio representation H_{a} and text representation H_{l} are input into the interactive connection network, the correlative attention network, and the bimodal attention network. The proliferation of mobile internet and smartphones has led to the widespread use of social networking platforms where users create and share content in various modalities, such as audio, text, and video. Extracting and analyzing emotions from this multimodal content has extensive applications in human-computer interaction, surveillance, robotics, and gaming [ 1 , 2 , 3 ] . However, effectively integrating multiple modalities remains a significant challenge in emotion recognition research. Previous multi-modal emotion recognition methods have achieved good performance [ 4 ] ; however, key challenges remain in multi-modal emotion recognition. Different modalities require independent preprocessing and feature extraction designs due to their heterogeneous nature [ 5 , 6 , 7 ] . To develop a model that is both applicable and generalizable across individual modalities and fusion models, it is essential to learn modality interactions and connections for learning discriminative emotional content. In multi-modal learning, modality connection in multi-modal learning refers to the extent to which information is shared across modalities, shaping a unified representation [ 8 ] . Unlike correlation, which quantifies modality dependence, modality connection captures the semantic alignment between modalities. For example, a higher vocal pitch may correlate with excitement, but modality connection determines if the audio tone, facial expressions, and text consistently reinforce that excitement. A strong modality connection ensures meaningful interaction among modalities, whereas a weak connection may lead to ambiguity. Leveraging modality connections enables a more nuanced understanding of emotions, improving the robustness and accuracy of emotion recognition systems. On the other hand, modality interaction refers to the relationships and dependencies between modalities, which can vary from strong correlations to weak or even conflicting signals [ 9 ] . While modality connection emphasizes shared content, modality interaction captures how different modalities influence one another. For example, imagine a meeting scenario where a participant says, “That’s great” in a flat, monotone voice while avoiding eye contact and crossing their arms. The verbal content suggests positivity, but the tone and body language convey disengagement or sarcasm. Such conflicting signals highlight the complexity of multi-modal interactions. Accurately interpreting these interactions requires understanding the context and modeling how modalities influence one another. Two core types of interactions are frequently encountered in multi-modal learning: intra-modal and inter-modal interactions [ 10 , 11 ] . Intra-modal interactions measure relationships within the same modality, while inter-modal interactions capture relationships between different modalities. Understanding and learning these interactions are essential for building an effective multi-modal emotion recognition system that achieves three key goals: multi-modal integration, robustness, and contextual awareness. Many multi-modal emotion recognition simply focused on exploring interactions across different modalities while often overlooking the importance of modality connections. Existing approaches simply concatenated data for joint fusion and aggregates decisions from various modalities through weighted averaging [ 5 , 12 , 13 ] , neglecting intra-modal interactions. Recently attention mechanisms have gained attraction due to their effectiveness in modeling modality interactions [ 14 , 15 , 16 ] . However, such approaches often lack a deeper understanding of modality interactions and connections. This limitation can lead to several issues, including suboptimal connection strategies, loss of critical cross-modal information, and reduced robustness when handling conflicting or incomplete data from specific modalities. Addressing these challenges is crucial for enhancing the performance and generalizability of multi-modal emotion recognition systems in real-world applications. Motivated by the above observations (Fig. 1), we propose a Bimodal Connection Attention Fusion (BCAF) framework for bimodal emotion recognition, which consists of three key modules: the uni-modal representation module, the connection attention fusion module, and the classification module (Fig. 2). The contribution is summarized as follows. First, we propose the interactive connection network to capture modality-specific features and analyze multi-modal connection between audio and text. Second, We investigate the bimodal attention network, which assigns dynamic weights to comprehensively learn intra- and inter-modal interactions between audio and text. Finally, We design the correlative attention network to effectively filter out incorrect cross-modal relationships and enhance learning of both intra- and inter-modal interactions between audio and text. Experimental results validate the effectiveness of the proposed method on public datasets. The remainder of this paper is organized as follows: Section II presents a brief literature review. Section III describes our method in detail. Section IV outlines the experiments conducted. Section V discusses the results. Finally, Section VI provides conclusions based on this work. II Related Work Emotion recognition in conversations has found widespread applications across various fields [ 4 ] . Humans convey emotions through multiple modalities, including speech, facial expressions, and body postures [ 12 , 17 , 18 ] . Among these, speech is a crucial modality, carrying emotional cues through both paralinguistic features and linguistic content. Since different modalities provide complementary information, relying on a single modality is often insufficient for accurate emotion recognition [ 5 , 19 ] . Therefore, combining information from multiple modalities enhances the ability to discern emotions, as each modality can complement or augment others, providing richer emotion-relevant information. Consequently, multi-modal approaches generally yield superior results compared to uni-modal methods, leading to substantial efforts in developing and exploring multi-modal fusion techniques for more accurate emotion recognition in conversations. Multi-modal fusion methods are broadly categorized into early fusion, late fusion, and hybrid fusion [ 1 , 2 , 4 , 20 ] . Early fusion combines features from different modalities at the input level but often overlooks complex inter-modal dependencies [ 21 , 22 ] . Late fusion integrates decision-level outputs from unimodal classifiers [ 23 , 24 ] , simplifying the process but limiting cross-modal interactions. Hybrid fusion addresses these limitations by combining intermediate representations of multiple modalities [ 25 , 26 ] . Recent multi-modal emotion recognition approaches employ advanced fusion strategies to capture intra- and inter-modal interactions. Transformer-based models [ 27 ] and gated recurrent units [ 28 ] have been explored for