WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Feng Li (🖂), Jiusong Luo, Wanjun Xia · 2024

Read WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition by Feng Li (🖂), Jiusong Luo, Wanjun Xia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05558.txt.

Introduction

Feng Li, Jiusong Luo, and Wanjun Xia build WavFusion in 2024 for speech mood reads that blend sound with text. Speech alone blurs mood lines, yet past blends gloss over how modes play on each other. Their frame pairs a strong speech coder with a gated cross-mode watch and a like-trait gap loss.

Tests on two bench sets, IEMOCAP and MELD, beat past top frames on mood calls. The gains back fine cross-mode play and trait-sharp codes as the levers that count. The work adds a clean path for joint speech mood reads.

Catalog record

CREATORFeng Li (🖂), Jiusong Luo, Wanjun Xia
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition Feng Li (🖂), Jiusong Luo, Wanjun Xia Abstract Speech emotion recognition (SER) remains a challenging yet crucial task due to the inherent complexity and diversity of human emotions. To address this problem, researchers attempt to fuse information from other modalities via multimodal learning. However, existing multimodal fusion techniques often overlook the intricacies of cross-modal interactions, resulting in suboptimal feature representations. In this paper, we propose WavFusion, a multimodal speech emotion recognition framework that addresses critical research problems in effective multimodal fusion, heterogeneity among modalities, and discriminative representation learning. By leveraging a gated cross-modal attention mechanism and multimodal homogeneous feature discrepancy learning, WavFusion demonstrates improved performance over existing state-of-the-art methods on benchmark datasets. Our work highlights the importance of capturing nuanced cross-modal interactions and learning discriminative representations for accurate multimodal SER. Experimental results on two benchmark datasets (IEMOCAP and MELD) demonstrate that WavFusion succeeds over the state-of-the-art strategies on emotion recognition. 1 Introduction Recently, speech emotion recognition (SER) is a fascinating field that utilizes technology to analyze and identify different emotions present in human speech [ 1 ] . This technology has various applications, including in customer service and market research [ 2 ] , learning and education [ 3 ] , mental health [ 4 ] , and social media analytics [ 5 ] . In real-life scenarios, humans express emotions not only through speech but also through alternative modalities, such as text and visuals [ 6 , 7 ] . Previous studies on SER typically rely on speech information. However, different modalities provide complementary information for emotion recognition, and emotion recognition of the single modality is not inadequate to meet real-world demands. To address this problem, researchers utilize multimodal information to identify emotional states [ 8 ] . In the domain of Multimodal Emotion Recognition (MER), the information of diverse modalities is complementary, providing additional cues to mitigate semantic and emotional ambiguities. In addition to multimodality, another challenge of SER is achieving better interaction during the fusion of different modalities. Firstly, multimodal data often exhibit asynchrony [ 9 ] . For instance, visual signals typically precede audio signals by approximately 120 ms in emotional expressions [ 10 ] . This asynchronicity poses a challenge to feature fusion and model design, necessitating methods to address temporal alignment and matching issues. To address this issue, Tsai et al. [ 11 ] have proposed specific asynchronous models and cross-modal attention mechanisms. Zheng et al. [ 12 ] solved heterogeneity among different encoder output features by employing unsupervised training of a multi-channel weight-sharing autoencoder. This approach minimizes the differences among features extracted from different modalities. Additionally, the interactions are simulated by supervised training of cascaded multi-head attention mechanisms. However, most methods with cross-modal attention mechanisms ignore redundant information during the fusion process, thus restricting the performance of MER. Additionally, samples with the same emotion in multimodal data may exhibit differences across modalities, referred to as homogeneous feature differences. For instance, some features in speech and text may exhibit formal similarity but convey different emotional states [ 13 ] . Hazarika et al. [ 14 ] projected each modality into two different subspaces capturing modality-invariant and modality-specific features. However, they only considered the differences between the different emotion of same modalities and ignored the differences between different modalities with the same emotion. DialogueTRM explores intra- and inter-modal emotional behaviors in conversations, using Transformers to model the context [ 15 ] . MMGCN proposes a multimodal fusion approach via a deep graph convolution network, modeling the interactions between different modalities using a graph [ 16 ] . MM-DFN introduces a dynamic fusion network that leverages intra- and inter-modal information at different levels of representation [ 17 ] . M2FNet proposes a multi-modal fusion network that learns and fuses complementary information from audio, visual, and textual modalities [ 18 ] . Therefore, in this paper, we propose a novel arhitecture called WavFusion for emotion recognition. Unlike DialogueTRM, WavFusion specifically focuses on incorporating wav2vec2.0 [ 19 ] with a gated cross-modal attention mechanism to dynamically fuse multimodal features. Additionally, WavFusion introduces multimodal homogeneous feature discrepancy learning to distinguish between same-emotion but different-modality representations. WavFusion does not rely on graph-based modeling but instead uses a transformer architecture with a modified cross-modal attention mechanism. WavFusion also emphasizes capturing both global and local visual information through the A-GRU-LVC module. While MM-DFN focuses on dynamic fusion strategies, WavFusion emphasizes the use of wav2vec2.0 pre-trained representations and a gated cross-modal attention mechanism to mitigate redundant information during fusion. Additionally, WavFusion incorporates multimodal homogeneous feature discrepancy learning to distinguish between representations of the same emotion across different modalities. The main contributions of this paper can be summarized as follows: • We propose a multimodal speech emotion recognition model (WavFusion) that leverages the power of wav2vec 2.0 and incorporates textual and visual modalities to enhance the performance of audio-based emotion recognition. • We integrate the designed gated cross-modal attention mechanism into the wav2vec 2.0 model to mitigate redundant information during the fusion process. Meanwhile, we employ multimodal homogeneous feature discrepancy learning to enhance the discriminative capability of the model. • Experimental results on two benchmark datasets demonstrate the effectiveness of the proposed method. Our WavFusion succeeds over existing state-of-the-art methods. 2 Proposed Method 2.1 Problem Statement Given a multimodal signal S_{j}=\left\{S_{j}^{a},S_{j}^{t},S_{j}^{v}\right\} , we can represent the unimodal raw sequence extracted from the video fragment j as S_{j}^{m},m\in\{a,t,v\} . Here, the modalities are denoted by \{a,t,v\} , which refer to audio, text, and visual modalities. In WavFusion, we aim to predict the emotion category for each utterance. It focuses on categorizing the emotion conveyed in each utterance, assigning it to a specific emotion class or category, y_{j}\in{R}^{c} . c is the number of emotion categories. Figure 1 illustrates the overall structure of WavFusion, including an auxiliary modal encoder, a primary modal encoder, and multimodal homogeneous feature discrepancy learning. The orange color represents positive emotions and the green color represents negative emotions. 2.2 Auxiliary Modality Encoder Video Representation For the visual modality, we use the EfficientNet pretrained model as a feature extractor to obtain visual features \mathbf{e}_{j}^{v} . This model is a self-supervised framework for visual representation learning. In this paper, we attempt to extend EfficientNet to emotion recognition. \mathbf{e}_{j}^{\mathrm{v}} can be formulated as: \mathbf{e}_{j}^{v}=\Phi_{{visual}}\left(S_{j}^{v}\right) (1) where \Phi_{{visual}} denotes the function of EfficientNet model. Figure 1: The overview of WavFusion. On the other hand, we consider the context and situation conveyed by the global information in the visual modality, along with the specific details of actions and expressions from local information. The visual feature is fed into the proposed A-GRU-LVC module, which aims to extract both global and local features. X_{j}^{v1}=F_{SA}\left(F_{GRU}\left(\mathbf{e}_{j}^{v}\right)\right) (2) where F_{SA} and F_{GRU} denote the learning functions of GRU and self-attentive mechanism, respectively. Simultaneously, to preserve local corner point regions and extract local information, a learnable visual center (LVC) is implemented on the visual features [ 20 ] . This LVC aggregates features from local areas, ensuring that important local information is retained. In contrast to the approach, we utilize one-dimensional convolution instead of two-dimensional convolution. X_{j}^{v2}=F_{LVC}\left(e_{j}^{v}\right) (3) where F_{LVC} denotes the learning functions of the LVC block. Finally, the output of the A-GRU-LVC block

Read the complete text