Enhancing Speech Emotion Recognition with Multi-Task Learning and Dynamic Feature Fusion

Honghong, Jing, Fanqin, Rong · 2025

Read Enhancing Speech Emotion Recognition with Multi-Task Learning and Dynamic Feature Fusion by Honghong, Jing, Fanqin, Rong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17878.txt.

Introduction

Honghong Wang, Jing Deng, Fanqin Meng, and Rong Zheng teach a speech model to hear emotion by making it do four jobs at once, in a 2025 paper. The catalog record lists the authors by first names only; the printed byline is used here.

Alongside emotion, the model learns gender, speaker identity, and the words spoken. A co-attention module lets the tasks trade features dynamically, so context flows where it helps. A new loss function leans on hard and rare examples, fighting the imbalance of real emotion data. The system fine-tunes a self-supervised speech model rather than starting cold. On a naturalistic challenge task, the approach improved performance markedly. The authors credit the cross-task conversation.

Catalog record

CREATORHonghong, Jing, Fanqin, Rong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Speech Emotion Recognition with Multi-Task Learning and Dynamic Feature Fusion Honghong, Jing, Fanqin, Rong Abstract This study investigates fine-tuning self-supervised learning (SSL) models using multi-task learning (MTL) to enhance speech emotion recognition (SER). The framework simultaneously handles four related tasks: emotion recognition, gender recognition, speaker verification, and automatic speech recognition. An innovative co-attention module is introduced to dynamically capture the interactions between features from the primary emotion classification task and auxiliary tasks, enabling context-aware fusion. Moreover, We introduce the Sample Weighted Focal Contrastive (SWFC) loss function to address class imbalance and semantic confusion by adjusting sample weights for difficult and minority samples. The method is validated on the Categorical Emotion Recognition task of the Speech Emotion Recognition in Naturalistic Conditions Challenge, showing significant performance improvements. Wang Deng Meng Zheng nocounter]Beijing Fosafer Information Technology Co., Ltd.China keywords speech emotion recognition, multi-task learning, class imbalance † † email: {wanghonghong, dengjing, mengfanqin, zhengrong}@fosafer.com 1 Introduction Recently, the rapid advancement of artificial intelligence has significantly improved the role of speech in human-computer interaction (HCI) [ 1 ] . As a crucial element of HCI, Speech Emotion Recognition (SER) has emerged as a key area of research in multimodal emotion analysis, with the goal of accurately identifying and interpreting the speaker’s emotional state. SER currently encounters several challenges. A major challenge is the scarcity of high-quality emotion-labeled datasets. Furthermore, the emotional features of speech are dynamic and varying over time, with valuable information distributed unevenly in the domain of time and frequency [ 2 ] . Traditional deep learning models, such as convolutional neural networks (CNNs) [ 3 ] , are constrained by a fixed perceptual field, making it difficult to accurately capture emotional cues, which often leads to subpar recognition performance. To address these challenges, researchers have increasingly turned to self-supervised learning (SSL) models, leading to the development of powerful end-to-end speech representation models like wav2vec 2.0 [ 4 ] , WavLM [ 5 ] , and HuBERT [ 6 ] . These models improve emotion recognition by extracting high-dimensional representations from raw waveform signals, leveraging large-scale unsupervised pre-training to capture rich semantic, phonetic, and paralinguistic information [ 7 ] . Since paralinguistic and acoustic features are believed to carry personalized emotional information [ 8 ] , researchers have increasingly turned to multi-task learning (MTL) approaches [ 9 , 10 ] , which simultaneously train the SER task alongside these auxiliary tasks. By sharing parameters across multiple tasks, the model can better extract emotional information from speech features, resulting in significant improvements in SER performance [ 11 ] . However, traditional MTL methods often fail to fully take advantage of the representations in each auxiliary task. As a result, integrating information from paralinguistic factors, such as speaker identity, gender, and speech content, with the emotion branch has been shown to further enhance emotion recognition performance [ 12 ] . The objective of the Speech Emotion Recognition in Naturalistic Conditions Challenge [ 13 ] is to predict the emotional state of speakers in real-life scenarios. This challenge consists of two tracks: (1) Categorical Emotion Recognition and (2) Emotional Attributes Prediction. The Categorical Emotion Recognition track involves classifying each sample into one of eight emotional categories: anger, happiness, sadness, fear, surprise, contempt, disgust, and neutral. As shown in Figure 1, the training and validation data for the challenge are typically imbalanced. Figure 1: Distribution of the challenge training and validation sets across the eight emotional labels. This paper presents a SER model that employs a MTL approach. The model incorporates a co-attention module to integrate emotion, gender, speaker identity, and speech content, enabling it to capture the relationships between emotional information and other auxiliary factors. To address the challenges of recognizing a limited number of categories and mitigating semantic confusion caused by the long-tailed distribution of emotion categories in the challenge data, we introduce the Sample Weighted Focal Contrastive (SWFC) loss function [ 15 ] . We have performed a comprehensive evaluation and demonstrated the effectiveness of the proposed approach. 2 Proposed Method 2.1 Model Architecture As shown in Figure 2, our research framework consists of three key stages: feature extraction, pooling, and classification. The network follows an end-to-end structure, where raw speech is first up-sampled using a convolutional feature encoder. Then, deep feature extraction is performed by a Transformer Encoder. Recognizing the varying importance of different feature layers in the SSL model for different downstream tasks, we employ a learnable weighting sum method [ 16 ] . This method adaptively integrates the hidden representations from all layers of the SSL model at the utterance level to generate a unified representation. Figure 2: Network structure of our proposed MTL method. The emotion, gender and speaker recognition branches take the global representations encoded by the attentive statistics pooling layer [ 16 ] as input, while the automatic speech recognition (ASR) branch uses the high-dimensional representations of the SSL model directly. The multi-task branching network then predicts emotion, gender, speaker, and ASR outputs, respectively. Notably, the emotion recognition branch incorporates a Co-attention module that facilitates dynamic interaction between emotion features and other auxiliary task features. During training, the emotion, gender, and speaker branches are optimized using a cross-entropy loss function, while the ASR branch uses a connectionist temporal classification (CTC) loss function [ 17 ] . To further enhance performance, we introduce a SWFC loss function, which operates on the global representation output from the SSL model. This loss function enables the SSL model to focus more on challenging, less frequent samples during fine-tuning. The overall objective function L of the model is defined as: \displaystyle L \displaystyle=(1-3\alpha)L_{\mathrm{emotion}}+\alpha L_{\mathrm{gender}} \displaystyle\quad+\alpha L_{\mathrm{speaker}}+\alpha L_{\mathrm{ASR}}+\beta L_{\mathrm{SWFC}} (1) where \alpha and \beta are hyperparameter for each auxiliary task. 2.2 Multi-task Learning Network The proposed MTL network includes a primary emotion recognition task and three auxiliary tasks. The emotion, gender, and speaker recognition branches each utilize a two-layer linear classification network, with each layer comprising LayerNorm, a ReLU activation function, and Dropout. Specifically, the ASR branch incorporates an LSTM network to enhance temporal modeling capabilities before the two-layer linear classification network. Inspired by [ 8 ] ,A unique feature of our approach is the dynamic fusion of hidden layer representations as SSL features. In addition, we employ a co-attention mechanism to interactively fuse the outputs of the first linear layer from each auxiliary task with the corresponding features from the emotion recognition task. This fusion process enables the emotion recognition task to be effectively guided by auxiliary task information, with the final predictions generated through a linear classifier. 2.3 Feature Fusion with Co-attention The co-attention module begins by mapping the features from each branch to the same dimension using a linear projection. Then it applies a dot-product attention mechanism to compute the interaction weights between the emotional features and each auxiliary task feature. Specifically, separate attention score matrices are calculated between the emotional features and the speaker, gender, and speech content features. These auxiliary features are weighted and aggregated through Softmax normalization. Finally, the dynamically fused multi-task representations are combined with the original emotional features, resulting in enhanced emotion representations that incorporate complementary cross-task information. 2.4 SWFC Loss This study leaves behind a novel loss function, SWFC, specifically designed to mitigate class

Read the complete text