Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition

Ruoyu Zhao, Xiantao Jiang, F. Richard Yu, Fellow, IEEE, Victor C.M. Leung, Fellow, IEEE, Tao Wang, Shaohu Zhang · 2025

Read Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition by Ruoyu Zhao, Xiantao Jiang, F. Richard Yu, Fellow, IEEE, Victor C.M. Leung, Fellow, IEEE, Tao Wang, Shaohu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10408.txt.

Introduction

Ruoyu Zhao and co-authors tackle a hard speech task in a 2025 paper: reading feelings in voice across tongues. A tool trained on one tongue often fails on the next, since sounds and speech habits shift. Their method joins three views of the voice at once.

It blends a deep speech model, classic tone marks, and rhythm traits such as pitch and pace. A cross-focus method fuses the views, and skills learned on one large set carry over to new sets. Tests ran on seven sets in five tongues. Such tools could help call aids and care apps respond with more warmth, no matter the tongue spoken.

Catalog record

CREATORRuoyu Zhao, Xiantao Jiang, F. Richard Yu, Fellow, IEEE, Victor C.M. Leung, Fellow, IEEE, Tao Wang, Shaohu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10408

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition Ruoyu Zhao, Xiantao Jiang, F. Richard Yu, Fellow, IEEE, Victor C.M. Leung, Fellow, IEEE, Tao Wang, Shaohu Zhang Abstract Speech Emotion Recognition (SER) plays a crucial role in enhancing human-computer interaction. Cross-Linguistic SER (CLSER) has been a challenging research problem due to significant variability in linguistic and acoustic features of different languages. In this study, we propose a novel approach HuMP-CAT , which combines HuBERT, MFCC, and prosodic characteristics. These features are fused using a cross-attention transformer (CAT) mechanism during feature extraction. Transfer learning is applied to gain from a source emotional speech dataset to the target corpus for emotion recognition. We use IEMOCAP as the source dataset to train the source model and evaluate the proposed method on seven datasets in five languages (e.g., English, German, Spanish, Italian, and Chinese). We show that, by fine-tuning the source model with a small portion of speech from the target datasets, HuMP-CAT achieves an average accuracy of 78.75% across the seven datasets, with notable performance of 88.69% on EMODB (German language) and 79.48% on EMOVO (Italian language). Our extensive evaluation demonstrates that HuMP-CAT outperforms existing methods across multiple target languages. I Introduction Speech emotion recognition (SER) has attracted considerable attention for its potential applications in various areas, including human-computer interaction, healthcare, education, entertainment, transportation systems, and the Internet of Things (IoT) [ 1 , 2 , 3 ] . SER enable IoT devices or applications to better understand the user’s emotions and to respond more appropriately to the user’s psychological state. For example, in smart assistants (such as Siri, Alexa, etc.), if anger or happiness in the user’s voice can be recognized, the system can adjust the voice tone and respond appropriately to enhance the interactive experience. SER involves on identifying and understanding emotional states through vocal cues such as pitch, prosody, and Mel frequency cepstral coefficients (MFCC) [ 4 ] . Traditional SER systems that leverage classical machine learning, such as support vector machine (SVM) [ 5 ] , naive Bayes classifier [ 6 ] , and k -nearest neighbor (KNN) [ 7 ] , have been limited by language-specific constraints and have struggled to generalize emotional patterns across linguistic speech. The emerging field of cross-linguistic speech emotion recognition (CLSER) seeks to transcend these limitations by developing robust, adaptive models that can interpret emotional states irrespective of linguistic variations. However, CLSER faces several challenges. The first key challenge is the scarcity of large and balanced cross-linguistic emotion datasets. Most existing datasets are heavily biased towards a few languages such as English and European languages, which create significant representational gaps. The second challenge is to identify effective features for emotion recognition. Although numerous features such as pitch, prosody, MFCC, linear prediction cepstral coefficients (LPCC), and Gammatone frequency cepstral coefficients (GFCC) [ 3 ] , have been shown to correlate with emotions, it remains to be explored how to combine these features optimally to improve CLSER performance. The third challenge involves improving the generalization and versatility of recognition across languages. Due to significant variations in language such as accent, culture, and other factors between training and testing datasets, the majority of SER systems are limited to a single language corpus but do not generalize well to other language corpus. Recent advances in deep learning and representation learning have provided opportunities for SER due to their robust feature representation capability, capacity to manage complex features, ability to learn from unlabeled data, and scalability with larger datasets. Various deep learning models, such as convolutional neural networks (CNN), deep neural networks (DNN), long- and short-term memory networks (LSTM) and transformers, have been applied for automatic SER [ 8 , 9 , 10 ] . Self-supervised speech representation learning (SSRL) is a form of unsupervised learning that seeks to capture rich representations from the input speech signal itself [ 11 , 12 , 13 ] . which have been widely adopted in various speech processing tasks, including SER [ 14 , 15 , 16 ] . HuBERT [ 17 ] and Wav2vec [ 18 ] are prominent examples of SSRL methods applied in SER. These approaches have also been successfully applied in multilingual and cross-corpus SER [ 19 , 20 ] . HuBERT integrates representation information from speech recognition models to train a multimodal emotion recognition framework [ 21 ] . Sharma et al. [ 22 ] fine-tuned Wav2vec2 for multilingual and multitask learning. Although these methods utilize the capacity of SSRL models and transformer layers to extract suitable representations for SER, they do not explore the specific contribution of these transformer layers to CLSER. Moreover, the role of these layers in the fusion with other acoustic features of CLSER remains unexplored. In this paper, we propose HuMP-CAT , a novel CLSER framework that integrates Hu BERT, M FCC, P rosody, and CAT . HuMP-CAT leverages multi-feature fusion and cross-attention transformer (CAT) models to create a more universal approach to understanding emotions in diverse linguistic speech. The main contributions of this paper are as follows. • We propose a novel CLSER framework that integrates CAT models and multi-feature fusion to achieve higher SER accuracy compared to existing methods. • HuMP-CAT leverages transfer learning by using only 10% or 20% of the speaker data to fine-tune the source model, achieving an impressive average accuracy of 78.75% across seven diverse datasets. This demonstrates the model’s capacity to generalize SER with small training data. • We perform a comprehensive evaluation across five languages (e.g., English, German, Spanish, Italian, and Chinese) in seven datasets, examining the performance of various SSRL models and the combination of speech characteristics. Our extensive experiments demonstrate HuMP-CAT ’s generalization and versatility in CLSER. The remainder of this paper is organized as follows. Section II provides a brief review of related work on SER and CLSER. Section III presents the proposed multi-feature fusion method based on the CAT mechanism and discusses each module of the approach. Section IV describes the dataset, experimental procedure, and results. Conclusions are provided in Section V . II RELATED WORK TABLE I: Summary of major abbreviations Abbreviation Description and full name SER Speech Emotion Recognition CLSER Cross-Linguistic Speech Emotion Recognition MFCC Mel-Frequency Cepstral Coefficient HuBERT Hidden-Unit BERT SSRL Self-Supervised Speech Representation Learning CAT Cross-Attention Transformer AM-Softmax Additive Margin Softmax LSTM Long Short-Term Memory Networks DSCNN Deep Stride Convolutional Neural Networks UA Unweighted Accuracy WA Weighted Accuracy Fig. 1: General process of SER. In this paper, we present a novel CLSER framework that integrates CAT models and multi-feature fusion. This section reviews the related work on various aspects of existing SER and CLSER approaches. We also highlight the differences and contributions of our proposed framework compared to the existing methods. Table I lists the abbreviations and their corresponding meanings that often appear in the paper. II-A Speech Emotion Recognition As shown in Fig. 1 , the computational pipeline of SER comprises three critical modules: speech preprocessing, feature extraction, and sentiment classification, each playing a pivotal role in transforming raw acoustic signals into meaningful emotional representations. The preprocessing stage is the prerequisite for the subsequent extraction of features, including speech framing, windowing, normalization, and noise reduction. Feature extraction represents the transformation stage, where

Read the complete text