HYFuse : Aligning Heterogeneous Speech Pre-Trained Representations in Hyperbolic Space for Speech Emotion Recognition

Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Pailla Balakrishna, Arun Balaji, Rajesh · 2025

Read HYFuse : Aligning Heterogeneous Speech Pre-Trained Representations in Hyperbolic Space for Speech Emotion Recognition by Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Pailla Balakrishna, Arun Balaji, Rajesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03403.txt.

Introduction

Orchid Chetia Phukan and co-authors fuse two kinds of speech features to read emotion from voice, in work from twenty twenty-five. Codec based features catch fine sound traits like pitch and tone color. Learned features from big speech models catch meaning and rhythm. No one had fused the two families before.

Their framework, called HYFuse, maps both feature sets into hyperbolic space and merges them there. Fusing a learned speaker feature with a codec feature beat each source alone. It also beat fusing features from within one family. The team reports top results on the task. They credit the richer shared picture that the two feature types give together.

Catalog record

CREATOROrchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Pailla Balakrishna, Arun Balaji, Rajesh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03403

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HYFuse : Aligning Heterogeneous Speech Pre-Trained Representations in Hyperbolic Space for Speech Emotion Recognition Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Pailla Balakrishna, Arun Balaji, Rajesh Abstract Compression-based representations (CBRs) from neural audio codecs such as EnCodec capture intricate acoustic features like pitch and timbre, while representation-learning-based representations (RLRs) from pre-trained models trained for speech representation learning such as WavLM encode high-level semantic and prosodic information. Previous research on Speech Emotion Recognition (SER) has explored both, however, fusion of CBRs and RLRs haven’t been explored yet. In this study, we solve this gap and investigate the fusion of RLRs and CBRs and hypothesize they will be more effective by providing complementary information. To this end, we propose, HYFuse , a novel framework that fuses the representations by transforming them to hyperbolic space. With HYFuse , through fusion of x-vector (RLR) and Soundstream (CBR), we achieve the top performance in comparison to individual representations as well as the homogeneous fusion of RLRs and CBRs and report SOTA. Phukan* Akhtar* Behera Reddy Buduru Sharma IIIT-DelhiIndia UPESIndia V.B.S.P.UIndia Independent ResearcherIndia Reliance AIIndia University of TartuEstonia Plaksha UniversityIndia keywords Speech Emotion Recognition, Pre-Trained Models, Neural Audio Codec, Representations † † email: Correspondence: [email protected] † † footnotetext: * Contributed equally as a first authors. 1 Introduction Speech Emotion Recognition (SER) plays a pivotal role in human-computer interaction, enabling systems to identify and understand the emotional nuances expressed in speech [ 1 ] . By analyzing vocal attributes such as pitch, tone, and rhythm, SER systems uncover the intricate nuances of human affect. SER has far-reaching implications, from enhancing mental health monitoring in healthcare [ 2 ] to transforming educational tools by gauging student engagement [ 3 ] . Initial research in SER mostly focused on the usage of spectral features such MFCC with classical ML models such as GMM [ 4 ] , SVM [ 5 ] . This was succeeded by the use of deep learning models such as LSTM [ 6 ] , CNN, CNN-LSTM [ 7 ] , etc. Recent strides in SER research have seen the usage of representations from state-of-the-art (SOTA) Pre-trained models (PTMs). These representations have provided substantial performance benefit and has led to sufficient development in SER. These representations can be primarily categorized into two types: Representation-learning based representations (RLRs) derived from speech PTMs such as Wav2vec2 [ 8 ] , WavLM [ 9 ] , XLS-R [ 10 ] , etc. and compression-based representations (CBRs) extracted from neural audio codecs such as EnCodec [ 11 ] , DAC [ 12 ] , and Soundstream [ 13 ] . PTMs for RLRs are generally trained for speech representation learning and it can be both for a particular language or multilingual, however, neural audio codecs (NACs) are trained for compression of input data following a encoder-decoder modeling architecture. Researchers have explored various RLRs such as Wav2vec2 [ 14 ] , HuBERT [ 15 ] , etc. for SER. Also, usage of compression-based representations (CBRs) from NACs for SER has gained recent traction in the community. Wu et al. [ 16 ] gave a initial exploration of CBRs for SER by investigating different NACs such as Encodec, DAC, Speech Tokenizer and so on. However, they only focused on English SER. Ren et al. [ 17 ] extended to chinese SER and gave a much more comprehensive analysis of various SOTA CBRs with the inclusion of more NACs. Furthermore, Mousavi et al. [ 18 ] presented the first comparative study of CBRs and RLRs for SER. Further. Wu et al. [ 19 ] also explored the fusion of RLRs such as Wav2vec2, WavLM, Unispeech-SAT, and so on for more due to existence of complementary behavior of such representations for more improved SER. Such improvement due to the combination of PTM representations can also be seen across various related speech processing tasks such as speech recognition [ 20 ] , synthetic speech detection [ 21 ] . However, no focus on the fusion of heterogeneous representations i.e. RLRs and CBRs have been given, despite extensive research into SER with PTM representations. In this work, for the first time, to the best of our knowledge, we explore such fusion of heterogenenous representations (RLRs and CBRs). We hypothesize that fusion of RLRs and CBRs will lead to further improvement in SER performance by the exploitation of complementary information of RLRs and CBRs. CBRs captures the low-level features like pitch, timbre and RLRs encodes higher-level prosodic patterns. To aid in effective fusion, we propose a novel framework, HYFuse (Fusion in Hyperbolic Space) that transforms the representations from euclidean space to hyperbolic space and performs fusion through mobius addition. As far as we know, this is the first study to investigate the usage of hyperbolic space for fusion of representations in the context of SER. The fusion of CBRs and RLRs in hyperbolic space allows for the preservation of their hierarchical relationships and complementary features, ensuring that both low-level acoustic details and high-level prosodic patterns are effectively aligned and integrated. Figure 1 : HYFuse ; x\oplus y represent mobius addition The key contributions of this work are: • We propose, HYFuse (Figure 1 ), a novel framework for fusing RLRs and CBRs by transforming them into hyperbolic space and leveraging the strengths of hyperbolic geometry for effective fusion of RLRs and CBRs. • Using HYFuse , with the fusion of x-vector (RLR) and SoundStream (CBR) representations, we achieve superior performance compared to individual representations and homogeneous fusions of RLRs and CBRs. Our framework sets new SOTA results on the CREMA-D and Emo-DB benchmark datasets, establishing the efficacy of combining RLRs and CBRs for SER. We have released the code and models from this work at: https://github.com/Helix-IIIT-Delhi/HYFuse-SER 2 Pre-Trained Representations In this section, we give a brief overview of the PTMs and NACs behind RLRs and CBRs. 2.1 Representation-learning WavLM 1 1 1 https://huggingface.co/microsoft/wavlm-base [ 9 ] has shown SOTA performance in multiple speech processing tasks within SUPERB. We adopt its base version, which consists of 94.70 million parameters and is pre-trained on 960 hours of Librispeech. Similarly, Wav2Vec2 2 2 2 https://huggingface.co/facebook/wav2vec2-base [ 8 ] is included in our study as a contrastive learning-based representation-learning based PTM. We use its base variant, which has 95.04 million parameters and is also pre-trained on 960 hours of Librispeech. We also incorporate HuBERT 3 3 3 https://huggingface.co/facebook/hubert-base-ls960 [ 22 ] , a speech PTM inspired by the BERT architecture, pre-trained on 960 hours of Librispeech. We utilize its base version, which consists of 94.68 million parameters. Additionally, we consider x-vector 4 4 4 https://huggingface.co/speechbrain/spkrec-xvect-voxceleb [ 23 ] , a time delay neural network designed specifically for speaker recognition, comprising 4.2 million parameters. It is particularly relevant to our research, as its representations have been shown to be effective for SER [ 24 ] . All audio recordings are resampled to 16 kHz before passing to the PTMs. The PTMs remain frozen, and we extract RLRs from their final hidden states using average pooling. The resulting feature dimensions are 768 for WavLM, Wav2Vec2, and HuBERT, while x-vector produces 512-dimensional representations. 2.2 Compression Soundstream 5 5 5 https://github.com/haydenshively/SoundStream [ 13 ] is an efficient NAC designed for low-bitrate compression, utilizing an

Read the complete text