PARROT : Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition
Orchid Chetia, Mohd Mujtaba, Girish*, Swarup Ranjan, Jaya Sai Kiran, Arun Balaji, Rajesh · 2025
Read PARROT : Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition by Orchid Chetia, Mohd Mujtaba, Girish*, Swarup Ranjan, Jaya Sai Kiran, Arun Balaji, Rajesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01138.txt.
Introduction
Orchid Chetia Phukan and co-authors fuse two kinds of speech models in a 2025 paper to read emotion in the voice. One family uses attention, and the other, newer family is built on the Mamba architecture.
Both families, pre trained without labels, reach like scores on emotion tasks. The authors guessed that joining their strengths would beat joining two models of the same kind. Their framework, PARROT, runs the two in parallel branches and merges them with optimal transport and a Hadamard product step. The fused system set new top results against single models, same kind fusions, and standard fusion methods. The work makes a case for mixed model fusion in speech.
Catalog record
| CREATOR | Orchid Chetia, Mohd Mujtaba, Girish*, Swarup Ranjan, Jaya Sai Kiran, Arun Balaji, Rajesh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01138
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PARROT : Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition Orchid Chetia, Mohd Mujtaba, Girish*, Swarup Ranjan, Jaya Sai Kiran, Arun Balaji, Rajesh Abstract The emergence of Mamba as an alternative to attention-based architectures has led to the development of Mamba-based self-supervised learning (SSL) pre-trained models (PTMs) for speech and audio processing. Recent studies suggest that these models achieve comparable or superior performance to state-of-the-art (SOTA) attention-based PTMs for speech emotion recognition (SER). Motivated by prior work demonstrating the benefits of PTM fusion across different speech processing tasks, we hypothesize that leveraging the complementary strengths of Mamba-based and attention-based PTMs will enhance SER performance beyond the fusion of homogenous attention-based PTMs. To this end, we introduce a novel framework, PARROT that integrates parallel branch fusion with Optimal Transport and Hadamard Product. Our approach achieves SOTA results against individual PTMs, homogeneous PTMs fusion, and baseline fusion techniques, thus, highlighting the potential of heterogeneous PTM fusion for SER. Phukan* Akhtar* Behera Patibandla Buduru Sharma IIIT-DelhiIndia V.B.S.P.UIndia UPESIndia Independent ResearcherIndia University of TartuEstonia Plaksha UniversityIndia keywords Speech Emotion Recognition, Pre-Trained Models, Mamba-based Models, Attention-based Models † † email: Correspondence: [email protected] † † footnotetext: * Contributed equally as a first authors. 1 Introduction Speech Emotion Recognition (SER) bridges human-computer interaction, finds applications in mental health monitoring as well as in empathetic AI systems [ 1 , 2 ] . It enables machines to understand and respond to human emotions, fostering more natural and intuitive interactions. Traditional SER research often employs handcrafted features such as MFCCs, which capture the spectral properties of speech and have proven effective in representing emotional cues. These features were initially modelled with classical ML techniques such as SVM [ 3 ] , tree-based methods [ 4 ] . This was followed by the use of deep learning techniques [ 5 , 6 ] . By the end of end of last decade, the landscape of SER research has changed for the better through the use and the wide scale availability of self-supervised learning (SSL) pre-trained models (PTMs). These PTMs trained on large-scale diverse data provides performance benefits as well as discard the necessity of training models from scratch. These PTMs have led to significant development in SER. As such, researchers have explored various state-of-the-art (SOTA) PTMs [ 7 , 8 , 9 , 10 ] . Pepino et al. [ 11 ] used wav2vec2 with LSTM and CNN downstreams and showed its effectiveness in comparison to conventational features such as eGeMAPS and spectrogram. Morais et al. [ 12 ] gave a comprehensive comparison of various SSL PTMs such as wav2vec2, HuBERT with different downstream networks. These PTMs have predominantly utilized attention-based architectures, which excel at capturing contextual dependencies in speech signals. In recent times , an alternative architecture type of PTMs has captured attention in the community: mamba-based PTMs [ 13 , 14 , 15 ] . These mamba-based PTMs are based on top of mamba architecture which is structured state-space model (SSM) and has emerged as a promising alternative due to its ability to efficiently model long-range dependencies with linear complexity [ 16 ] . These mamba-based PTMs have set benchmarks in sequence modeling tasks and initial research suggest that mamba-based PTMs perform competitively with, or even surpass, attention-based PTMs in SER [ 13 ] . Prior research has also shown the benefits of PTM fusion for SER [ 17 ] due to their complementary behavior. This is also observed across various speech processing tasks such as speech recognition [ 18 ] and speech deepfake detection [ 19 ] . However, previous works have integrated only attention-based models. In this study, we fuse mamba and attention-based PTMs for SER and hypothesize that the fusion of these heterogeneous PTMs will yield richer and more robust representations for improved SER as attention-based PTMs will capture intricate global dependencies while mamba-based PTMs excels at efficient long-range processing . We are the first study to the best of our knowledge to explore fusion of such heterogeneous PTMs for SER. To our end, we propose, PARROT ( PAR allel B R anch Hadamard O ptimal T ransport), a novel framework to align and integrate heterogeneous mamba and attention-based PTMs. It employs parallel branch fusion, incorporating the hadamard product and optimal transport. Hadamard product captures local interactions by performing element-wise operations between the representations, preserving fine-grained details. Meanwhile, Optimal Transport operates at a global scale, aligning the distributions of features across the two PTMs, ensuring that the fused representations are coherent and effectively integrated for improved performance. The key contributions of our study are as follows: • We introduce PARROT , a novel framework that encompasses parallel branch fusion with Hadamard Product and Optimal Transport that inherently captures local interactions and global interaction for improved SER performance. • With PARROT through the synergy of mamba and attention-based PTMs, we achieve the topmost most performance across different SER datasets (CREMA-D ( English ), emo-DB ( German ), MESD ( Mexican Spanish )) than individual PTMs and homogeneous fusion of attention-based PTMs. It also reports better performance in comparison to baseline fusion methods. These PTMs are SOTA SSL PTMs for SER, thus, PARROT achieves SOTA performance for SER with its heterogeneous fusion. All code and models used in this study are accessible at: https://github.com/Helix-IIIT-Delhi/PARROT-SER Figure 1 : Proposed Framework: PARROT ; OTFB, HPFB stands for Optimal Transport Fusion Block and Hadamard Product Fusion Block respectively; U_{1}1 , U_{2}2 represents the representational space of PTM 1 and PTM2; U_{1}2 , U_{2}1 represents the transported space of PTM 2 to PTM 1 and vice versa 2 Pre-Trained Models In this section, we discuss the PTMs considered in our study. Audio-MAMBA [ 13 ] * * * https://github.com/SarthakYadav/audio-mamba-official?tab=readme-ov-file : Audio Mamba is a selective state space model that is trained in a self-supervised fusion to learn general-purpose representations from randomly masked spectrogram patches. Trained on the AudioSet dataset, it outperforms its attention-based counterparts baselines across diverse speech and audio tasks including SER. We use the tiny, small, and base versions of 4.8M, 17.9M, and 69.3M parameters. WavLM [ 20 ] † † † https://huggingface.co/microsoft/wavlm-base : It is a SOTA attention-based PTM on SUPERB that integrates masked speech modeling with denoising objectives during its pre-training, effectively learning robust representations from noisy and clean speech alike. We have used the base version with 94.70M parameters and trained on librispeech 960 hours of english speech. UniSpeech-SAT [ 21 ] ‡ ‡ ‡ https://huggingface.co/microsoft/unispeech-sat-base : It is also a SOTA attention-based PTM on SUPERB and trained in a self-supervised fashion with speaker-aware multi-task learning. We utilize the base version of 94.68M parameters and pre-trained on 960 hours of librispeech english data. Wav2vec2 [ 22 ] § § § https://huggingface.co/facebook/wav2vec2-base : This contrastive SSL attention-based PTM that learns speech representations by masking segments of latent features. We use its base version trained on librispeech 960 hours english data with 95.04M parameters. Wav2vec2 improves previous SOTA methods in speech recognition. HuBERT [ 23 ] ¶ ¶ ¶ https://huggingface.co/facebook/hubert-base-ls960 : HuBERT employs a SSL framework that iteratively refines its representations using k-means clustering and solves a BERT-like masked prediction objective. HuBERT improves over Wav2vec2 in speech recognition. We