Enhancing In-Domain and Out-Domain EmoFake Detection via Cooperative Multilingual Speech Foundation Models

Orchid Chetia, Mohd Mujtaba, Girish*, Arun Balaji · 2025

Read Enhancing In-Domain and Out-Domain EmoFake Detection via Cooperative Multilingual Speech Foundation Models by Orchid Chetia, Mohd Mujtaba, Girish*, Arun Balaji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12595.txt.

Introduction

Chetia Phukan and co-authors spot fake emotional speech with models raised on many languages, in a 2025 paper. Faked voices that mimic feelings threaten trust in calls and media. A model trained on speech in many tongues learns fine shades of pitch and tone, which may help it smell a fake. The team benchmarked the top speech models head to head on same-language and cross-language fakes.

Models raised on many languages won both rounds. The team then fused models with a blend of two math tricks that let each cover the other's blind spots. The fused system, THAMA, beat single models, older fusion styles, and past best methods in both settings. The authors pitch broad language upbringing as the base skill for fake-voice hunting, whatever the tongue.

Catalog record

CREATOROrchid Chetia, Mohd Mujtaba, Girish*, Arun Balaji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing In-Domain and Out-Domain EmoFake Detection via Cooperative Multilingual Speech Foundation Models Orchid Chetia, Mohd Mujtaba, Girish*, Arun Balaji Abstract In this work, we address EmoFake Detection (EFD). We hypothesize that multilingual speech foundation models (SFMs) will be particularly effective for EFD due to their pre-training across diverse languages, enabling a nuanced understanding of variations in pitch, tone, and intensity. To validate this, we conduct a comprehensive comparative analysis of state-of-the-art (SOTA) SFMs. Our results shows the superiority of multilingual SFMs for same language (in-domain) as well as cross-lingual (out-domain) evaluation. To our end, we also propose, THAMA for fusion of foundation models (FMs) motivated by related research where combining FMs have shown improved performance. THAMA leverages the complementary conjunction of tucker decomposition and hadamard product for effective fusion. With THAMA , synergized with cooperative multilingual SFMs achieves topmost performance across in-domain and out-domain settings, outperforming individual FMs, baseline fusion techniques, and prior SOTA methods. Phukan* Akhtar* Buduru IIIT-DelhiIndia V.B.S.P.UIndia UPESIndia keywords EmoFake Detection, Speech Foundation Models, Multilingual Speech Foundation Models † † email: Correspondence: [email protected] 1 Introduction Driven by the development of comprehensive datasets and rigorous benchmarking challenges, the field of fake audio detection has seen significant development. Early efforts, ASVspoof [ 1 ] , laid the foundation for detecting spoofed speech, with subsequent iterations like ASVspoof 2017 [ 2 ] and ASVspoof 2019 [ 3 ] addressing replay attacks and synthetic speech generated via text-to-speech (TTS) and voice conversion (VC) systems. ASVspoof 2021 [ 4 ] further expanded its scope to encompass further more deepfake speech samples, reflecting the growing complexity of audio manipulation techniques. More recently, ADD challenges [ 5 , 6 ] introduced datasets incorporating partially manipulated audio, fostering advancements in identifying hybrid fake audio scenarios. Further, Muller et al. [ 7 ] explores multi-lingual TTS-generated audio across diverse languages. Additionally, specialized tasks, such as singing voice deepfake detection [ 8 ] , scenefake detection [ 9 ] have emerged, while the MLAAD dataset [ 7 ] explores multi-lingual TTS-generated audio across diverse languages. However, despite these advancements, an underexplored yet critical dimension remains: scenarios where the emotional state of speech is artificially modified i.e. EmoFake (EF) (Figure 1 ). While such technology holds promise for applications in entertainment, personalized speech systems, and human-computer interaction, its misuse raises serious concerns. EF can distort the intent of communication, sow distrust in audio-based media, and enable malicious applications, such as emotional manipulation, misinformation campaigns, and fraudulent schemes. For instance, altering the emotional tone of recorded evidence in legal contexts could undermine judicial integrity, while fabricating emotionally persuasive messages could exploit listeners’ psychological vulnerabilities at scale. These adverse implications highlight the urgent need for advanced detection frameworks that can effectively identify emotion-manipulated audio and mitigate its risks across sensitive domains, including security, media integrity, and forensics. As such Zhao et al. [ 10 ] presented the first database for EmoFake detection (EFD) and carried out an initial benchmarking. Figure 1 : Demonstration of EmoFake: Person A’s neutral speech is manipulated to synthesize an angry emotional tone in the audio while maintaining the same spoken content (’How are you?’) Building on this foundation, we extend the exploration of EFD and investigate speech foundation models (SFMs) for EFD as SFMs holds immense potential for improving EFD also shown in previous research for audio deepfake detection [ 11 , 12 , 13 ] . To our end, we hypothesize that multilingual SFMs will be most effective for EFD in comparison to other types of SFMs, as their broad linguistic pre-training will enable them to capture subtle emotional alterations in speech characteristics such as pitch, tone, intensity. For validating our hypothesis, we present a comparative study of state-of-the-art (SOTA) SFMs including multilingual, monolingual, speaker recognition and music foundation models (MFMs). We are the first study to explore foundation models (FMs) for EFD to best of our knowledge. We include MFMs in our analysis as we thought that MFMs due to their pre-training on diverse music data may provide them capability of extracting pitch, timbre, etc. characteristics effectively important for detecting emotions and might be helpful for EFD. Further motivated by related research in speech recognition [ 14 ] , speech emotion recognition [ 15 ] , synthetic speech detection [ 16 ] , where combination of FMs have shown improved performance, we investigate the fusion of FMs for EFD. We propose a novel framework, THAMA ( T ucker- HA da MA rd Conjunction), which utilizes the synergistic combination of Tucker decomposition and Hadamard product to enable effective fusion of the FMs. THAMA with the fusion of cooperative multilingual SFMs approach sets a new benchmark for same-language (in-domain) and cross-lingual (out-domain) EFD. To summarize, the main contributions of this work are as follows: • To the best of our knowledge, we conduct the first large-scale and comprehensive comparative study of various SOTA SFMs, encompassing multilingual, monolingual, speaker recognition and music FMs for EFD. Our experiments shows that multilingual SFMs outperforms other FMs in both in-domain and out-domain evaluations. Our work marks the first-ever out-domain assessment of SOTA FMs for EFD also the first study exploring out-domain EFD. • We introduce a novel framework, THAMA for the fusion of FMs. With THAMA , through synchronization of multilingual SFMs, we achieve the best performance in comparison to individual FMs, baseline fusion techniques and setting new SOTA for EFD compared to previous SOTA works. We will open-source the models and codes after the double-blind review for future research to build upon our work. 2 Foundation Models In this section, we present the SOTA FMs used in our study. Speech Foundation Models (SFMs) : For monolingual SFMs, we consider WavLM 1 1 1 https://huggingface.co/microsoft/wavlm-base [ 17 ] , Unispeech-SAT 2 2 2 https://huggingface.co/microsoft/unispeech-sat-base [ 18 ] , Wav2vec2 3 3 3 https://huggingface.co/facebook/wav2vec2-base [ 19 ] , and HuBERT 4 4 4 https://huggingface.co/facebook/hubert-base-ls960 [ 20 ] . WavLM and Unispeech-SAT are SOTA SFMs in SUPERB benchmark. WavLM is trained to solve speech denoising together with masked modeling whereas Unispeech-SAT is trained in a speaker-aware multi-task learning format. Wav2vec2 solves a contrastive learning objective and HuBERT solves BERT-like masked prediction loss. We make use of base versions of WavLM, Unispeech-SAT, wav2vec2, HuBERT with 94.70M, 94.68M, 95.04M, 94.68M parameters and all of them are pre-trained on librispeech 960 hours english data. For multilingual SFMs, we leverage XLS-R 5 5 5 https://huggingface.co/facebook/wav2vec2-xls-r-300m [ 21 ] , Whisper 6 6 6 https://huggingface.co/openai/whisper-base [ 22 ] , and MMS 7 7 7 https://huggingface.co/facebook/mms-1b [ 23 ] . XLS-R was pre-trained on 128 languages, Whisper on 96 and MMS extended it to above 1400 languages. XLS-R and MMS follows wav2vec2 architecture and Whisper is built as a vanilla transformer encoder-decorder architecture. Whisper is trained in weakly-supervised manner and we use the base version with 74M parameters. For XLS-R and MMS, we use 300M and 1B parameters versions respectively. Further, we also include x-vector 8 8 8 https://huggingface.co/speechbrain/spkrec-xvect-voxceleb [ 24 ] , a speaker recognition SFM to our experiments. We believe it can be useful as it has shown its effectiveness in related areas such as synthetic speech detection [ 16 ] and speech emotion recognition [ 25 ] . x-vector is a time-delay neural network

Read the complete text