Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System

Hashim Ali, Surya Subramani, Lekha Bollinani, Nithin Sai Adupa, Sali El-Loh, Hafiz Malik · 2025

Read Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System by Hashim Ali, Surya Subramani, Lekha Bollinani, Nithin Sai Adupa, Sali El-Loh, Hafiz Malik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20983.txt.

Introduction

Hashim Ali and co-authors build a system to detect fake speech for a challenge that spans many languages and tricks. Synthetic voices can be compressed or laundered to hide the traces detectors look for. The challenge tests clean audio, processed audio, and laundered audio as separate tasks.

Their system pairs a self-taught speech front end with a proven detection back end and a noise-adding training trick. It learned from a large multilingual set built from many text-to-speech systems and public sources. After wide tests of front ends, data mixes, and clip lengths, the system placed second in the clean and laundered tasks, showing strong robustness.

Catalog record

CREATORHashim Ali, Surya Subramani, Lekha Bollinani, Nithin Sai Adupa, Sali El-Loh, Hafiz Malik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System Hashim Ali, Surya Subramani, Lekha Bollinani, Nithin Sai Adupa, Sali El-Loh, Hafiz Malik Abstract The SAFE Challenge evaluates synthetic speech detection across three tasks: unmodified audio, processed audio with compression artifacts, and laundered audio designed to evade detection. We systematically explore self-supervised learning (SSL) front-ends, training data compositions, and audio length configurations for robust deepfake detection. Our AASIST-based approach incorporates WavLM large frontend with RawBoost augmentation, trained on a multilingual dataset of 256,600 samples spanning 9 languages and over 70 TTS systems from CodecFake, MLAAD v5, SpoofCeleb, Famous Figures, and MAILABS. Through extensive experimentation with different SSL front-ends, three training data versions, and two audio lengths, we achieved second place in both Task 1 (unmodified audio detection) and Task 3 (laundered audio detection), demonstrating strong generalization and robustness. I Introduction The rapid advancement of text-to-speech (TTS) synthesis technologies has created an urgent need for robust audio deepfake detection systems. Although these technologies offer beneficial applications, their misuse for creating convincing fake audio poses significant security and societal risks. The SAFE (Synthetic Audio Forensics Evaluation) Challenge 1 1 1 https://stresearch.github.io/SAFE/ represents a critical step forward in addressing these challenges by providing a comprehensive evaluation framework that emphasizes real-world applicability and generalizability across diverse audio sources. Unlike traditional evaluation protocols that often focus on controlled laboratory conditions, the SAFE Challenge is designed to test detection systems in realistic scenarios where spoofing attacks may originate from unknown synthesis methods, undergo various processing operations, or be deliberately laundered to evade detection. This emphasis on generalizability across diverse sources reflects the practical challenges faced by deepfake detection systems deployed in real-world environments, where the characteristics of spoofed audio can vary dramatically from training data. A significant limitation of current deepfake detection research is the reliance on single, often clean datasets for training. Most existing systems are trained on well-curated datasets such as ASVspoof [ 1 , 2 , 3 ] , which, while valuable for controlled evaluation, may not adequately prepare models for the diversity of spoofing techniques and audio characteristics encountered in practice. This training paradigm can lead to models that perform well on in-domain test sets but fail to generalize to new attack types, different languages, or varying audio quality conditions. To address this generalization challenge, we conducted four iterative experiments that systematically explored dataset integration strategies for training robust audio deepfake detection systems. Our approach progressively incorporated multiple datasets that span different languages, spoofing techniques, and audio quality conditions, while also investigating the impact of factors such as audio segment length and SSL (self-supervised learning) front-end model selection. These experiments were designed to understand how diverse multilingual datasets can improve the robustness of detection systems when faced with unknown spoofing attacks. We evaluated our approach on both the SAFE Challenge tasks and the In-The-Wild (ITW) dataset [ 4 ] , a community-standard benchmark to assess generalized performance of audio deepfake detection systems. Our work makes several practical contributions to the audio deepfake detection community. First, we provide an empirical evaluation of how the combination of datasets affects detection performance across the three SAFE Challenge tasks and on the ITW benchmark [ 4 ] . Second, we analyze performance patterns that reveal insights into task-specific challenges, particularly the difficulty of detecting laundered audio. Third, we demonstrate the value of incorporating diverse multilingual training data for improving cross-domain generalization capabilities. Finally, we performed a source-level analysis for both generated and pristine (authentic) sources from the SAFE Challenge, revealing critical vulnerabilities and failure patterns across different synthesis methods and audio processing scenarios. These findings offer actionable insights for researchers and practitioners working to develop more robust audio deepfake detection systems. The remainder of this paper is organized as follows. Section II reviews existing audio deepfake detection datasets and self-supervised learning approaches relevant to spoofing detection. Section III describes the SAFE Challenge framework and evaluation protocol. Section IV presents our model architecture and SSL front-end selection methodology. Section V details our dataset integration strategies and systematic experimental design. Section VI covers training configurations and implementation details. Section VII provides comprehensive results and analysis from both SAFE Challenge and ITW evaluations. Finally, Section VIII concludes with implications for the deepfake detection community. II Related Work II-A Audio Deepfake Detection Datasets The development of robust audio deepfake detection systems has been driven by the availability of diverse evaluation datasets, each addressing specific aspects of the spoofing detection challenge. ASVspoof Series: The ASVspoof Challenges [ 2 , 5 , 6 , 7 ] have provided foundational datasets for the community. ASVspoof 2015 [ 8 ] introduced the first spoofing database with 10 TTS systems. ASVspoof 2019 [ 2 ] LA expanded this with 19 TTS systems (6 known, 13 unknown), including WaveNet, Tacotron2, and traditional approaches, establishing the standard for controlled evaluation. ASVspoof 2021 [ 6 ] introduced two tracks: Logical Access (LA) with 13 systems and Deepfake (DF) with more than 100 attack methods, significantly expanding the attack diversity. The recent ASVspoof 5 [ 7 ] represents the most complex design with multilingual data and a wide variety of attacks on a large scale. Multilingual and Cross-Domain Datasets: MLAAD (Multi-Language Audio Anti-Spoofing Dataset) [ 9 ] addresses linguistic diversity with 91 TTS systems across 38 languages, providing 420.7 hours of synthetic speech from 42 different architectures. This dataset specifically targets the language bias present in predominantly English-focused datasets. MLAAD dataset only provides the synthetic audio samples. The M-AILABS Speech Dataset 2 2 2 https://github.com/imdatceleste/m-ailabs-dataset complements this by providing authentic multilingual speech samples across multiple languages, sourced from audiobooks and public figures speeches. Real-World and Noisy Conditions: SpoofCeleb [ 10 ] leverages VoxCeleb1 [ 11 ] as source data, training 23 contemporary TTS systems in real-world noisy conditions to bridge the gap between clean laboratory data and practical applications. In-The-Wild (ITW) [ 4 ] provides audio deepfakes collected from social media platforms, offering genuine “in-the-wild” evaluation conditions. Hashim et al. [ 12 ] applied various real-world processing on ASVSpoof19 audio to generate laundered version of the dataset. Technology-Specific Datasets: CodecFake [ 13 ] introduces the first dataset focused specifically on neural codec-based synthesis, featuring 15 codec models from 6 frameworks, including SpeechTokenizer [ 14 ] , Encodec [ 15 ] , and novel approaches like FunCodec [ 16 ] . The DFADD dataset [ 17 ] focuses on diffusion- and flow-matching-based TTS systems, including GradTTS [ 18 ] , NaturalSpeech2 [ 19 ] , Style-TTS2 [ 20 ] , Matcha-TTS [ 21 ] , PFlow-TTS [ 22 ] , etc. Famous Figures Dataset [ 23 ] : Motivated by recent incidents such as the Biden robocall [ 24 ] , and fabricated recordings of London Mayor Sadiq Khan making inflammatory remarks [ 25 ] , we curated a specialized dataset for protecting famous figures from voice cloning attacks. This dataset was developed by collecting high-quality bonafide speech samples of famous figures from YouTube, and then generating the corresponding synthetic speech using various TTS approaches. This dataset provides a deep coverage of political personalities using cutting-edge 2024-2025 TTS systems, such as StyleTTS2 [ 20 ] , XTTSv2 [ 26 ] , F5TTS [ 27 ] , E2TTS

Read the complete text