A General Model for Deepfake Speech Detection: Diverse Bonafide Resources or Diverse AI-Based Generators

Lam Pham, Khoi Vu, Dat Tran, David Fischinger, Simon Freitter, Marcel Hasenbalg, Davide Antonutti, Alexander Schindler, Martin Boyer, Ian McLoughlin · 2026

Read A General Model for Deepfake Speech Detection: Diverse Bonafide Resources or Diverse AI-Based Generators by Lam Pham, Khoi Vu, Dat Tran, David Fischinger, Simon Freitter, Marcel Hasenbalg, Davide Antonutti, Alexander Schindler, Martin Boyer, Ian McLoughlin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27557.txt.

Introduction

Pham and co-authors ask what makes a fake-speech detector work broadly. Two factors matter: real speech sources and fake generators. A model trained on skewed mixes fails on new sets. The team first builds a baseline detector. Tests show how each factor moves the decision threshold. They then build a balanced dataset from public sets. Models trained on it score well across benchmarks. Balance, they conclude, is the key to general detection.

Catalog record

CREATORLam Pham, Khoi Vu, Dat Tran, David Fischinger, Simon Freitter, Marcel Hasenbalg, Davide Antonutti, Alexander Schindler, Martin Boyer, Ian McLoughlin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22052
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A General Model for Deepfake Speech Detection: Diverse Bonafide Resources or Diverse AI-Based Generators Lam Pham, Khoi Vu, Dat Tran, David Fischinger, Simon Freitter, Marcel Hasenbalg, Davide Antonutti, Alexander Schindler, Martin Boyer, Ian McLoughlin Abstract In this paper, we analyze two main factors of Bonafide Resource (BR) or AI-based Generator (AG) which affect the performance and the generality of a Deepfake Speech Detection (DSD) model. To this end, we first propose a deep-learning based model, referred to as the baseline. Then, we conducted experiments on the baseline by which we indicate how Bonafide Resource (BR) and AI-based Generator (AG) factors affect the threshold score used to detect fake or bonafide input audio in the inference process. Given the experimental results, a dataset, which re-uses public Deepfake Speech Detection (DSD) datasets and shows a balance between Bonafide Resource (BR) or AI-based Generator (AG), is proposed. We then train various deep-learning based models on the proposed dataset and conduct cross-dataset evaluation on different benchmark datasets. The cross-dataset evaluation results prove that the balance of Bonafide Resources (BR) and AI-based Generators (AG) is the key factor to train and achieve a general Deepfake Speech Detection (DSD) model. I INTRODUCTION Over the past few years, generative AI presents an impressive development due to remarkable advancements in deep-learning techniques. Regarding applying generative AI in audio domain, especially focusing on human speech, a wide range of high-quality applications such as media production, AI-based music composer, audio-based chatbot, etc. have been introduced and used popularly. However, this also poses potential risks when generated human speech by AI-based generators is used for criminal purposes. To address this concern, the audio forensic task of Deepfake Speech Detection (DSD) has been raised and drawn much attention from the audio research community. Indeed, various public datasets from scientific papers and challenge competitions have been proposed for the DSD task recently (i.e., The survey paper [ 19 ] indicates that four DSD datasets each year have been published from 2021). Given the public datasets, a wide range of deep-learning based models have been proposed to detect the deepfake audio, referred to as DSD models. The state-of-the-art DSD models [ 16 , 6 , 10 ] have leveraged pre-trained deep neural networks which were trained on large-scale datasets of human speech for the up-stream task of Speech-to-Text. Then, these pre-trained models are fine-tuned on the down-stream task of DSD. In particular, authors in [ 16 ] achieved the best performance by fintuning XLSR model, a variant of Wave2Vec2 [ 7 ] released by Meta. Meanwhile, authors in [ 6 ] leveraged and finetuned WavLM [ 4 ] released by Microsoft. The authors in [ 10 ] made effort to benchmark a wide range of popular pre-trained models proposed for audio tasks. However, the state-of-the-art DSD models present two main concerns. First, these DSD models use Equal Error Rate (EER) with free threshold score as the main metric to compare the performance among proposed models. The EER scores are reported and treated as individual values as they are computed from different testing datasets rather than on all testing datasets. For each EER value obtained from a specific testing dataset, there is a corresponding threshold score which is used to detect fake or bonafile audio. These threshold scores may be significantly different from different testing datasets. However, only one threshold score must be selected to detect fake or bonafile audio from an unseen data in the inference process. Therefore, if the threshold scores are much different from different testing datasets, it leads the overfitting issue and the DSD model is not general. In other words, proposed high-performance DSD models with low EER values evaluated on individual testing datasets may be not really reliable if the corresponding threshold scores are not presented, compared, and analyzed. However, the state-of-the-art DSD systems have not reported or analyzed threshold scores across testing datasets. Fig. 1: The proposed baseline architecture Second, during training a DSD model, Softmax-based layer is normally used to separate fake and bonafide audio into two distributions with the boundary probability of 0.5. However, the unbalance issues of Bonafide Resource (BR) or AI-based Generators (AG) may lead the fake or bonafide distributions to mainly locate at the boundary (e.g., around the probability of 0.5). Therefore, increasing or decreasing the probability boundary (e.g., upper or lower than 0.5) helps to find the optimized boundary probability from which the EER and the corresponding threshold score are computed. However, none of research has analyzed the role of Bonafide Resource (BR) or AI-based Generators (AG) which may significantly affect the final probability boundary. By addressing these concerns, we make the following main contributions in this paper: • We conduct extensive experiments to indicate the roles of Bonafide Resource (BR) and AI-based Generator (AG) which affect a DSD model performance, the EER metric and the corresponding threshold score. • Given the experimental results, we propose a dataset for DSD task which re-uses public DSD datasets and shows a balance between Bonafide Resource (BR) and AI-based Generator (AG). • With the proposed dataset, we train various deep-learning based model for DSD task. The proposed models proves general when we conduct cross-dataset evaluation on different benchmark datasets and using a fixed threshold. II Evaluation of Bonafide Resource and AI-Based Generator Factors II-A Proposed Baseline Model TABLE I: Public and Benchmark Datasets Proposed for Deepfake Speech Detection (DSD) in English Datasets Years Speakers Utt. No. Fake speech Real Speech (Male/Female) (Bonafide/Fake) Generators Resources ASVspoof 2015 [ 29 ] (audio) 2015 45/61 16,651/246,500 10 (7 VC, 3 TTS) Speaker Volunteers FoR [ 23 ] (audio) 2019 140 -/195541 7 TTS Kaggle [ 2 ] ASVspoof 2019 (LA task) [ 27 ] (audio) 2019 46/61 12,483/108,978 19 (8VC, 11 TTS) Speaker Volunteers DFDC [ 9 ] (video) 2020 3426 128,154/104,500 1 TTS Speaker Volunteers ASVspoof 2021 (LA task) [ 30 ] (audio) 2021 21/27 18,452/163,114 13 TTS/VC Speaker Volunteers ASVspoof 2021 (DF task) [ 30 ] (audio) 2021 21/27 22,617/589,212 100+ TTS/VC Speaker Volunteers WaveFake [ 11 ] (audio) 2021 0/2 -/117,985 6 TTS LJSPEECH [ 12 ] , JSUT [ 24 ] FakeAVCeleb [ 13 ] (video) 2022 250/250 570/25,000 2 TTS Vox-Celeb2 [ 5 ] In-the-Wild [ 14 ] (video) 2022 58 19963/11816 n/a Self-collected Voc.v [ 28 ] (audio) 2023 46/61 14,250/41,280 5 TTS ASVspoof 2019 LibriSeVoc [ 25 ] (audio) 2023 n/a 13,201/79,206 6 TTS/VC Librispeech [ 17 ] AV-Deepfake1M [ 3 , 1 ] (video) 2023 2,068 286,721/860,039 2 TTS Voxceleb2 [ 5 ] MLAAD [ 15 ] (audio) 2024 n/a -/76,000 54 TTS Librispeech [ 17 ] ASVspoof5 [ 26 ] (audio) 2024 n/a 188,819/815,262 28 (15 TTS, 6 VC, 7 AT) Librispeech [ 17 ] To evaluate the role of Bonafide Resource (BR) and AI-Based Generator (AG) to affect the DSD model performance, we first propose a deep-learning based model which is referred to as the baseline in this paper. Inspired by leveraging pre-trained models [ 6 , 10 , 16 ] from the state-of-the-art, we also use the pre-trained WavLM [ 4 ] model to construct the baseline. In particular, the baseline is shown in Fig. 1 . First, 4-second audio segments are fed into the pre-trained WavLM [

Read the complete text