Frustratingly Easy Zero-Day Audio DeepFake Detection via Retrieval Augmentation and Profile Matching
2025
Read Frustratingly Easy Zero-Day Audio DeepFake Detection via Retrieval Augmentation and Profile Matching (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21728.txt.
Introduction
Audio deepfake detectors trained on fixed data go stale as synthesis methods evolve. Xin Wang and 3 co-authors test a retrieval augmented detector in a 2025 paper. At inference, the detector queries an external knowledge database, much like retrieval augmented generation in large language models. The database can update continuously during operation. When that helps, and when it does not, was not well characterized before.
That added lookup helps most when database content differs from the fine tuning data of the underlying self supervised model. The content must still align with the test domain. Purely model only baselines win in other conditions. Knowledge cutoff issues in detection motivate the design. The study maps optimal use scenarios for external sources in zero day deepfake detection. Fresh external data, used in the right conditions, can extend a detector life past its training cut off. It stays live as new fakes appear.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21728
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Frustratingly Easy Zero-Day Audio DeepFake Detection via Retrieval Augmentation and Profile Matching Abstract Modern audio deepfake detectors using foundation models and large training datasets have achieved promising detection performance. However, they struggle with zero-day attacks , where the audio samples are generated by novel synthesis methods that models have not seen from reigning training data. Conventional approaches against such attacks require fine-tuning the detectors, which can be problematic when prompt response is required. This study introduces a training-free framework for zero-day audio deepfake detection based on knowledge representations, retrieval augmentation, and voice profile matching. Based on the framework, we propose simple yet effective knowledge retrieval and ensemble methods that achieve performance comparable to fine-tuned models on DeepFake-Eval-2024, without any additional model-wise training. We also conduct ablation studies on retrieval pool size and voice profile attributes, validating their relevance to the system efficacy. 1 1 1 This study is supported by JST AIP Acceleration Research (JPMJCR24U3) and the New Energy and Industrial Technology Development Organization (NEDO, JPNP22007). This study was partially carried out using the TSUBAME4.0 supercomputer at the Institute of Science Tokyo. document 1 Introduction With the advancement of deep neural networks (DNNs), modern audio deepfake detection (ADD) systems, or so-called spoofing countermeasures (CM), have achieved strong performance in both controlled and complex scenarios. Such progress is demonstrated through established evaluation frameworks like the evolving ASVspoof challenge series [ 1 , 2 , 3 ] , as well as recent datasets including In-the-Wild [ 4 ] , SpoofCeleb [ 5 ] , and DeepFake-Eval-2024 (DE2024) [ 6 ] . Meanwhile, recent advances in self-supervised learning (SSL) audio feature extractors [ 7 , 8 ] have significantly improved CM detection performance on these datasets, particularly when combined with backend classifiers such as AASIST [ 9 , 10 ] and simple multi-layer perceptrons [ 11 , 12 ] . Figure 1 : Demonstration of the vulnerability of deepfake detection models caused by the time lag between the zero-day attack period and model update. The time progression indicates natural time flow. However, current CMs primarily focus on empirical performance. Although they can be updated through fine-tuning, it demands significant computational resources and time, and thus often cause considerable delay in adapting to newly-emerged attacking algorithms, as illustrated in Figure 1 . These algorithms, represented by not only research outputs, but also commercial services like ElevenLabs 2 2 2 https://elevenlabs.io , may not be represented in currently-rolling CM training data during that period . Thus, they are called zero-day attacks and should ideally be addressed promptly without fine-tuning or additional training. This need for training-free ADD methodologies motivates the present study. We address the research question: Can we prevent zero-day spoofing attacks without any form of additional fine-tuning? We propose a simple yet comprehensive framework for handling unseen zero-day attacks using retrieval augmentation (RA) techniques. Our method maintains a knowledge base containing reference “seen” data, and addresses novel attacks through knowledge retrieval and ensemble methods based on k-nearest neighbors (KNN). The knowledge database stores both representation-level features and score-level predictions from a well-trained SSL-based CM. Additionally, we incorporate speaker and voice-related information to evaluate whether such profile matching can enhance the performance. Both the methods and their related models do not involve any additional re-training or fine-tuning, enabling prompt reaction and adaptation for robustness to the zero-day attacks. While RA has been applied to ADD, existing methods significantly differ from our approach. [ 13 ] maintains a vector data pool and retrieves the vectors of the most closely-matched samples during inference to augment the detection process. [ 14 ] also uses KNN to select features for fine-tuning the backend classifier. However, apart from fine-tuning the SSL frontend for ADD, both work involve additional training of backend classifier for feature selection and decision making. Our proposed method employs RA by directly using existing massively-trained CMs, enabling more broadly applicable solution for advanced SSL-based CMs. Figure 2 : Outline of our methodology. The knowledge source data contains input recordings and ground truth binary labels. The lock symbols indicate that both the CM and profile feature extractors remain frozen without fine-tuning. Retrieved vectors can be either CM features, profile features, or both, depending on the strategy detailed in Section 2.3 . Example number of retrieved utterances is k=4 in this outline. 2 Retrieval Augmentation Based on Knowledge Resources 2.1 Source data: DeepFake-Eval-2024 We select DE2024 [ 6 ] as our source dataset for this study, as a representative of real-world deepfake threats with current generation techniques due to its sourcing methodology. DE2024 is a multi-modal dataset containing 56.5 hours of audio content collected from social media platforms and deepfake detection services in 2024. It includes over 40 languages from more than 80 web sources, with audio sampled at 44.66 kHz by default. In order to adapt and perform a fair comparison with the existing benchmark and up-to-date, widely used CMs, we implement the following adaptation schemes on DE2024: 1) Following the fine-tuning setup in [ 6 ] , 60% and 40% of the original dataset is used for training and evaluation, respectively. Among the training data, we randomly picked 5% of them as a development set. The ratio of real to fake audios across all splits is approximately 39% vs. 61%, with minor variations observed across different durations; 2) To compare with the established baseline 3 3 3 https://huggingface.co/nii-yamagishilab/wav2vec-large-anti-deepfake-nda , whose pre-trained SSL-based CM is acquired in this study, we segment both training and evaluation partitions into various lengths ranging from 4 seconds (as reported in [ 6 ] ) to 50 seconds. The training partition is used to construct a knowledge base, maintaining the same real-to-fake data ratio. The evaluation partition serves as the query data, containing zero-day attacks. They are detailed in next subsection. 2.2 Knowledge Database Construction As shown in the upper part of Fig. 2 , our methodology utilizes a knowledge database to enhance both classification accuracy and feature representation for query recordings that may contain zero-day audio deepfake attacks. Let \mathcal{K} denote a knowledge database of N labeled utterances. For each i\in\mathcal{K} , we have the following information extracted and stored. • \mathbf{x}_{i,cm} : CM feature vector. This vector is extracted from the SSL-based CM system, specifically from the output of the SSL frontend (e.g., Wav2Vec2 [ 15 ] ). The frame-level features are averaged across all frames to produce a single utterance-level representation vector. • \mathbf{x}_{i,prof} : Profile feature vector. This vector captures multiple voice and speaker attributes using well-trained neural network models. Various attributes including age, gender, emotion, and voice quality such as pitch and rhythm, are inferred and concatenated to form a comprehensive profile feature vector that describes different aspects of the voice and speaker identity. • y_{i}\in\{0,1\} : This is ground truth binary label indicating whether the utterance is real (0) or fake (1). These labels are manually annotated and provided in the source metadata. • s_{i}\in(0,1) : This is the prediction score for knowledge ensemble. The scalar value is also obtained from the CM, where the SSL feature vector is passed through the linear layer of the CM, followed by a softmax to produce the score. The detailed description of the SSL-based CM system and the profile