Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes
Neta, David, Idan, Sharon, Ethan · 2025
Read Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes by Neta, David, Idan, Sharon, Ethan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23619.txt.
Introduction
Neta Glazer, David Chernin, Idan Achituve, Sharon Gannot, and Ethan Fetaya adapt fake-voice detectors with very little data in this 2025 paper. Voice cloning keeps improving, so detectors must quickly learn each new generator from only a few samples. Their framework pairs a strong sound embedding model with a flexible statistical classifier.
The classifier, based on Gaussian processes, adjusts smoothly to new fake types. The team shows the pairing gives strong detection with minimal examples. It also supports personal detection tuned to one speaker, staying robust against fresh cloning tools and adapting from a single sample. A new benchmark built from current cloning models supports their tests.
Catalog record
| CREATOR | Neta, David, Idan, Sharon, Ethan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes Neta, David, Idan, Sharon, Ethan Abstract Recent advancements in Text-to-Speech (TTS) models, particularly in voice cloning, have intensified the demand for adaptable and efficient deepfake detection methods. As TTS systems continue to evolve, detection models must be able to efficiently adapt to previously unseen generation models with minimal data. This paper introduces ADD-GP, a few-shot adaptive framework based on a Gaussian Process (GP) classifier for Audio Deepfake Detection (ADD). We show how the combination of a powerful deep embedding model with the Gaussian processes flexibility can achieve strong performance and adaptability. Additionally, we show this approach can also be used for personalized detection, with greater robustness to new TTS models and one-shot adaptability. To support our evaluation, a benchmark dataset is constructed for this task using new state-of-the-art voice cloning models. Glazer Chernin Achituve Gannot Fetaya Bar Ilan UniversityIsrael keywords Audio Deepfake Detection, Few-Shot Adaptation, Gaussian Processes † † email: [email protected], [email protected] 1 Introduction The rapid advancements in text-to-speech (TTS) generation have significantly improved the ability to synthesize highly realistic speech from textual input [ 1 , 2 ] . Recent models also excel in voice cloning, enabling the generation of speech that closely mimics a specific speaker’s voice with just a single reference utterance [ 3 , 4 ] . This breakthrough has significantly enhanced applications in accessibility, entertainment, and human-computer interaction. However, these advancements also introduce significant security and ethical concerns, particularly in the realm of voice cloning. With only a short audio sample, modern voice cloning models can generate speech that is nearly indistinguishable from the original speaker’s voice. This capability has been exploited in various malicious activities, including fraud, misinformation, and identity theft. For instance, fraudsters have successfully mimicked corporate executives to authorize fraudulent financial transactions, leading to substantial financial losses. Similarly, scammers have used synthetic voices to pose as family members in distress, coercing victims into transferring money under false pretenses [ 5 ] . This problem is addressed by Audio Deepfake Detection (ADD) algorithms. We observed that current ADD detectors easily detect deepfakes from known TTS models. However, they may generalize poorly to deepfakes from unseen generation models. This is a significant security concern as new models are introduced at an accelerating rate. Moreover, advancement in parameter efficient fine-tuning, e.g., LoRA [ 6 ] , allows attackers to modify an existing TTS model on a relatively small amount of data. To address this challenge, a detection system must be capable of efficiently adapting to novel deepfake generation methods with minimal data, allowing it to continuously evolve against new threats. We also note that many leading commercial TTS models, such as 11Labs, are only accessible via paid APIs. This limits the feasibility of large-scale data collection for retraining, further emphasizing the need for efficient few-shot adaptation. In this paper, we introduce a new system for adaptive deepfake detection based on Gaussian Processes (GP), which we name ADD-GP. We use a Deep Kernel Learning (DKL) framework, based on XLS-R features [ 7 ] , to improve the generalization capability of detection models. Gaussian Processes (GPs) are well-suited for few-shot detection due to their non-parametric nature and ability to provide uncertainty estimates. By integrating GPs with deep neural networks through Deep Kernel Learning (DKL), we enable the model to learn highly expressive feature representations while maintaining the flexibility and robustness of GPs. Finally, we explore the use of our framework for personalized deepfake detection. This shows improved robustness to new TTS models and better adaptability even in cases where only a single generated example is available. Our key contributions include: • A novel method for audio deepfake detection, enabling effective few-shot adaptation for new TTSs. • A state-of-the-art personalized few-shot deepfake detection with greater robustness and adaptability. • The introduction of LibriFake , a dataset specifically designed for evaluating few-shot and one-shot deepfake audio detection approaches on novel TTS models. To support future research and ensure the reproducibility of our results, we have made both our source code and datasets publicly available at: https://github.com/NetaGlazer/ADD-GP 2 Background We provide a brief introduction to Gaussian Process (GP) classifiers, which form the foundation of our proposed method. Scalars are denoted with lowercase letters (e.g., x ), vectors with boldface lowercase letters (e.g., \mathbf{x} ), and matrices with boldface uppercase letters (e.g., \mathbf{X} ). In general, \mathbf{y}=[y_{1},...,y_{N}]^{T} represents the vector of labels, and \mathbf{X}\in\mathbb{R}^{N\times d} represents the design matrix comprising N data points of dimension d , where the i -th row is \mathbf{x}_{i} . 2.1 Gaussian Processes Gaussian Processes (GPs) define a distribution over functions. A GP model is fully specified by a mean function m({\mathbf{x}}) and a covariance function (kernel) k({\mathbf{x}},{\mathbf{x}}^{\prime}) . We say that {\mathbf{f}}\sim\mathcal{GP}(m({\mathbf{x}}),k({\mathbf{x}},{\mathbf{x}}^{\prime})) if for a given dataset \mathbf{X} , the function values \mathbf{f}=[f({\mathbf{x}}_{1}),...,f({\mathbf{x}}_{N})]^{T} are normally distributed, i.e., \mathbf{f}\sim\mathcal{N}(\bm{\mu},\mathbf{K}) , where \bm{\mathbf{\mu}}_{i}=m({\mathbf{x}}_{i}) and {\mathbf{K}}_{ij}=k({\mathbf{x}}_{i},{\mathbf{x}}_{j}) [ 8 ] . Commonly, the mean function is the constant zero function. The predictive power of GPs mainly comes from the kernel which encodes similarity between the data points. For regression tasks if the likelihood p({\mathbf{y}}|{\mathbf{f}},\mathbf{X}) takes a Gaussian form, the posterior distribution p({\mathbf{f}}|{\mathbf{y}},\mathbf{X}) is a Gaussian with a closed-form formula for the mean and variance. At inference, given a new data point {\mathbf{x}}_{*} , the predictive distribution p({\mathbf{f}}_{*}|{\mathbf{x}}_{*},{\mathbf{y}},\mathbf{X})=\int p({\mathbf{f}}_{*}|{\mathbf{x}}_{*},{\mathbf{f}})p({\mathbf{f}}|{\mathbf{y}},\mathbf{X})d{\mathbf{f}} is also a Gaussian with parameters given by: \displaystyle p({\mathbf{f}}_{*}|{\mathbf{x}}_{*},{\mathbf{y}},\mathbf{X})=\mathcal{N}(\mu_{*},\sigma_{*}^{2}) (1) \displaystyle\mu_{*}={\mathbf{k}}_{*}^{T}(\mathbf{K}+\sigma^{2}\mathbf{I})^{-1}{\mathbf{y}} (2) \displaystyle\sigma_{*}=k_{**}-{\mathbf{k}}_{*}^{T}(\mathbf{K}+\sigma^{2}\mathbf{I})^{-1}{\mathbf{k}}_{*} (3) where k_{**}=k({\mathbf{x}}_{*},{\mathbf{x}}_{*}) , and {\mathbf{k}}_{*}[i]=k({\mathbf{x}}_{i},{\mathbf{x}}_{*}) . This means that given a kernel and a dataset, the GP prediction is given by a closed-form solution without any optimization. 2.2 Gaussian Process Classification (GPC) Unlike GP regression with a Gaussian likelihood, in Gaussian Process Classification (GPC), the likelihood p(\mathbf{y}|\mathbf{f},{\mathbf{X}}) is non-Gaussian, making exact posterior inference intractable. To address this, approximate inference techniques, such as variational inference (VI) [ 9 , 10 , 11 ] , or Markov chain Monte Carlo (MCMC) [ 12 , 13 , 14 , 15 ] , are commonly used. Here, we follow the approach presented in [ 16 ] , although our approach can be combined with any standard GP classifier. It is important to note that this approach still uses a closed-form solution for the posterior and at inference. 2.3 Deep Kernel Learning (DKL) The performance of GP models is heavily dependent on the choice of kernel function k({\mathbf{x}}_{i},{\mathbf{x}}_{j}) with standard kernels often failing to capture meaningful semantic similarities in complex data modalities such as images and speech. Deep Kernel Learning (DKL) addresses this limitation by combining classical kernels with a neural network transformation [ 17 , 18 ] . For example, it is common to use the RBF kernel on features computed by a deep network g_{\theta} : k_{\theta}({\mathbf{x}}_{i},{\mathbf{x}}_{j})=\sigma\cdot\exp\left(-\frac{||g_{\theta}({\mathbf{x}}_{i})-g_{\theta}({\mathbf{x}}_{j})||^{2}}{2\ell^{2}}\right). (4) Here, the kernel parameters are (1) the neural network parameters \theta , (2) the length scale \ell and (3) the output scale \sigma . Typically, all the parameters are learned using a Bayesian loss, such as the log marginal likelihood or the evidence lower bound (ELBO). 2.4 Audio Deepfake Detection Audio Deepfake Detection (ADD) methods generally follow two main architectures: pipeline-based and end-to-end (E2E) based approach. In pipeline-based models [ 19 ] , the system is split into two distinct components: a front-end feature extractor and a back-end classifier [ 20 ] .