Towards Data Drift Monitoring for Speech Deepfake Detection in the context of MLOps
Xin Wang, Wanying Ge, Junichi Yamagishi · 2025
Read Towards Data Drift Monitoring for Speech Deepfake Detection in the context of MLOps by Xin Wang, Wanying Ge, Junichi Yamagishi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10086.txt.
Introduction
Xin Wang and 2 co-authors ask whether a deployed speech deepfake detector can notice the world drifting away from its training data in a 2025 machine learning operations paper. A frozen detector weakens as new synthesis engines appear, and nobody may realise until fakes slip through. Watching for drift, a widening gap between fresh traffic and the reference data, could raise the alarm early enough to act.
On a small toy set and on the large MLAAD collection, distance measures between data distributions did flag drift caused by fresh text-to-speech attacks. Better still, fine-tuning the detector on examples from the new generators shrank both the measured drift and the detection error rate. Maintenance, not just accuracy on day one, becomes the paper's measure of a detector.
Catalog record
| CREATOR | Xin Wang, Wanying Ge, Junichi Yamagishi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Data Drift Monitoring for Speech Deepfake Detection in the context of MLOps Xin Wang, Wanying Ge, Junichi Yamagishi Abstract When being delivered in applications or services on the cloud, static speech deepfake detectors that are not updated will become vulnerable to newly created speech deepfake attacks. From the perspective of machine learning operations (MLOps), this paper tries to answer whether we can monitor new and unseen speech deepfake data that drifts away from a seen reference data set. We further ask, if drift is detected, whether we can fine-tune the detector using similarly drifted data, reduce the drift, and improve the detection performance. On a toy dataset and the large-scale MLAAD dataset, we show that the drift caused by new text-to-speech (TTS) attacks can be monitored using distances between the distributions of the new data and reference data. Furthermore, we demonstrate that fine-tuning the detector using data generated by the new TTS deepfakes can reduce the drift and the detection error rates. † † address: National Institute of Informatics, Japan 1 Introduction A speech deepfake detector takes an input waveform \bm{x}\in\mathbb{R}^{T} and produces a label y\in\{\text{fake},\text{real}\} indicating whether the input is a deepfake or not. Our community has mainly approached the task from a machine learning (ML) perspective: we learn a parametric distribution \tilde{p}_{\text{trn.}}(\bm{\mathrm{y}}_{\text{trn.}}|\bm{\mathrm{\bm{x}}}_{\text{trn.}},\Theta) given a training set \mathcal{D}_{\text{trn.}}=\{\bm{x}_{\text{trn}}^{(i)},y_{\text{trn}}^{(i)}\}_{i} , where \Theta is the learnable parameter set. We expect good performance of using \tilde{p}_{\text{trn.}} to classify any new test sample {\bm{x}_{\text{test}}} from a test set \mathcal{D}_{\text{test}} if \tilde{p}_{\text{trn.}} well approximates the true distribution {p}_{\text{trn.}}(\bm{\mathrm{y}}_{\text{trn.}}|\bm{\mathrm{\bm{x}}}_{\text{trn.}}) and the training and testing data are independent and identically distributed, or {p}_{\text{trn.}}(\bm{\mathrm{y}}_{\text{trn.}}|\bm{\mathrm{\bm{x}}}_{\text{trn.}})\approx{p}_{\text{test}}(\bm{\mathrm{y}}_{\text{test}}|\bm{\mathrm{\bm{x}}}_{\text{test}}) . In a common laboratory research setup (top panel of Fig. 1 ), we are given a pair of \mathcal{D}_{\text{trn}} and \mathcal{D}_{\text{test}} from benchmarking databases [ 1 , 2 , 3 ] , where \mathcal{D}_{\text{test}} is usually designed to be slightly different from \mathcal{D}_{\text{trn}} . We then iterate over the loop of model training, testing, detector re-designing, and hyper-parameter tuning so that the learned \tilde{p}_{\text{trn.}} better approximates {p}_{\text{trn.}} and generalizes to the mismatched {p}_{\text{test}} of \mathcal{D}_{\text{test}} . This setup accelerates the research iterations, and many effective solutions have been found, e.g., end-to-end deep neural network (DNN) classifiers [ 4 ] or their combination with self-supervised learning (SSL) speech feature extractors [ 5 ] . However, when deploying a deepfake detection application from the perspective of ML-operations (MLOps), we cannot assume that the detector trained on a training set created five years ago will perform well against the latest deepfakes, regardless of how well the detector ‘generalizes’ to the test set paired with the training set in the laboratory research setup. More specifically, in the MLOps setup, we are facing a sequence of (\mathcal{D}_{\text{test}}^{(1)},\mathcal{D}_{\text{test}}^{(2)},\cdots) that may incrementally include deepfakes built upon newly proposed text-to-speech (TTS) or voice conversion (VC) algorithms. These new test sets may gradually become different, or drift away , from the seen data. Monitoring the drift is hence helpful to signify the new and unseen incoming test data. When the drift is signified, we need to update the detector so that the drift can be reduced and the detection error rates will not be severely degraded. In the context of MLOps, we investigate the data drift caused by new deepfake attacks and focus on two research questions: RQ1: Whether drift caused by new TTS attacks can be monitored. RQ2: Whether the drift can be reduced by fine-tuning the detector using similarly drifted data when it is available. For RQ1, we monitor the drift by measuring the distance between the feature distributions of the incoming test data and a reference data (i.e., a development set). We conducted experiments using features extracted from SSL or non-SSL-based speech detectors on multiple datasets, including the constantly evolving MLAAD dataset [ 6 ] . The results demonstrated that new TTS attacks indeed caused higher drift values than earlier ones. For RQ2, we demonstrated on the MLAAD dataset that the drift on the test set can be reduced if the detectors are fine-tuned using data synthesized from the new TTS attacks, and the degree of reduction is affected by the amount of fine-tuning data. Furthermore, fine-tuning reduces the detection error rates. Figure 1 : Illustration of speech deepfake detection in laboratory and MLOps setups. The highlighted blocks are investigated in this paper. As far as we are aware of, this paper is the first to quantify the data drift of speech deepfake detection for MLOps. Other contributions include the comparison of multiple distance metrics over multiple datasets and detectors and the experiment of fine-tuning that demonstrates the reduction of drift and detection errors. 2 Methods 2.1 Definition of the drift Given {p}(\bm{\mathrm{y}}|\bm{\mathrm{\bm{x}}})={p}(\bm{\mathrm{\bm{x}}}|\bm{\mathrm{y}})\cdot{p}(\bm{\mathrm{y}})/{p}(\bm{\mathrm{\bm{x}}}) , 1 1 1 To avoid the notation clutter, we drop the suffix in p_{\text{test}} and use p to denote the distribution unless otherwise stated. the change of {p}(\bm{\mathrm{\bm{x}}}|\bm{\mathrm{y}}) is referred to as concept drift in the previous artificial intelligence era [ 7 ] , wherein the word concept refers to association between the data \bm{x} and its label y . For detecting speech deepfakes, we focus on the concept drift (or simply drift) of the fake data’s conditional distribution {p}(\bm{\mathrm{\bm{x}}}|\bm{\mathrm{y}}=\text{fake}) . The drift may happen when the speech data \bm{x} is generated using new TTS and VC systems. 2.2 Measuring the drift Inspired by existing studies [ 8 , 9 ] , we define the drift of the fake data as a distance D_{\text{t}-\text{r}} between {p}(\bm{\mathrm{\bm{x}}}|\bm{\mathrm{y}}=\text{fake}) of a test set and the distribution {p}_{\text{ref.}}(\bm{\mathrm{\bm{x}}}_{\text{ref.}}|\bm{\mathrm{y}}_{\text{ref.}}=\text{fake}) of a reference dataset \mathcal{D}_{\text{ref}} . This reference data can be \mathcal{D}_{\text{trn.}} , an early version of \mathcal{D}_{\text{test}} , or a dedicated development set (in the case of this paper). Directly estimating \tilde{p}(\bm{\mathrm{\bm{x}}}|\bm{\mathrm{y}},\Theta) is challenging because the waveform data is high-dimensional and varied in length. This is addressed with two approximations. First, we extract a fixed-dimensional embedding vector \bm{a}\in\mathbb{R}^{M} from the waveform \bm{a}=\mathcal{H}_{\Theta}(\bm{x}) , hoping that \tilde{p}(\bm{\mathrm{a}}|\bm{\mathrm{y}}) captures all the information in \tilde{p}(\bm{\mathrm{x}}|\bm{\mathrm{y}}) . The embedding extractor \mathcal{H}_{\Theta} is part of the detector, for example, the SSL-model front end [ 5 ] plus a global average pooling layer. Although the embedding vector is more compact than a waveform (i.e., M\ll T ), it is still challenging to accurately model \tilde{p}(\bm{\mathrm{a}}|\bm{\mathrm{y}}) for an M larger than a few hundred. Hence, the second approximation is to model each dimension of \bm{a} independently. For the i -th dimension {a}_{i} , i\in[1,M] , we estimate the discrete probability mass function (PMF) f (i.e., a histogram of {a}_{i} ) or its cumulative distribution function (CDF) F . The same procedure is applied to estimate the PMF or CDF for the reference data {a}_{\text{ref},i} . Then, for example using the PMFs, we compute the drift distance between the test and reference data as D_{\text{t}-\text{r}}\approx\sum_{i=1}^{M}D\Big({f}({i}),{f}_{\text{ref.}}({i})\Big) , where D is a distance function of two univariate PMFs, and {f}({i}) is the probability of taking the i -th value in the PMF support (i.e., the i -th histogram bin). We compare three representative distance functions [ 9 ] listed in Table 1 . Note that the Kolmogoro-Smirnov (K-S) test was defined from the perspective of a statistical test, even though its implementation is similar to the Wasserstein-1 distance. Table 1 : Investigated distance metrics D_{p-q} for discrete univariate