Anomaly Detection and Localization for Speech Deepfakes via Feature Pyramid Matching
Emma Coletta, Davide Salvi, Viola Negroni, Daniele Ugo Leonzio, Paolo Bestagini · 2025
Read Anomaly Detection and Localization for Speech Deepfakes via Feature Pyramid Matching by Emma Coletta, Davide Salvi, Viola Negroni, Daniele Ugo Leonzio, Paolo Bestagini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18032.txt.
Introduction
Emma Coletta and four co-authors hunt fake speech by learning only what real speech looks like, in a 2025 paper. The catalog title carries stray funding text; the printed title is clean. Voice deepfakes now imitate targets well enough to threaten security.
Supervised detectors stumble on new synthesis tricks and cannot explain themselves. This framework treats fakes as anomalies: train on real voices only, flag the rest. A student-teacher pair compares feature pyramids, and mismatches paint anomaly maps over time and frequency. A discrepancy scaling trick steadies the method on unseen data. Tests beat the baselines considered. The maps show analysts exactly where a clip sounds wrong. Interpretability, the authors argue, is a security feature.
Catalog record
| CREATOR | Emma Coletta, Davide Salvi, Viola Negroni, Daniele Ugo Leonzio, Paolo Bestagini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Anomaly Detection and Localization for Speech Deepfakes via Feature Pyramid Matching Thanks: This work was supported by the FOSTERER project, funded by the Italian Ministry of Education, University, and Research within the PRIN 2022 program. This work was partially supported by the European Union - Next Generation EU under the Italian National Recovery and Resilience Plan (NRRP), Mission 4, Component 2, Investment 1.3, CUP D43C22003080001, partnership on “Telecommunications of the Future” (PE00000001 - program “RESTART”). This work was partially supported by the European Union - Next Generation EU under the Italian National Recovery and Resilience Plan (NRRP), Mission 4, Component 2, Investment 1.3, CUP D43C22003050001, partnership on “SEcurity and RIghts in the CyberSpace” (PE00000014 - program “FF4ALL-SERICS”). Emma Coletta, Davide Salvi, Viola Negroni, Daniele Ugo Leonzio, Paolo Bestagini Abstract The rise of AI-driven generative models has enabled the creation of highly realistic speech deepfakes—synthetic audio signals that can imitate target speakers’ voices—raising critical security concerns. Existing methods for detecting speech deepfakes primarily rely on supervised learning, which suffers from two critical limitations: limited generalization to unseen synthesis techniques and a lack of explainability. In this paper, we address these issues by introducing a novel interpretable one-class detection framework, which reframes speech deepfake detection as an anomaly detection task. Our model is trained exclusively on real speech to characterize its distribution, enabling the classification of out-of-distribution samples as synthetically generated. Additionally, our framework produces interpretable anomaly maps during inference, highlighting anomalous regions across both time and frequency domains. This is done through a Student-Teacher Feature Pyramid Matching system, enhanced with Discrepancy Scaling to improve generalization capabilities across unseen data distributions. Extensive evaluations demonstrate the superior performance of our approach compared to the considered baselines, validating the effectiveness of framing speech deepfake detection as an anomaly detection problem. I Introduction In recent years, the rapid advancement of AI-driven generative systems has made the creation of high-quality synthetic content easier than ever. While the democratization of generative tools brings numerous benefits, it also introduces serious social risks. Malicious actors could exploit these technologies to produce deceptive synthetic media, enabling identity theft, financial fraud, and disinformation campaigns [ 1 ] . Among these threats, speech deepfakes represent a particularly concerning issue in the audio domain. These are synthetically generated speech samples that mimic the voice of a target speaker, making them say arbitrary utterances. Speech deepfakes have already been used in fraud, blackmail, and other security-related threats, prompting the multimedia forensics community to actively develop deepfake detection systems to preserve the integrity of digital communications [ 2 ] . Diverse strategies have been proposed for speech deepfake detection, spanning from the analysis of low-level acoustic features [ 3 , 4 ] to that of higher-level semantic aspects [ 5 ] . These approaches leverage diverse frameworks, including traditional machine learning techniques [ 6 ] , advanced DL architectures [ 7 , 8 , 9 ] , and pre-trained models [ 10 , 11 ] . Most existing detection methods follow a supervised learning strategy, where models are trained on labeled datasets containing both real and synthetic speech samples. However, this approach typically presents two critical challenges: generalization and interpretability [ 12 ] . Generalization refers to the model’s ability to detect deepfakes generated by speech synthesis methods not encountered during training. Interpretability concerns the “black box” nature of the detection systems, which typically provide little insight into the rationale underlying their predictions. These two limitations significantly reduce the practical effectiveness of deepfake detectors in real-world scenarios. Recent research has explored novel strategies to address these limitations. One promising direction involves One-Class (OC) classification systems to improve generalization [ 13 , 14 ] . These models are trained exclusively on authentic speech data, learning its distribution so that any sample deviating from it is classified as synthetic. On the other hand, efforts to enhance interpretability have focused on identifying the specific aspects of the audio signal that drive model predictions, improving the transparency of the detection process [ 15 ] . In this paper, we propose a novel method that addresses these challenges through an interpretable, OC speech deepfake detector. Specifically, we reframe the detection problem as an Anomaly Detection (AD) task, training a OC model exclusively on real speech samples. This enables the model to learn the statistical distribution of genuine speech and classify out-of-distribution samples as fake, preventing it from being fooled by unseen synthesis methods. Our approach builds upon the Student-Teacher Feature Pyramid Matching (STFPM) framework proposed in [ 16 ] , enhanced with Discrepancy Scaling (DS) [ 17 ] , and is capable of detecting and localizing speech anomalies in both time and frequency domains. To evaluate our approach, we test it across diverse scenarios, comparing its performance against multiple baselines. Our results indicate its superiority and validate the effectiveness of framing speech deepfake detection as an AD problem. II Proposed Method In this work, we address the problem of speech deepfake detection and propose an AD framework to tackle it. Our approach is based on the hypothesis that synthetic speech can be conceptualized as authentic speech embedded with anomalies, which arise from imperfections in the speech generative process. These imperfections manifest as subtle artifacts that an analyst can leverage to discriminate between synthetic and authentic signals. Traditional supervised learning systems explicitly learn the characteristics of these artifacts by training on labeled datasets containing both real and synthetic samples. Then, at inference time, they use their acquired knowledge to differentiate between the two classes. Although effective in controlled scenarios, this approach has significant limitations, particularly in its ability to generalize detection performance over speech generation methods that are unseen during training. Different generative models introduce distinct artifacts, and detectors trained on a limited set of synthesis techniques may fail to recognize deepfakes produced by other methods. To overcome this limitation, the proposed framework adopts a OC learning paradigm, considering only authentic speech data during training. The goal is to characterize the statistical distribution of genuine speech so that, during inference, any deviation from this distribution is flagged as an anomaly, indicating potential deepfake content. By focusing only on authentic speech, the model’s performance becomes independent of any speech synthesis method, enhancing its generalization capabilities. Moreover, our AD framework enhances interpretability by generating anomaly maps that localize these deviations in both time and frequency domains, providing valuable forensic insights into the input speech signal. II-A Problem Formulation The speech deepfake detection problem can be formally defined as follows. Let us consider a discrete-time speech signal \mathbf{x} associated with a class y\in\{0,1\} , where 0 denotes that the signal is authentic and 1 indicates that it has been synthetically generated. The goal of this task is to develop a detector \mathcal{D} capable of estimating the class of the signal \mathbf{x} as \hat{y} , where \hat{y} is the likelihood of the signal \mathbf{x} being fake. II-B Proposed System We address the speech deepfake detection task by adopting a One-Class (OC) Anomaly Detection (AD) strategy. In this framework, the detector \mathcal{D} is trained exclusively on authentic speech samples to learn their underlying statistical distribution. During inference, the model evaluates how much a given speech signal deviates from this learned distribution. Signals that closely align with