Phoneme-Level Analysis for Person-of-Interest Speech Deepfake Detection
Davide Salvi, Viola Negroni, Sara Mandelli, Paolo Bestagini, Stefano Tubaro · 2025
Read Phoneme-Level Analysis for Person-of-Interest Speech Deepfake Detection by Davide Salvi, Viola Negroni, Sara Mandelli, Paolo Bestagini, Stefano Tubaro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08626.txt.
Introduction
Davide Salvi and co-authors catch speech fakes sound by sound in a 2025 paper. Person-of-interest detectors guard specific voices against imitation. Whole-clip verdicts hide where a fake slips. Phoneme-level checks promise both detail and explanation.
Reference speech is split into phonemes to build a speaker profile. Each phoneme of a test clip is then compared against that profile. Odd sounds betray synthetic artifacts one unit at a time. Accuracy matches classic detectors while robustness and interpretability improve. Forensic users gain a verdict they can point to, phoneme by phoneme, rather than a bare score.
Catalog record
| CREATOR | Davide Salvi, Viola Negroni, Sara Mandelli, Paolo Bestagini, Stefano Tubaro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Phoneme-Level Analysis for Person-of-Interest Speech Deepfake Detection Davide Salvi, Viola Negroni, Sara Mandelli, Paolo Bestagini, Stefano Tubaro Abstract Recent advances in generative AI have made the creation of speech deepfakes widely accessible, posing serious challenges to digital trust. To counter this, various speech deepfake detection strategies have been proposed, including Person-of-Interest (POI) approaches, which focus on identifying impersonations of specific individuals by modeling and analyzing their unique vocal traits. Despite their excellent performance, the existing methods offer limited granularity and lack interpretability. In this work, we propose a POI-based speech deepfake detection method that operates at the phoneme level. Our approach decomposes reference audio into phonemes to construct a detailed speaker profile. In inference, phonemes from a test sample are individually compared against this profile, enabling fine-grained detection of synthetic artifacts. The proposed method achieves comparable accuracy to traditional approaches while offering superior robustness and interpretability, key aspects in multimedia forensics. By focusing on phoneme analysis, this work explores a novel direction for explainable, speaker-centric deepfake detection. 1 Introduction Recent advancements in generative AI have made it possible to produce hyper-realistic synthetic content with unprecedented ease. While these technologies hold enormous potential for applications in entertainment, education, and accessibility, they also pose significant risks to security, privacy, and trust in digital communication [ 17 ] . One of the most concerning menaces is represented by deepfakes, synthetic multimedia content generated through Deep Learning (DL) techniques that depict individuals performing actions or making statements they never actually did [ 40 , 22 ] . In the audio domain, speech deepfakes allow the generation of synthetic speech that mimics the voice of a target speaker with remarkable realism, enabling malicious applications such as impersonation, fraud, and disinformation. To combat the growing risks associated with the misuse of synthetic speech, the multimedia forensics community has developed a wide range of countermeasures, primarily aimed at protecting Automatic Speaker Verification (ASV) systems and enhancing deepfake detection by reliably distinguishing between real and fake speech signals [ 31 , 13 ] . To this end, a variety of approaches have been explored, including transformer-based architectures [ 45 , 6 , 14 ] , Mixture of Experts (MoE) models [ 26 , 42 ] , and others DL -based techniques [ 4 , 37 , 19 , 44 , 47 ] . Figure 1 : Proposed phoneme-based POI speech deepfake detection pipeline. More recently, POI -oriented methods have emerged as a promising strategy to enhance speaker-specific protection [ 1 , 12 ] . These approaches create a speaker profile based on a set of reference recordings from a target individual requiring protection. During inference, the profile is used to compute the similarity between a given speech sample and the reference speaker’s characteristics, determining the authenticity of the analyzed track [ 29 ] . Compared to conventional deepfake detection systems, POI -based methods offer two key advantages. First, their speaker-centric nature leads to higher accuracy in detecting attacks against specific individuals. Second, since they rely exclusively on real speech data rather than training on synthetic samples, they proved remarkable generalization capabilities and enhanced robustness against various deepfake generation techniques [ 48 ] . One of the most widely used approaches in POI -based methods involves representing each speech sample with a single feature vector, obtained using either supervised learning models or pre-trained foundation models [ 30 ] . While this approach has demonstrated strong detection performance, it is inherently limited in granularity. Representing an entire speech segment with a fixed-length embedding may result in the loss of fine-grained cues that are critical for detecting synthetic speech. Additionally, this coarse representation reduces interpretability, as it does not provide information about which specific aspects of the speech signal contribute to the final decision. As a result, it may be more susceptible to relying on spurious artifacts or shortcuts rather than genuine indicators of synthetic speech [ 41 ] . In this work, we propose a phoneme-level approach to POI -based speech deepfake detection, shifting from full-signal analysis to a novel fine-grained phoneme-centric evaluation, as illustrated in Figure 1 . Rather than processing the entire speech signal as a whole and extracting a single, global feature representation, our method decomposes the input audio into individual phonemes and then processes each phonetic unit independently to extract a dedicated feature vector from it. These phoneme-level embeddings, derived from reference speech, are aggregated to construct a speaker-specific phoneme profile. During inference, the test audio is similarly decomposed into phonemes, and each of these is individually compared to the speaker profile, enabling the detection of synthetic artifacts at a much finer temporal and linguistic resolution. Our method is grounded on the idea that speech deepfake generators, despite their sophistication, struggle to replicate short, fundamental units of speech, such as phonemes, with perfect fidelity. By refining the analysis and focusing only on the most critical parts of the signal, our method identifies the specific phonemes that deviate significantly from the reference speaker’s characteristics, resulting in a more robust and reliable analysis. Moreover, the proposed approach inherits the advantages of one-class POI -based methods, making it more resilient to different deepfake generators and diverse recording conditions. This work explores phoneme-level analysis as a promising approach for POI -based speech deepfake detection. Our findings support the viability of this method and suggest future research directions, such as integrating phonetic knowledge into end-to-end detectors and developing more interpretable, speaker-centric solutions. Understanding which specific aspects of speech can be reliably used to discriminate between real and fake audio, and localizing generation artifacts within a signal in both time and frequency domains, could serve as valuable tools to combat voice spoofing in an era of increasingly realistic synthetic media. The rest of the paper is organized as follows. Section 2 provides background on POI and phoneme-based speech deepfake detection methods. Section 3 formally defines the problem at hand and details the proposed detection pipeline. Section 4 outlines the experimental setup used to evaluate the method. Section 5 presents the results along with analysis and discussion. Finally, Section 6 summarizes the key findings and outlines directions for future work. 2 Background POI-Based Speech Deepfake Detection. The rapid advancement of synthetic speech generation techniques has led to growing interest in developing reliable methods for detecting speech deepfakes. In recent years, the research community has proposed a wide range of techniques based on diverse detection paradigms [ 3 , 20 ] , framing the task as a binary classification problem and relying heavily on supervised learning. While these supervised methods have shown promising results, they also have important limitations. For instance, accurately identifying speech generated by methods not seen during training remains a significant challenge and undermines the robustness of these systems in real-world conditions. To address this limitation, Person-of-Interest (POI) -based methods have emerged as an alternative. These approaches focus on speaker-specific analysis, training exclusively on real data from a given target speaker. This approach leads to improved accuracy and robustness for the enrolled speaker, as well as enhanced generalization across a wide range of speech generation techniques. While several POI -based deepfake detection methods have been proposed in the video domain [ 1 , 2 , 11 , 12 ] , their