Alethia: A Foundational Encoder for Voice Deepfakes

Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti · 2026

Read Alethia: A Foundational Encoder for Voice Deepfakes by Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00251.txt.

Introduction

Yi Zhu and 3 co-authors shift attention in voice deepfake detection from fine-tuning to pretraining. Detectors today mostly borrow representations from general speech foundation models, and squeezing more out of fine-tuning has hit diminishing returns. Their encoder, Alethia, is trained from the start for the job of spotting and locating fake speech.

Alethia's training mixes masked embedding prediction with a flow-based method that rebuilds spectrograms. Across five tasks and fifty-six benchmark datasets, it beat leading speech foundation models, resisted real-world distortions, and generalized to unseen domains such as singing deepfakes. The authors also show that predicting discrete tokens is a weak target for this job. Continuous embedding prediction and generative pretraining matter more for catching the artifacts that give fakes away.

Catalog record

CREATORYi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Alethia: A Foundational Encoder for Voice Deepfakes Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti Abstract Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction . The outcome, Alethia , is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts. 1 Introduction State-of-the-art speech foundation models (SFMs), such as Wav2vec ( 7 ) , WavLM ( 14 ) , and HuBERT ( 29 ) , provide efficient audio representations that facilitate strong downstream performance on a variety of learning tasks ( 80 ) . At the core of these models is the self-supervised learning paradigm built on a well-designed pretraining recipe and a massive corpus of unlabeled data. A standard pretraining recipe is the BERT-like masked token prediction, where discrete targets computed from the speech sample provide a supervisory signal for the representation learning ( 7 ; 14 ; 29 ; 31 ; 42 ; 81 ) . This was found to be effective for a variety of audio encoders, for example, the MERT model for music ( 42 ) , SPEAR for speech and audio ( 81 ) , and MERaLiON for Singlish data ( 31 ) , among others. Of particular interest in this paper is the representation learning for voice deepfakes, which have now become a major concern due to the rapid emergence of high-quality generative methods. Related learning tasks include speech deepfake detection ( 39 ) , singing voice deepfake detection ( 85 ) , partially fake speech localization ( 27 ) , and deepfake source tracing ( 36 ) . While the downstream finetuning architectures vary across different state-of-the-art (SOTA) models, SFMs remain as the key integral component (often termed as ‘frontend’), wherein a better encoder representation yields a strong boost in performance ( 84 ; 88 ; 86 ) . Most voice deepfake models today reuse off-the-shelf SFMs, and focus primarily on the downstream finetuning task for performance gains ( 86 ) . However, recent recent trends suggest that despite finetuning on massive volumes of real and fake speech, models still lack generalizability to unseen generation methods ( 22 ; 11 ) and are sensitive to real-world perturbations ( 40 ; 90 ; 51 ) . These trends suggest that the finetuning paradigm may be fundamentally limited by the representations inherited from the general-purpose SFMs, which are not optimized for capturing generative artifacts. A few studies have recently explored altering the pretraining recipe instead, such as continual pretraining with vocoded speech ( 72 ) and low-rank adaptation of the encoder ( 26 ) , but the reported performance gains were only incremental ( 72 ; 22 ) . We note that the pretraining objective formulation as well as the scale and variety of pretraining data need to be revisited in order to fully capture the discriminative artifacts in voice deepfakes and thereby achieve better generalization. In this work, our goal is to build a foundational encoder for voice deepfakes that well encapsulates the discriminative patterns between real and generated voice, demonstrates robustness to real-world perturbations and, with simple finetuning, generalizes well to unseen synthesis methods. We propose a pretraining recipe, wherein, the encoder receives supervision from two branches in parallel. In the first branch, the encoder is tasked to perform bottleneck masked embedding prediction , where the target embeddings are extracted by a frozen teacher model from unmasked waveforms, and the student model learns to predict different teacher layers from its bottleneck (i.e., layer-averaged) representation obtained from masked waveforms. In the second branch, we introduce a flow-matching decoder that generates the unmasked spectrogram conditioned on the bottleneck representation. Together, these objectives formulate a strong pretraining recipe for a foundational encoder for voice deepfake tasks. Our main contributions are summarized as follows: Contribution 1: We introduce Alethia , the first foundational encoder for voice deepfakes. Alethia markedly outperforms commonly used SFMs on 56 datasets across 5 different voice deepfake tasks. Contribution 2: We show the limitation of existing pretraining methods, and propose a new pretraining strategy, consisting of bottleneck masked embedding prediction and flow-matching based spectrogram generation to jointly enhance generalization and robustness to unseen attacks. Contribution 3: We show that Alethia demonstrates zero-shot generalizability to completely unseen attacks (e.g., singing voice deepfakes) and is robust to various real-world perturbations without needing complex augmentations during finetuning. 2 Related Works 2.1 Predictive Pretraining and Target Choice Mainstream SFMs usually rely on predictive pretraining, where the predictive targets are pseudo-labels/features extracted from the signal itself ( 49 ) . Among different pretraining components (e.g., target, model architecture, data, etc.), the target choice has been demonstrated to be critical for model generalizability ( 49 ; 3 ; 8 ) . Predominant encoders in the deepfake domain, such as Wav2vec2 ( 7 ) , WavLM ( 14 ) , and HuBERT ( 29 ) , apply quantization via codebook or clustering techniques to generate discrete tokens as targets. While discretization effectively extracts high-level phonetic structures and filter out attributes less relevant to speech content or speaker ( 7 ; 29 ; 49 ) , other signal nuances such as generation traces of voice deepfakes could also be overlooked. More recently, masked embedding prediction has shown promising results across audio and vision domain. Some representative works include Data2vec2 ( 6 ) , JEPA ( 3 ) , and V-JEPA ( 4 ) , which leverage self-distillation to align student and teacher output in the latent space. While in voice deepfake tasks, masked token prediction models still yield leading performance ( 39 ) , we hypothesize that reconstruction in the continuous embedding space could provide a more granular objective for modeling generation traces and have the potential to be improved for learning voice deepfake patterns. 2.2 Combining Generative and Predictive Pretraining Audio generative pretraining focuses on recovering a low-level acoustic representation (e.g., waveform or spectrogram) from latent embeddings or discrete codes ( 50 ; 44 ) . Intuitively, the acoustic details learned by the generative objective should also benefit discriminative speech tasks. However, recent models that attempted combining predictive and generative pretraining were observed to consistently downperform predictive-only models on discriminative tasks ( 45 ; 32 ; 50 ) . Missing in previous works is an effective method to integrate generative pretraining without degrading the representation’s discriminative power. It is also unclear whether such a pretraining recipe can produce strong representations for various voice deepfake tasks. 3 Motivation and Early Exploration When building a foundational encoder for voice deepfake tasks, the pretraining objective should help capture traces in deepfakes on both the spoken content and the acoustic detail that make the voice sound realistic.

Read the complete text