SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment
Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang · 2026
Read SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment by Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25140.txt.
Introduction
Shahzad and co-authors detect fake videos by training only on real ones. Detectors taught on fake sets inherit the quirks of those sets and fail on new tricks. SAVe instead blends parts of a real face into itself, making practice fakes that keep the person's identity. It learns from those blends at several face scales. A second part checks whether lips and speech line up in time. On two test sets it scores well at home and, more telling, holds up across datasets.
Catalog record
| CREATOR | Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35680 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang Abstract Multimodal deepfakes can exhibit subtle visual artifacts and cross-modal inconsistencies, which remain challenging to detect, especially when detectors are trained primarily on curated synthetic forgeries. Such synthetic dependence can introduce dataset and generator bias, limiting scalability and robustness to unseen manipulations. We propose SAVe , a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos. SAVe generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts, enabling the model to learn complementary visual cues across multiple facial granularities. To capture cross-modal evidence, SAVe also models lip-speech synchronization via an audio-visual alignment component that detects temporal misalignment patterns characteristic of audio-visual forgeries. Experiments on FakeAVCeleb and AV-LipSync-TIMIT demonstrate competitive in-domain performance and strong cross-dataset generalization, highlighting self-supervised learning as a scalable paradigm for multimodal deepfake detection. I Introduction Recent progress in audio-visual generative models has increased the realism of talking-face forgeries, motivating detection methods that utilize multimodal evidence rather than relying on uni-modal ones [ 25 , 31 , 17 ] . Early visual detectors used artifacts such as boundary blending, spatial inconsistencies, and texture anomalies [ 31 , 1 , 30 ] ; however, these cues are increasingly attenuated as the quality of the synthesis improves, shifting attention to speech-related signals and lip-speech correspondence [ 50 , 56 , 44 , 18 ] . Most existing detectors remain fully supervised, as shown in Fig. 1 , and depend on labeled deepfake datasets, which limits scalability and degrades transfer due to shortcut learning and sensitivity to dataset-specific factors (e.g., compression and preprocessing) rather than manipulation evidence [ 37 , 36 , 5 , 52 ] . This motivates label-efficient, manipulation-agnostic learning, where self-supervised objectives exploit intrinsic temporal and cross-modal consistency without forgery annotations [ 46 , 47 ] . We propose SAVe , a self-supervised audio-visual deepfake detection framework trained exclusively on real videos. SAVe consists of four complementary modules: FaceBlend for generic spatial editing artifacts, LipBlend for fine-grained lip-region irregularities, LowerFaceBlend for discriminative lower-face cues related to viseme-phoneme coupling, and AVSync for temporal synchronization between speech and lip motion. By unifying intra-modal visual consistency and cross-modal audio-visual coherence, SAVe promotes label-efficient, manipulation-agnostic generalization by discouraging shortcut learning and improving robustness to unseen deepfake generation techniques, while enabling scalable training without labeled forgeries. Fig. 1: Prior visual/audio-visual deepfake detectors typically rely on manipulated training data and task-specific preprocessing, using either visual-only models or audio-visual synchronization cues with ad-hoc fusion, which can lead to poor generalization under unseen generators. In contrast, the proposed model, SAVe, is trained exclusively on authentic videos by generating on-the-fly, identity-preserving self-blended pseudo-manipulations. SAVe combines FaceBlend , LipBlend , LowerFaceBlend , and AVSync features via a Fusion head for robust real/fake prediction. Our contributions are as follows. • We propose SAVe , a self-supervised audio-visual deepfake detection framework trained exclusively on authentic videos, which jointly learns visual manipulation artifacts and audio-visual synchronization inconsistencies without any synthetic deepfake training data. • We unify intra-modal and cross-modal consistency objectives within a single framework to improve generalization to diverse and unseen manipulations. • We mitigate shortcut learning through manipulation-agnostic consistency cues, enabling label-efficient and cross-dataset robust detection. • Extensive experiments on two benchmarks demonstrate strong performance and cross-dataset robustness without forged training data. II RELATED WORK Deepfake detection has been studied in visual, audio, and audio-visual settings [ 25 , 17 ] . While supervised detectors achieve strong in-domain performance, their reliance on specific forgery generators and labeling pipelines often induces dataset bias and limits robustness to unseen manipulations. This motivates recent interest in self-supervised and training paradigms that aim to learn intrinsic authenticity cues using authentic data only. II-A Visual deepfake detection Visual-only detectors commonly train convolutional neural network (CNN)/Transformer classifiers on manipulated datasets [ 1 , 40 ] and exploit artifacts and inconsistencies (e.g., local texture, facial dynamics, and temporal cues) using explicit sequence modeling [ 15 , 3 , 22 , 23 , 58 ] . Recent Transformer-based designs improve temporal context modeling and attempt to mitigate shortcut cues such as identity leakage, which can otherwise harm cross-dataset generalization [ 53 , 12 , 26 , 13 ] . Nevertheless, purely visual cues may be unreliable when synthesis is highly photorealistic or when the audio track is manipulated. Fig. 2: Overview of the proposed SAVe framework, a self-supervised learning (SSL) audio-visual deepfake detection method trained solely using authentic data. Given an input video, visual frames and audio are extracted, in the AVSync branch, both modalities are encoded by AV-HuBERT to obtain audio ( F_{a} ) and visual ( F_{v} ) representations, which are then processed by an alignment network to produce an audio-visual misalignment score that reflects lip-speech synchronization consistency. In parallel, the SS-VPFG module synthesizes on-the-fly pseudo-forged visual samples via identity-preserving self-blending over multiple facial regions, enabling region-aware supervision: (face (real \text{I}_{\text{FR}} vs. fake \text{I}_{\text{FB}} ), lip (real \text{I}_{\text{LR}} vs. fake \text{I}_{\text{LB}} ), and lower-face (real \text{I}_{\text{LFR}} vs. fake \text{I}_{\text{LFB}} )) to provide region-aware supervision. Finally, region-specific visual features and AVSync misalignment cues are aggregated by the Fusion module to output the final prediction (Real/Fake). II-B Audio deepfake detection Audio-only spoofing detection targets artifacts introduced by text-to-speech (TTS) or voice conversion (VC) systems and neural vocoders, evolving from hand-crafted spectral features with classical classifiers [ 52 ] to end-to-end architectures (e.g., RawNet2 and AASIST) and self-supervised speech representations with augmentation for improved robustness [ 27 , 48 , 21 , 49 , 51 , 34 , 54 ] . Generalization remains challenging under unseen generators, codec distortions, and multilingual conditions, motivating one-class and codec-aware strategies [ 54 , 9 , 57 , 28 ] . Audio-only approaches also fail when forged video is paired with genuine audio, because they cannot detect cross-modal inconsistencies and visual artifacts. II-C Audio-visual deepfake detection Audio-visual detectors exploit speech-face coupling to expose cross-modal mismatches through synchronization constraints, contrastive objectives, and lip-sync modeling [ 44 , 32 , 7 , 2 , 6 , 42 ] . Fusion mechanisms such as cross-attention and pretrained audio-visual representations further improve temporal context modeling [ 18 , 59 , 19 , 20 ] . However, most methods, including LLM-based models that generalize poorly [ 43 ] , remain supervised on labeled forgeries and may inherit dataset bias from specific manipulation pipelines, limiting transfer to unseen generators and editing strategies. To reduce dependence on synthetic data, one-class and real-only approaches learn from authentic samples and flag anomalous deviations [ 4 , 29 , 14 , 35 , 38 , 10 , 39 ] . Related self-supervised strategies generate pseudo forgeries from real data using self-blended image techniques [ 46 , 16 ] , allowing artifact-aware learning without forged training sets. However, the techniques are designed for broad artifacts across the entire face; consequently, they may miss specific visual artifacts that occur during lip-syncing. Building on this direction, we propose an audio-visual framework that performs region-wise self-blending (full face, lips, and lower face) and incorporates an audio-visual alignment module to capture lip-sync irregularities by leveraging only authentic data. This combination targets both spatial artifacts and cross-modal timing