KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features DOI: XXXXXXX.XXXXXXX Conference: 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/10 CCS: Security and privacy Spoofing attacks

Ivan Kukanov, Jun Wah Ng · 2025

Read KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features DOI: XXXXXXX.XXXXXXX Conference: 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/10 CCS: Security and privacy Spoofing attacks by Ivan Kukanov, Jun Wah Ng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07337.txt.

Introduction

Fake videos of talking heads are getting harder to spot, and a detector that works on one trick may fail on the next. Ivan Kukanov and Jun Wah Ng built a detection system for a 2025 multimedia challenge, described in a 2025 paper. Their system, KLASSify, judges both the picture and the sound of a clip.

For the picture, they use hand-built features chosen to stay readable and adaptable. For the sound, they adapt a self-supervised speech model joined with a graph attention network that picks out rich audio patterns. On the challenge dataset the pair reached an AUC score of 92.78 percent for telling real from fake. Sound alone also located the fake stretches in time with a fair overlap score. The authors aim for a balance of accuracy, cost, and clear reasoning that suits real use.

Catalog record

CREATORIvan Kukanov, Jun Wah Ng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07337

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features DOI: XXXXXXX.XXXXXXX Conference: 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/10 CCS: Security and privacy Spoofing attacks CCS: Security and privacy Biometrics Ivan Kukanov, Jun Wah Ng Abstract . The rapid development of audio-driven talking head generators and advanced Text-To-Speech (TTS) models has led to more sophisticated temporal deepfakes. These advances highlight the need for robust methods capable of detecting and localizing deepfakes, even under novel, unseen attack scenarios. Current state-of-the-art deepfake detectors, while accurate, are often computationally expensive and struggle to generalize to novel manipulation techniques. To address these challenges, we propose multimodal approaches for the AV-Deepfake1M 2025 challenge. For the visual modality, we leverage handcrafted features to improve interpretability and adaptability. For the audio modality, we adapt a self-supervised learning (SSL) backbone coupled with graph attention networks to capture rich audio representations, improving detection robustness. Our approach strikes a balance between performance and real-world deployment, focusing on resilience and potential interpretability. On the AV-Deepfake1M++ dataset, our multimodal system achieves AUC of 92.78 % 92.78\% for deepfake classification task and IoU of 0.3536 0.3536 for temporal localization using only the audio modality. Figure 1. KLASS Solution Pipeline for Deepfake Classification. Self-supervised Learning (SSL)-based features followed by graph attention networks (GAT) are fine-tuned for audio stream, crafted video features are trained with temporal convolution networks (TCN). The prediction scores from audio-visual models are calibrated and max-out decision is obtained per video clip. 1. Introduction The advancement of audio-driven talking head generators has made it trivial to create temporal deepfakes, where only the speech segment and the corresponding mouth-region motion are manipulated, while the remainder of the video stays authentic. State-of-the-art talking head generators Li et al. 2025 ; Mukhopadhyay et al. 2023 ; Wang et al. 2023 can produce frame-level lip movements that are indistinguishable from real videos. More importantly, video deepfake manipulations are now accompanied by equally powerful TTS systems. Zero-shot TTS models Casanova et al. 2023 ; Casanova et al. 2024 can closely clone a speaker’s voice without requiring extra fine-tuning. When sufficient data are available, TTS models such as VITS Kim et al. 2021 can be further fine-tuned to produce intonation and prosody that are nearly indistinguishable from the real speaker. These new waves of deepfake attacks are poised to amplify disinformation, erode trust in audiovisual evidence, and threaten national security. Large-scale temporal deepfake benchmark datasets Cai et al. 2024 ; Cai et al. 2023 ; Cai et al. 2025 highlight both the scale and the challenges faced by existing methods when detecting short manipulated segments embedded in long, genuine videos. The deepfake research community has increasingly focused on multi-modal temporal detectors that jointly analyze audio and video streams. Existing state-of-the-art multimodal temporal deepfake detectors Cai et al. 2023a ; Cai et al. 2022 ; Zhang et al. 2023a typically rely on large, pre-trained encoders and complex fusion layers to detect and localize deepfakes. These heavy backbones, often with hundreds of billions of parameters, achieve state-of-the-art accuracy but require significant computational resources, making them impractical for real-time deployment. However, such models often suffer from poor robustness to distribution shifts, unseen or new manipulations, of which are common in real-world forensics applications. An example would be BA-TFD+ Cai et al. 2023a , which scored 96.30 [email protected] on the LAV-DF Cai et al. 2023 but struggles at 14.7 [email protected] in the AV-Deepfake1M++ Cai et al. 2025 testA split. To address these challenges, we propose multimodal approaches for both the detection and localization tracks of the AV-Deepfake1M 2025 Cai et al. 2025 competition. We first decouple the audio and video modalities, allowing independent analysis and providing more interpretable insights. For the visual modality, we extract handcrafted features that capture the visual artifacts inherent in deepfakes. These features enhance both explainability and robustness to unseen attacks. For the audio modality, we leverage SSL-based features, enabling us to capture complex, high-level audio cues such as intonation, prosody, and phonetic patterns. These features are particularly effective in detecting manipulated audio in deepfakes Tak & Todisco 2022 , even when faced with unseen types of attacks. This improves the model’s resilience to new and evolving deepfake manipulation techniques, ensuring reliable detection across diverse audio manipulations. Detection Task For Task 1, we propose a simple score-based shallow fusion approach that achieved high accuracy in the detection task. By training a TCN Lea et al. 2016 on handcrafted visual features alongside Wav2Vec-AASIST Tak & Todisco 2022 trained on audio features, we were able to achieve competitive results. Localization Task For Task 2, the boundary-aware attention Zhong et al. 2024 approach is adapted for localizing fake segments in the audio stream. 2. Problem Statement Tasks Overview . The proposed methods were developed according to the AV-Deepfake1M 2025 Cai et al. 2025 challenge rules. The challenge comprises two tasks supported by the AV-Deepfake1M++ Cai et al. 2025 dataset: •Task 1. Given a video clip featuring a single speaker, the objective is to determine whether the video is real or manipulated. For this task, participants are restricted to using video-level labels for training. •Task 2. Given similar video samples, the goal is to identify time stamps of manipulated fake segments induced in real video. Participants are provided with the ground-truth segment-level labels that indicate the exact locations of the fake content. Evaluation Metrics . For Task 1, the evaluation metric for deepfake detection is the Area Under the Curve (AUC) score. The AUC measures the discrimination ability of a deepfake detection system on multiple operating points. For Task 2, the metrics are the Average Precision (AP) and the Average Recall (AR). The overall score is computed as: (1) \displaystyle\text{Score} \displaystyle=\frac{1}{8}\sum_{\text{IoU}\in\{0.5,0.75,0.9,0.95\}}\text{AP@IoU}\,\,+ \displaystyle\frac{1}{10}\sum_{\text{N}\in\{50,30,20,10,5\}}\text{AR@N} 3. Related Work Visual deepfake Early visual deepfake detectors operated frame-by-frame, focusing on spatial artifacts within individual frames Zheng et al. 2021 . However, since most deepfakes are generated in a frame-by-frame manner, they often exhibit clear temporal inconsistencies that single-frame detectors fail to capture. To address this, recent approaches have incorporated temporal modeling to detect inter-frame anomalies such as flickering, jitter, or abrupt shifts in facial landmarks Gu et al. 2021 . Studies have shown that audio-driven lip-sync deepfakes frequently disrupt the smooth, natural kinematics of lip movement, with synthetic fakes failing to adhere to normal physiological dynamics, resulting in irregular motion patterns Liu et al. 2025 . Spatiotemporal detectors such as UMMAFormer Zhang et al. 2023a leverage frame differences across both RGB and optical flow to capture such inconsistencies. Other approaches, like Tan et al. 2023 , model facial action unit dynamics to detect irregularities, improving the detection of speech-driven and reenactment fakes. Most detectors lack generalizability, performing well on known generators but struggling with unseen ones due to overfitting to generator specific artifacts Xiong et al. 2025 . Despite recent advances, common artifacts persist across generators. Early face swaps often displayed boundary and lighting mismatches Xiong et al. 2025 , while modern talking-head generators continue to struggle with regional consistency. Many lip-sync fakes alter only the mouth region, leaving the surrounding face unnaturally static during speech Datta et al. 2024 . Additionally, forged mouth regions often exhibit blurriness, color shifts, or unnatural landmark kinematics. These cross-generator artifacts

Read the complete text