CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation
Yuxuan Du, Zhendong Wang, Yuhao Luo, Caiyong Piao, Zhiyuan Yan, Hao Li, Li Yuan · 2025
Read CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation by Yuxuan Du, Zhendong Wang, Yuhao Luo, Caiyong Piao, Zhiyuan Yan, Hao Li, Li Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15233.txt.
Introduction
Yuxuan Du and co-authors build a detector reading both content and craft of fake videos. Modern fakes manipulate picture and sound together, so detectors that check only one clue fall short. The authors show two kinds of evidence complement each other. One is fine traces left inside each channel, like face-swap edges or audio distortions. The other is mismatches between channels, like lips out of sync with speech. Their framework, CAD, checks both.
One component aligns the channels' high-level meaning to catch asynchrony. A second component distills features across channels, settling conflicts without erasing the forensic traces. On multi-modal and single-modal test sets the system beat previous methods. The authors argue the lesson is general: fake media betrays itself at several levels, and a detector should listen for all of them rather than betting on one clue.
Catalog record
| CREATOR | Yuxuan Du, Zhendong Wang, Yuhao Luo, Caiyong Piao, Zhiyuan Yan, Hao Li, Li Yuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15233
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation Yuxuan Du, Zhendong Wang, Yuhao Luo, Caiyong Piao, Zhiyuan Yan, Hao Li, Li Yuan Abstract The rapid emergence of multimodal deepfakes (visual and auditory content are manipulated in concert) undermines the reliability of existing detectors that rely solely on modality-specific artifacts or cross-modal inconsistencies. In this work, we first demonstrate that modality-specific forensic traces (e.g., face-swap artifacts or spectral distortions) and modality-shared semantic misalignments (e.g., lip-speech asynchrony) offer complementary evidence, and that neglecting either aspect limits detection performance. Existing approaches either naively fuse modality-specific features without reconciling their conflicting characteristics or focus predominantly on semantic misalignment at the expense of modality-specific fine-grained artifact cues. To address these shortcomings, we propose a general multimodal framework for video deepfake detection via C ross-Modal A lignment and D istillation (CAD). CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization ( e.g. , lip-speech mismatches); 2) Cross-modal distillation that mitigates feature conflicts during fusion while preserving modality-specific forensic traces ( e.g. , spectral distortions in synthetic audio). Extensive experiments on both multimodal and unimodal ( e.g. , image-only/video-only)deepfake benchmarks demonstrate that CAD significantly outperforms previous methods, validating the necessity of harmonious integration of multimodal complementary information. 1 Introduction The rapid advancement of deepfake generation has led to highly sophisticated multimodal forgeries that seamlessly integrate manipulated visual, audio, and textual elements to produce realistic and deceptive content [ 41 , 59 , 47 ] . Modern multimodal generation techniques increasingly exploit cross-modal coherence, such as syncing synthetic voices with lip movements [ 42 , 72 ] , generating contextually plausible scripts [ 40 , 60 ] , or blending facial reenactments with cloned speech [ 29 , 16 ] , thereby making the detection of such multimodal forgeries increasingly challenging for existing approaches. Existing unimodal detection methods ( e.g. , facial blending anomaly detectors [ 48 , 11 , 62 ] or spectral artifact analysis in audio [ 65 , 31 ] ) mainly rely on isolated artifacts in a single modality, making them inherently blind to cross-modal discrepancies and failing to identify cross-modal manipulations like synchronized lip movements with AI-generated voices. Generally, the multimodal forgeries essentially expose detection traces from two key perspectives. 1) Modality-Specific Inconsistencies: These include artifacts confined to a single modality, such as unnatural facial texture blending in manipulated videos ( [ 30 , 48 ] ) or spectral distortions in synthesized audio ( [ 2 , 36 ] ). 2) Modality-Shared Semantic Misalignments: These involve inconsistencies between modalities, such as mismatches between lip movements and speech ( [ 20 , 21 ] ), or incongruent emotional tones across audio and visual streams ( [ 39 ] ). However, we find that most existing multimodal detectors fail to consider how to fully utilize both modality-specific and shared forgeries for comprehensive detection . To illustrate, most multimodal frameworks (such as [ 33 , 39 , 63 ] ) primarily focus on semantic coherence ( e.g. , lip-speech alignment) but often overlook modality-specific forensic traces, such as unnatural texture blending or flickering in synthesized faces and subtle acoustic artifacts in synthetic speech. Additionally, approaches like [ 67 ] directly combine modality-specific cues without effectively leveraging modality-shared information, missing the opportunity to exploit cross-modal correlations that could strengthen forgery detection. Figure 1 : Venn diagram illustrating different types of artifacts in multimodal deepfakes, categorized into modality-specific and modality-shared cues. Specifically, visual artifacts ( x_{1} ) may include blending boundaries of face-swapping, while audio artifacts ( x_{2} ) might exhibit spectral anomalies. The shared space ( x_{12} ) captures semantic cross-modal mismatches, such as inconsistency between lip movements and speech. Ideally, a robust system should integrate both perspectives for improved accuracy. By revisiting the existing detection works, we argue that a truly effective deepfake detection system must integrate both modality-specific and shared perspectives, ensuring that neither local artifacts nor cross-modal mismatches escape detection . By leveraging the full multimodal space rather than treating modalities independently, we can build a stronger defense against increasingly sophisticated fakes. As illustrated in Figure 1 , we divide the traces of multimodality deepfake detection into modality-specific and modality-shared inconsistencies. The visual domain ( x_{1} ) contains artifacts like blending boundaries or unrealistic motion, while the audio domain ( x_{2} ) may exhibit synthetic spectral anomalies. More critically, their joint representation ( x_{12} ) captures discrepancies that neither modality alone can reveal, such as a deepfake that synchronizes well but exhibits unnatural articulation patterns or mismatched speech emotion. To achieve this, we propose CAD , a novel multimodal detection framework that unifies modality-shared semantic alignment analysis and modality-specific forensic verification. CAD operates on the principle that multimodal deepfakes inevitably leave traces in two forms : (1) semantic misalignments between modalities ( e.g. , discrepancies between spoken words and lip movements) and (2) modality-specific inconsistencies ( e.g. , residual artifacts from facial swapping or synthetic voice generation). Unlike prior works that prioritize one aspect at the expense of the other, CAD introduces a dual-path architecture : a cross-modal alignment module to detect semantic disharmony and a distillation mechanism to preserve and harmonize modality-specific forensic signals . By synergizing these complementary cues, CAD closes the detection gap left by conventional approaches, offering a unified defense against the escalating threat of multimodal synthetic media. Evaluated on the comprehensive IDForge dataset, which spans diverse forgery types, CAD achieves state-of-the-art performance (99.96% AUC), demonstrating its robustness against evolving multimodal attacks. Our main contributions are summarized as three-fold: • Unified Detection of Modality-Shared and Modality-Specific Cues : We propose CAD, a novel detection framework that integrates both modality-shared semantic alignment analysis ( e.g. , lip-speech mismatches) and modality-specific forensic verification ( e.g. , residual artifacts in face-swapped videos or synthetic audio). • Dual-Path Architecture Design : We introduce a cross-modal alignment module to detect semantic disharmony across different modalities, and develop a cross-modal distillation mechanism that preserves forensic artifacts within individual modalities while mitigating conflicts during multimodal feature fusion. • SOTA Performance on Multimodal Deepfake Detection : We evaluate CAD on the IDForge dataset, which covers a wide range of deepfake manipulation techniques, achieving 99.96% AUC, significantly outperforming existing unimodal and multimodal detection methods, validating the necessity of a holistic approach to multimodal deepfake detection. 2 Related Works 2.1 Deepfake Detection Unimodal Deepfake Detection. Most existing deepfake video detection methods focus on identifying visual artifacts in image modality [ 49 , 61 , 48 , 30 , 58 ] and video modality [ 21 , 53 , 57 , 66 , 62 ] . Notably, early works, such as [ 21 ] , leveraged lip-reading pre-trained models to detect inconsistencies in lip movements, while deepfake image detection initially focused on blending boundaries [ 30 ] , a common artifact in face-swapping forgeries. Later, [ 48 , 11 ] further explored blending-based detection, refining the objective from cross-ID to within-ID forgery analysis [ 48 ] and examining the role of blended data in detection [ 11 ] . Meanwhile, video-based detection evolved to balance spatial and temporal artifact learning [ 51 , 62 ] , mitigating overfitting to a single modality ( e.g. , spatial artifacts only) and