AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization

Christos Koutlis, Symeon Papadopoulos · 2025

Read AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization by Christos Koutlis, Symeon Papadopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18993.txt.

Introduction

Christos Koutlis and Symeon Papadopoulos locate the faked parts of a video by cross checking sound against lips. Deepfake clips can swap a face or a voice in short spans, so a whole clip verdict is not enough; the fake seconds must be marked. Their AuViRe method rebuilds speech features of one channel from the other.

In true speech, lip motion predicts the sound and the sound predicts the lips. In a faked span that cross rebuild strains, and the gap swells into a clear cue. Scores beat the past state of the art by wide margins on two bench sets and on a wild test. Fine time marks help moderators cut the fake and keep the rest.

Catalog record

CREATORChristos Koutlis, Symeon Papadopoulos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18993

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization Christos Koutlis, Symeon Papadopoulos Abstract With the rapid advancement of sophisticated synthetic audio-visual content, e.g., for subtle malicious manipulations, ensuring the integrity of digital media has become paramount. This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe). Specifically, our approach reconstructs speech representations from one modality (e.g., lip movements) based on the other (e.g., audio waveform). Cross-modal reconstruction is significantly more challenging in manipulated video segments, leading to amplified discrepancies, thereby providing robust discriminative cues for precise temporal forgery localization. AuViRe outperforms the state of the art by +8.9 [email protected] on LAV-DF, +9.6 [email protected] on AV-Deepfake1M, and +5.1 AUC on an in-the-wild experiment. Code available at https://github.com/mever-team/auvire . 1 Introduction The advancement of generative AI has led to the rise of highly realistic synthetic media, often referred to as deepfakes [ 27 ] . These audio-visual manipulations threaten digital integrity, as they are designed to be perceptually convincing and capable of deceiving both humans and automated systems. This deceptive capacity fuels misinformation, erodes public trust, and carries significant societal and security risks. Ensuring the integrity of audio-visual content has become a critical challenge, driving the need for robust deepfake detection and temporal localization solutions. While significant progress has been made in deepfake detection, accurately identifying and temporally localizing manipulations in audio-visual content remains a substantial challenge. Existing methods [ 18 , 9 , 35 , 19 , 25 , 34 , 36 , 31 , 8 , 21 ] , primarily focus on coarse-grained video classification instead of temporal forgery localization, while neglecting to also leverage the subtle cross-modal inconsistencies that are crucial for fine-grained temporal localization. In contrast, spatial forgery localization, being out of this work’s scope, has been widely explored [ 32 , 40 ] . Crucially, most state of the art multimodal deepfake detectors [ 10 , 7 ] , directly process the raw audio-visual signal instead of utilizing high-level features extracted by large models, which renders them prone to overfitting and less robust to content distortions, or utilize general purpose feature extractors [ 6 , 39 ] limiting their representational capacity. AuViRe addresses these limitations by explicitly modeling cross-modal discrepancies in speech representations. It leverages pre-trained speech-related feature extractors [ 30 ] for both visual and audio modalities, providing highly relevant and robust cues for detecting forged speech. Designed for Temporal Forgery Localization (TFL), our approach enables both fine-grained (frame-level) and coarse-grained (video-level) predictions rendering it also suitable for Deepfake Detection (DFD). Specifically, the proposed architecture first independently extracts visual and audio speech-related features. A representation reconstruction module then predicts visual speech representations from audio, complemented by unimodal (audio-audio and visual-visual) reconstructions. Finally, a reconstruction-discrepancy encoder processes these reconstruction errors to identify forgery features. AuViRe outperforms the state of the art by +8.9 [email protected] on LAV-DF [ 7 ] , +9.6 [email protected] on AV-Deepfake1M [ 5 ] benchmarks. In addition, we provide a real-world implementation accounting for the presence of both visual and audio streams, large video lengths, and talking human presence, yielding +5.1 AUC on in-the-wild experiments. Our main contributions are: 1. The proposed AuViRe architecture, a novel framework for audio-visual deepfake temporal localization based on cross-modal reconstruction discrepancy modeling. 2. An extensive evaluation on established benchmarks (LAV-DF, AV-Deepfake1M) achieving state of the art performance on both TFL and DFD. 3. A real-world evaluation of AuViRe, demonstrating strong performance in the wild. 2 Related Work Video Deepfake Detection. Pertinent methods focus on spatio-temporal inconsistencies [ 9 , 18 ] , analyze frequency domain artifacts [ 14 , 19 ] , or use a multi-attentional approach [ 40 ] . Approaches like SBI [ 31 ] and SLADD [ 8 ] use sophisticated augmentations to train on diverse synthetic forgeries, while SeeABLE [ 21 ] reframes the problem as one-class anomaly detection. TALL transforms video clips into thumbnails, efficiently capturing spatio-temporal inconsistencies. Datasets like FaceForensics++ [ 29 ] , Celeb-DF [ 23 ] , and DFDC [ 13 ] have been instrumental in advancing these techniques. While achieving notable success in controlled environments, these video-level detectors struggle under partial multimodal forgeries. Multimodal Deepfake Detection. The emergence of audio-visual deepfakes, where both visual and auditory streams are potentially manipulated, has necessitated the development of multimodal detection approaches. These methods look for inconsistencies across modalities, as maintaining perfect synchronization and consistency between visual and audio streams is still challenging for deepfake generators. Existing multimodal detectors such as those by Chugh et al. [ 10 ] and Cai et al. [ 7 ] , typically concatenate or employ attention mechanisms to combine features extracted from raw audio and visual signals. However, directly processing raw signals can lead to overfitting to specific artifacts, reduced robustness to real-world distortions, and may not capture high-level semantic inconsistencies effectively. Some approaches use pre-extracted features, but rely on general-purpose encoders [ 6 , 39 ] lacking the capacity for speech-related understanding, which is crucial for the task. Temporal Deepfake Localization. Beyond binary classification (real vs. fake), the temporal localization of manipulated segments is important for forensic analysis, as it enables more fine-grained and transparent decisions about suspicious content. Recent works address this challenge, proposing methods that analyze spatio-temporal cross-modal consistency [ 6 , 39 ] . While these methods constitute an important step towards fine-grained deepfake analysis, they utilize general purpose feature extractors limiting their representational capacity. Our work addresses this by modeling frame-level cross-modal discrepancies in the speech latent space. DiMoDif [ 20 ] also considers speech-related features for cross-modal incongruity identification; however, our framework is fundamentally different leveraging cross-modal reconstruction discrepancy instead of plain features encoding and comparison. 3 Methodology In this section we elaborate on the proposed framework AuViRe, which is illustrated in Fig. 1 . Figure 1 : The proposed AuViRe architecture. 3.1 Problem Formulation Consider a dataset \mathcal{V} comprising pairs of video and ground truth (\mathbf{v},\mathbf{y}) . Each input video \mathbf{v} has a duration of \Delta seconds and consists of a visual stream \boldsymbol{\Phi}\in\mathbb{R}^{t\times h\times w\times c} and an audio stream \mathbf{h}\in\mathbb{R}^{s\times n} . Specifically, \boldsymbol{\Phi} contains t frames, each of size h\times w\times c , while \mathbf{h} contains s samples across n channels. The corresponding ground-truth annotation \mathbf{y}\in\mathbb{R}^{t\times 3} is a concatenation of a per-frame manipulation target p\in\{0,1\}^{t} and manipulation boundary information b\in[0,\Delta]^{t\times 2} . For each frame \tau\in[0,t] , p^{\tau}=1 indicates manipulation, and b^{\tau} specifies the start and end times of the manipulated segment it belongs to, being valid for manipulated frames only. Our goal is to train a model f(\mathbf{v})\rightarrow\mathbf{\hat{y}} to temporally localize the manipulated parts of deepfake videos. 3.2 Visual and Audio Speech Representations Our architecture first extracts visual \mathbf{x}_{v}\in\mathbb{R}^{t\times d} and audio \mathbf{x}_{a}\in\mathbb{R}^{t\times d} representations from a self-supervised model [ 30 ] 1 1 1 We consider the AV-Hubert Base/LRS3/No-finetuning version from https://facebookresearch.github.io/av_hubert/ . , \mathfrak{B} , primarily designed for speech recognition, to incorporate multimodal speech information in the processing pipeline. Specifically, the visual stream \boldsymbol{\Phi} and audio stream \mathbf{h} of the video are processed separately by the backbone model avoiding modality blending that could eliminate fake samples’ cross-modal discrepancies: \mathbf{x}_{v}=\mathfrak{B}(\boldsymbol{\Phi},\mathtt{null}),\quad\mathbf{x}_{a}=\mathfrak{B}(\mathtt{null},\mathbf{h}) (1) \mathfrak{B} is a self-supervised model that uses separate audio and visual encoders feeding into

Read the complete text