Referee: Reference-aware Audiovisual Deepfake Detection
Hyemin Boo, Eunsang Lee, Jiyoung Lee · 2025
Read Referee: Reference-aware Audiovisual Deepfake Detection by Hyemin Boo, Eunsang Lee, Jiyoung Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27475.txt.
Introduction
Hyemin Boo, Eunsang Lee, and Jiyoung Lee catch fake videos by checking identity, in a 2025 paper. Deepfakes swap faces or voices in clips, and detectors trained on one trick often fail on the next. Their detector, Referee, anchors on one real example of the person. It asks whether the clip's speaker cues still match that anchor.
Two modules do the work: one squeezes identity into a tight bottleneck form, the other matches clip cues against the anchor. Most rivals chase brief visual glitches that fade as fakes improve. Identity mismatch is harder for fakers to hide. On three test sets, including cross-language trials, Referee led the field. On the Korean KoDF set it scored near the top of the scale. The code is public.
Catalog record
| CREATOR | Hyemin Boo, Eunsang Lee, Jiyoung Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27475
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Referee: Reference-aware Audiovisual Deepfake Detection Hyemin Boo, Eunsang Lee, Jiyoung Lee Abstract Since deepfakes generated by advanced generative models have rapidly posed serious threats, existing audiovisual deepfake detection approaches struggle to generalize to unseen forgeries. We propose a novel reference-aware audiovisual deepfake detection method, called Referee . Speaker-specific cues from only one-shot examples are leveraged to detect manipulations beyond spatiotemporal artifacts. By matching and aligning identity-related queries from reference and target content into cross-modal features, Referee jointly reasons about audiovisual synchrony and identity consistency. Extensive experiments on FakeAVCeleb, FaceForensics++, and KoDF demonstrate that Referee achieves state-of-the-art performance on cross-dataset and cross-language evaluation protocols. Experimental results highlight the importance of cross-modal identity verification for future deepfake detection. The code is available at https://github.com/ewha-mmai/referee . † † address: Ewha Womans University, Republic of Korea 1 Introduction With the rapid development of generative AI, recent models are increasingly capable of synthesizing media content, including both video and audio. However, this advancement carries a dual nature: while it enables creative and beneficial applications, it also introduces significant risks of misuse and societal harm. The risk has arisen from synthesized media that convincingly imitate humans, known as deepfake [ 1 ] . The goal of deepfake detection is to distinguish genuine ( i.e. , real) or spoof ( i.e. , fake) content. Prior works have relied on a single modality. Image-based approaches have focused on detecting shape distortion [ 2 , 3 ] , boundary artifacts [ 4 , 5 , 6 ] , or anomalies in the frequency domain [ 7 , 8 ] that arise during the synthesis process. On the other hand, audio-based approaches [ 9 , 10 ] have learned discriminative features to capture vulnerable spectral patterns. Although these methods have shown promising performance, the synthetic contents are generated with higher fidelity, making unimodal detection strategies less reliable. On the other hand, audiovisual deepfake detection [ 11 , 12 , 13 ] has exploited the natural correspondence between facial movements and speech. The dynamic motion patterns [ 14 , 15 , 16 ] , such as lip synchronization, or unnatural temporal transitions, have been investigated to capture manipulation cues. However, such methods often fail when the lips are partially occluded by hands, microphones, or head poses, and their performance further degrades under low-resolution conditions. Figure 1: Referee introduces robust audiovisual deepfake detection guided by a one-shot reference example, verifying the cross-modal biometrics consistency as well as temporal unnaturalness. When humans encounter a video, they do not fixate on lip motion in isolation; instead, they weave together subtle cues, e.g. , tone of voice, facial appearances, and identity coherence, to form a judgment of authenticity. Motivated by this natural ability, we introduce Referee , a novel model that examines the consistency of speakers’ identity for robust audiovisual deepfake detection. In contrast, artifact-based or synchronization-based methods often rely heavily on the generative models used to create deepfakes in the training set, which limits their generalizability to unseen forgeries [ 17 , 18 ] . As generative techniques continue to advance, such low-level visual or acoustic artifacts become increasingly subtle or even imperceptible. By comparison, preserving a speaker’s unique identity across modalities remains a highly challenging task for generative models. While recent work [ 19 ] has proposed a method to measure speaker consistency using contrastive learning, it requires long reference footage ( i.e. , at least 10 samples of 30 seconds per person), which is often impractical in real-world scenarios. Figure 2: The overall framework of Referee . Target and reference videos are encoded into audiovisual features, which are passed through IDB module to generate identity queries. The identity matching module refines the target tokens with respect to the reference tokens, after which the reference-aware identity queries, together with the target audiovisual features and a [CLS] token, are processed by the AV-Transformer for deepfake classification and identity matching. Referee presents an identity bottleneck module (IDB) with learnable identity queries to encode audiovisual sequences into a unified identity representative feature, as shown in Fig. 1 . Identity queries obtained from both reference and target content are subsequently aligned through cross-attention to verify speaker identity. This process is optimized by our ID matching loss to ensure that the models can capture speaker-specific cues rather than superficial artifacts. The fused identity representation is then attached to the audiovisual feature, enabling the system to reason jointly about two critical aspects: (1) whether lip-speech synchronization is consistent over time, and (2) whether either or both modalities have been manipulated in a way that violates identity integrity. Finally, the model integrates these signals to classify the content as real or fake. Extensive experiments show Referee effectively surpasses current state-of-the-art works on FakeAVCeleb [ 20 ] , FaceForensics++ [ 21 ] , and KoDF [ 22 ] . It also demonstrates that Referee is inherently more resilient to the rapid progress of generative models and better equipped to handle unseen types of deepfakes. To sum up, our contributions are: • We propose Referee , a reference-aware deepfake detection method that leverages audiovisual correspondence in both identity- and temporal-levels. • Referee employs a novel identity bottleneck module with learnable queries and a cross-identity matching mechanism, enabling robust reasoning for audiovisual integrity. • Extensive experiments on FakeAVCeleb, FaceForesics++, KoDF demonstrate that our Referee achieves state-of-the-art performance on both unseen forgery manipulations and cross-lingual evaluation. 2 Method 2.1 Motivation and overview Most prior approaches [ 11 , 12 , 13 ] for audiovisual deepfake detection have shown impressive performance while concentrating on correct synchronization between audio and video streams. However, there are several challenges: (1) methods relying on synchronization are inherently tied to lip-sync manipulation methodologies, making it difficult to generalize to unseen forgeries. (2) These methods also remain vulnerable under occlusion, low resolution, or degraded signals. To solve these limitations, we propose Referee to detect forgery with speaker identity information. We claim that cross-modal biometric cues [ 23 , 24 ] , such as verifying the correspondence between the speaker’s voice and facial appearance, provide a strong basis for deepfake detection. Given a target and reference video including audio, Referee detects subtle identity inconsistencies within the same individual. The proposed method mainly consists of three parts; first, we propose an identity bottleneck module, IDB, to capture the cross-modal identity cues into learnable ID queries. Secondly, identity matching layers learn to check the consistency of speaker identity with the compressed ID query tokens, where the target tokens are re-aligned from the perspective of the reference identity. Finally, the refined ID query is concatenated with the audiovisual feature, and jointly learned within the AV-Transformer to classify the video as real or fake. The overall framework is illustrated in Fig. 2 . 2.2 Audiovisual feature extraction Given an input video and audio, our video and audio encoders extract visual and audio features, respectively. In this work, pretrained audiovisual encoders [ 25 ] are used to initialize the parameters. The video and audio are divided into N_{seg} fixed-length segments of length T seconds. From each of the N_{seg} segments, the visual and audio encoders yield segment-level token embeddings. To construct unified representations, these embeddings are concatenated along the temporal axis, forming continuous token sequences for each