Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection

Peipeng Yu, Jianwei Fei, Hui Gao, Xuan Feng, Zhihua Xia, Chip-Hong Chang · 2025

Read Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection by Peipeng Yu, Jianwei Fei, Hui Gao, Xuan Feng, Zhihua Xia, Chip-Hong Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14853.txt.

Introduction

Peipeng Yu and co-authors aim big vision-language models at deepfake spotting in this 2025 paper. Such models read images and text well, yet their stock knowledge does not line up with forgery clues. The team builds a frame that retunes them for the task.

One part scores how image traits match true and fake captions, which both sorts and locates fakes. A second part turns those scores into fine prompt codes. A language model then reads the codes and writes a verdict in words. Tests across six shared sets beat past best methods on new-set scores. The setup also supports back-and-forth chats about a verdict, which aids trust.

Catalog record

CREATORPeipeng Yu, Jianwei Fei, Hui Gao, Xuan Feng, Zhihua Xia, Chip-Hong Chang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection Peipeng Yu, Jianwei Fei, Hui Gao, Xuan Feng, Zhihua Xia, Chip-Hong Chang Abstract Current Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in understanding multimodal data, but their potential remains underexplored for deepfake detection due to the misalignment of their knowledge and forensics patterns. To this end, we present a novel framework that unlocks LVLMs’ potential capabilities for deepfake detection. Our framework includes a Knowledge-guided Forgery Detector (KFD), a Forgery Prompt Learner (FPL), and a Large Language Model (LLM). The KFD is used to calculate correlations between image features and pristine/deepfake image description embeddings, enabling forgery classification and localization. The outputs of the KFD are subsequently processed by the Forgery Prompt Learner to construct fine-grained forgery prompt embeddings. These embeddings, along with visual and question prompt embeddings, are fed into the LLM to generate textual detection responses. Extensive experiments on multiple benchmarks, including FF++, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme surpasses state-of-the-art methods in generalization performance, while also supporting multi-turn dialogue capabilities. 1 Introduction The rapid advancement of generative artificial intelligence has significantly accelerated the development of deepfake technology, facilitating realistic facial manipulation and reenactment. While these technologies have notable applications in the entertainment and art fields, such as Stable Diffusion ( Esser et al., 2024 ) and DALL·E ( Ramesh et al., 2021 ) , their misuse poses critical security risks to society ( Wang et al., 2024b ) . These tools allow users to synthesize realistic but nonexistent content by merely inputting carefully designed prompts, making deepfake generation more accessible and potentially dangerous than ever before. Large Vision-Language Models (LVLMs) present a promising leverage to this issue. Pre-trained on extensive and diverse datasets, LVLMs capture vast amounts of knowledge about natural objects, providing significant potential to improve generalization in identifying manipulated content. An LVLM typically uses an image encoder to extract image features, which are then combined with text prompts and input into the Large Language Model (LLM) to generate responses. For instance, inputting an image along with a prompt like “ This is a facial image designed for deepfake detection, and it should not exhibit any localized color discrepancies or evident signs of splicing. Is this a deepfake image? ” allows the LVLM to assess potential manipulations. However, existing LVLMs are primarily optimized for general image understanding tasks and may not effectively capture the fine-grained features required for deepfake detection. Directly performing fine-tuning presents challenges, as the LVLM may struggle to interpret specialized terms like “ color discrepancies ” or “ visual artifacts ” as intended in the context of forgery detection. Therefore, it is crucial to design fine-grained prompt embeddings to facilitate LVLM training. Figure 1: Comparison of existing deepfake detection methods. Existing approaches can perform localization and classification but ignore integrating external knowledge. Our scheme introduces a Knowledge-guided detection module to enhance generalization capabilities, and enables multi-turn dialogues. In this paper, we aim to unlock the capabilities of Large Vision-Language Models for deepfake detection tasks. Humans naturally use specific descriptors, such as subtle visual artifacts , localized lighting inconsistencies , and overly smoothed textures , to characterize manipulated content. However, these features are difficult to accurately replicate by data simulation or feature augmentation alone, limiting the ability of current methods to fully interpret manipulated content ( Zhang et al., 2024 ) . To address this limitation, we propose to explore the correction between images and textual descriptions to assist deepfake detection. As shown in Figure 1 , we propose a novel LVLM-based deepfake detection framework under the guidance of pretrained knowledge to perform generalizable and explainable detection ( Lester et al., 2021 ; Liu et al., 2022 ) . We first integrate external knowledge to train a forgery detector, and then incorporate its features into LLM to generate responses. Specifically, the framework comprises two key stages: Knowledge-guided Forgery Detector Training and LLM Prompt Tuning. In the Knowledge-guided Forgery Detector Training phase, we aim to train a high-precision deepfake detector. Leveraging pre-trained multimodal encoders, we extract image features from images and textual features from descriptions of pristine and forged images. By calculating the correlation between these image features and description text embeddings, we generate consistency maps that represent the alignment between visual content and textual descriptions. These maps are subsequently processed by the Forgery Locator and Classifier to produce a forgery segmentation map and forgery score. Subsequently, in the LLM Prompt Tuning phase, we incorporate deepfake detection knowledge into the LLM to generate forgery detection descriptions. To ensure accurate deepfake detection, we train the LVLM using simulated forgery image-text pairs specifically tailored for this task. Our contributions are summarized as follows: • We propose a novel LVLM-based framework for deepfake detection that integrates fine-grained forgery prompt embeddings through prompt tuning, which significantly enhances model generalization and explainability. • We introduce a Knowledge-guided Forgery Detector to generate forgery consistency maps to align image features with description text embeddings of both pristine and deepfake images for enhanced generalization. • Extensive experiments on multiple benchmarks, including FF++, CDF1, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme outperforms existing methods in generalization performance, with the added capability of supporting multi-turn dialogue. 2 Related Works Deepfake Detection Methods: Conventional classification architectures have achieved significant success in detecting forgery clues in early deepfake content. Various strategies, such as data augmentation ( Li et al., 2020a ; Shiohara & Yamasaki, 2022 ; Nguyen et al., 2024 ) , feature consistency analysis ( Zhao et al., 2021b ; Yan et al., 2023 ) , and frequency domain anomaly detection ( Jeong et al., 2022 ; Liu et al., 2021 ; Wang et al., 2023a ) , have been explored in recent years to enhance the generalization of deepfake detection models. While these methods achieve a high detection accuracy, they primarily rely on data augmentation or enhanced feature extraction ( Yan et al., 2024 ) , and often neglect the integration of external human knowledge. However, many deepfake characteristics are embedded in human knowledge, which is challenging to capture through data or feature augmentation alone. This limitation significantly constrains the generalization capabilities of existing algorithms. In this paper, we propose an LVLM-based deepfake detection framework that aligns image features with real/fake descriptions to enhance the model’s capacity to detect unseen deepfakes. Large Vision-Language Models: Recent advancements in Large Vision-Language Models (LVLMs) have showcased their potential in multimodal tasks ( Gunjal et al., 2024 ; Leng et al., 2024 ; Gu et al., 2024 ) . A typical LVLM architecture comprises an image encoder, a projector, and a LLM. The image encoder extracts visual features from input images, which are then transformed by the projector into visual prompt embeddings. These visual embeddings, combined with textual prompt embeddings, are fed into the LLM to generate responses. Building on this architecture, models such as BLIP-2 ( Li et al., 2023 ) , LLaVA ( Liu et al., 2024a ) , and MiniGPT-4 ( Zhu et al., 2024 ) have achieved notable advancements in language instruction following ( Su et al., 2023 ; Yang et al., 2024 ) and visual reasoning ( Chen et al., 2024 ) for natural scenes.

Read the complete text