Visual Language Models as Zero-Shot Deepfake Detectors

Viacheslav Pirogov · 2025

Read Visual Language Models as Zero-Shot Deepfake Detectors by Viacheslav Pirogov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22469.txt.

Introduction

Viacheslav Pirogov tests sight-language models as fake face spotters in a 2025 paper. Fake faces, swapped by smart tools, threaten media trust and ID checks alike. Most spotters train a lone sorter on pixels, with no side task to harden them when fakes shift style.

Pirogov asks big sight-language models to sort faces cold, with no task training at all. He built a fresh set of 60,000 images for the test, half true and half fake in high grade. The cold models beat nearly all past trained spotters on the set. The work hints that broad models guard media best.

Catalog record

CREATORViacheslav Pirogov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22469

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Language Models as Zero-Shot Deepfake Detectors Viacheslav Pirogov Abstract The contemporary phenomenon of deepfakes, utilizing GAN or diffusion models for face swapping, presents a substantial and evolving threat in digital media, identity verification, and a multitude of other systems. The majority of existing methods for detecting deepfakes rely on training specialized classifiers to distinguish between genuine and manipulated images, focusing only on the image domain without incorporating any auxiliary tasks that could enhance robustness. In this paper, inspired by the zero‑shot capabilities of Vision–Language Models, we propose a novel VLM‑based approach to image classification and then evaluate it for deepfake detection. Specifically, we utilize a new high‑quality deepfake dataset comprising 60,000 images, on which our zero‑shot models demonstrate superior performance to almost all existing methods. Subsequently, we compare the performance of the best-performing architecture, InstructBLIP, on the popular deepfake dataset DFDC-P against traditional methods in two scenarios: zero-shot and in-domain fine-tuning. Our results demonstrate the superiority of VLMs over traditional classifiers. 1 Introduction Deepfakes have evolved from a technological curiosity to an everyday reality. Just a few years ago, producing a high-quality, AI-generated image required specialized knowledge, powerful hardware and strong technical skills, so only a handful of enthusiasts could do it. However, thanks to the growing popularity of AI and the democratization of computing resources such as Google Colab ( Google LLC 2020 ) , as well as dozens of ready-made tools ( ComfyUI 2023 ; OpenAI 2022 ; Google LLC 2025 ) , almost anyone with a laptop, or even a smartphone, can now create deepfakes for free or for the price of a small subscription. Most of this AI-generated content appears on social media platforms, where people use it purely for entertainment. However, deepfakes also pose serious dangers, particularly with regard to Know Your Customer (KYC) onboarding and liveness verification. According to the Sumsub report ( Sumsub 2024 ) , the number of detected deepfakes increased fourfold between 2023 and 2024, accounting for 7% of all fraud attempts. Many studies have emphasized that the problem of detecting deepfakes remains significant and unsolved ( Le et al. 2024 ; Liu et al. 2024b ; Heidari et al. 2024 ; Le et al. 2023 ) . One primary challenge is the absence of a comprehensive dataset that covers all types of deepfake encountered in real life. In addition, existing detectors are fragile: even simple post‑processing such as noise ( Haliassos et al. 2021 ; Jiang et al. 2020 ) or compression ( Le & Woo 2023 ; Le & Woo 2021 ) can break them. While these models achieve high accuracy on their evaluation datasets, they often fail to generalize to new types of deepfake, and not ready for real-world applications, as demonstrated in our parallel work ( Pirogov & Artemev 2025 ) . A truly practical solution would offer a robust zero‑shot or few‑shot method that can effectively detect previously unseen deepfakes. Modern large language models (LLMs) ( Fraser 2023 ; Team 2024a ; DeepSeek-AI et al. 2025 ) are popular precisely because they exhibit strong zero‑shot and generalization abilities. Visual Language Models (VLMs) with instruction tuning ( Dai et al. 2023 ; Liu et al. 2023 ; Laurençon et al. 2024 ) demonstrate comparable potential. In natural‑language processing, most downstream systems now build on huge pretrained backbones and adapt them with lightweight methods. Therefore, a similar trend could be expected in computer vision, with VLMs fitting best in the role of such a backbone. Previous studies have demonstrated the potential of both open-source and closed-source Visual Language Models (VLMs) in the task of deepfake detection ( Chang et al. 2023 ; Zhang et al. 2024 ; Li et al. 2024 ; Shi et al. 2024 ; Jia et al. 2024 ) . However, these studies have not fully explored the pure zero-shot capabilities of VLMs and have not focused on integrating such models into real-life systems, such as liveness checks and verifications. In this paper, we introduce a VLM‑based image‑classification framework and evaluate it on deepfake detection. Using a new high‑quality deepfake dataset, we benchmark state‑of‑the‑art detectors together with open and closed source VLMs in zero‑shot and few‑shot settings, showing that VLMs achieve the best out‑of‑distribution performance. Furthermore, we demonstrate that, when the data are in-distribution, simple language fine-tuning enables the leading VLM to achieve near-perfect performance; we illustrate this using the widely adopted deepfake dataset DFDC-P ( Dolhansky et al. 2020 ) . 2 Related work This section provides an overview of the techniques used to generate deepfakes in the modern world, with a particular focus on face swapping between two individuals 2.1 and methods for detecting such deepfakes 2.2 . Subsequently, we present the current state of Visual Language Models (VLMs), with a particular focus on the VLMs that we have utilized 2.3 . In the last subsection 2.4 , we offer a comprehensive overview of existing research that employs VLMs as deepfake detectors. 2.1 Deepfake Generation Face swapping approaches can be categorized based on the number of images required for the source and target faces. These approaches range from using large datasets, commonly referred to as ”facesets”, to employing few-shot or one-shot methods. The most powerful method utilizing large datasets is DeepFaceLab ( Perov et al. 2020 ) . In contrast, few-shot or one-shot methods ( Chen et al. 2020 ; Nirkin et al. 2019 ; Nirkin et al. 2022 ; Li et al. 2019 ; Jia Guo 2017 ) are the most popular today within the community ( Sangwan 2023a ; Sangwan 2023b ; C0untFloyd 2023 ; machineminded 2024 ) and among fraudsters due to their ease of use and low resource requirements. SimSwap : An Efficient Framework For High Fidelity Face Swapping is a state-of-the-art (SOTA) open-source model for high-fidelity one-shot face swapping, requiring only one image each from source and target. This model supports relatively high resolutions with two variants: one at 224x224 pixels, trained on the VGGFace2 ( Cao et al. 2018 ) dataset, and other at 512x512, trained on enhanced VGGFace2-HQ dataset. SimSwap utilizes an adversarially trained encoder-decoder architecture, augmented by an Identity Injection Module, which separates image attributes and identity, thereby enabling the transfer of only the attributes. To this end, a face recognition network ( Deng et al. 2019 ) is employed to extract an identity embedding, which is then integrated via Adaptive Instance Normalization (AdaIN) ( Huang & Belongie 2017 ) . The model is trained in a GAN style ( Goodfellow et al. 2014 ; Liu et al. 2019 ; Brock et al. 2019 ; Gulrajani et al. 2017 ; Karras et al. 2019 ; Isola et al. 2017 ) to ensure the realism of generated images, with the objective of achieving an indistinguishable result from that of the original. 2.2 Deepfake Detection For the last five years, researchers have been actively working on deepfake detection methods. They usually propose new datasets that better represent real-life scenarios than previous ones ( Rossler et al. 2019 ; Dolhansky et al. 2020 ; Li et al. 2020 ; Jiang et al. 2020 ; Shiohara & Yamasaki 2022 ) or introduce new analytical approaches such as novel architectures ( ( Zhao et al. 2021 ; Wang et

Read the complete text