AdaptPrompt : Parameter-Efficient Adaptation of VLMs for Generalizable Deepfake Detection

Yichen Jiang, Mohammed Talha Alam, Sohail Ahmed Khan, Duc-Tien Dang-Nguyen, Fakhri Karray · 2025

Read AdaptPrompt : Parameter-Efficient Adaptation of VLMs for Generalizable Deepfake Detection by Yichen Jiang, Mohammed Talha Alam, Sohail Ahmed Khan, Duc-Tien Dang-Nguyen, Fakhri Karray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17730.txt.

Introduction

Yichen Jiang and four co-authors retrain a fake-image detector so it stops memorizing one generator's fingerprints. A detector raised on GAN images learns GAN blemishes as the meaning of fake, then waves through diffusion and commercial tool output. The team first builds Diff-Gen, a balanced set of 100,000 diffusion images plus matching real photos, whose frequency signature differs sharply from GAN data.

On that diet they adapt the CLIP model with AdaptPrompt, tuning only about 0.1 percent of its weights through a small visual adapter and learnable text prompts. Cutting CLIP's last block, oddly, helped too. Across 25 test sets, from GANs to Midjourney and DALL-E 3, the detector topped rivals at 98.60 percent mean average precision. Cheap tuning, broad vigilance.

Catalog record

CREATORYichen Jiang, Mohammed Talha Alam, Sohail Ahmed Khan, Duc-Tien Dang-Nguyen, Fakhri Karray
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17730

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaptPrompt : Parameter-Efficient Adaptation of VLMs for Generalizable Deepfake Detection Yichen Jiang, Mohammed Talha Alam, Sohail Ahmed Khan, Duc-Tien Dang-Nguyen, Fakhri Karray Abstract . Recent advances in image generation have led to the widespread availability of highly realistic synthetic media, increasing the difficulty of reliable deepfake detection. A key challenge is generalization, as detectors trained on a narrow class of generators often fail when confronted with unseen models. In this work, we address the pressing need for generalizable detection by leveraging large vision-language models, specifically CLIP, to identify synthetic content across diverse generative techniques. First, we introduce Diff-Gen , a large-scale benchmark dataset comprising 100k diffusion-generated fakes that capture broad spectral artifacts unlike traditional GAN datasets. Models trained on Diff-Gen demonstrate stronger cross-domain generalization, particularly on previously unseen image generators. Second, we propose AdaptPrompt , a parameter-efficient transfer learning framework that jointly learns task-specific textual prompts and visual adapters while keeping the CLIP backbone frozen. We further show via layer ablation that pruning the final transformer block of the vision encoder enhances the retention of high-frequency generative artifacts, significantly boosting detection accuracy. Our evaluation spans 25 challenging test sets, covering synthetic content generated by GANs, diffusion models, and commercial tools, establishing a new state-of-the-art in both standard and cross-domain scenarios. We further demonstrate the framework’s versatility through few-shot generalization (using as few as 320 images) and source attribution, enabling the precise identification of generator architectures in closed-set settings. Figure 1. Comparative visualization of the training datasets. The top row displays random samples from our proposed Diff-Gen dataset, generated via diffusion models, while the bottom row shows samples from the GAN-based ProGAN dataset. Both datasets share an identical class distribution covering 20 object categories (e.g., airplane, bird, bottle) to ensure fair comparison. Visually, Diff-Gen introduces distinct high-frequency noise artifacts compared to the structural periodic artifacts typical of ProGAN, challenging the detector to generalize beyond GAN-specific fingerprints. Figure 2. Performance landscape of state-of-the-art deepfake detectors. This bubble chart plots Average Precision (AP) against Accuracy on the combined test set. The size of each bubble corresponds to the relative size of the training dataset used. Our proposed method, AdaptPrompt trained on Diff-Gen (green bubble), achieves the optimal trade-off in the top-right corner, demonstrating superior efficiency and performance compared to fully fine-tuned models and other parameter-efficient baselines. 1. Introduction Deepfake technology has revolutionized modern media, but this advancement poses significant security risks, ranging from misinformation campaigns to privacy violations ( 4 ; 18 ; 48 ; 49 ) . As synthesis techniques evolve from Generative Adversarial Networks (GANs) ( 24 ) to Diffusion Models (e.g., Stable Diffusion, Midjourney) ( 46 ) , a critical "generalization gap" has emerged. Traditional detectors, trained primarily on GAN-generated imagery, struggle to generalize to diffusion-based synthetic media ( 11 ; 3 ; 54 ; 6 ) . This failure stems from the "sink label" problem: binary classifiers often learn specific GAN fingerprints (e.g., checkerboard artifacts) as the definition of "fake," consequently misclassifying unseen diffusion artifacts as "real" ( 39 ) . To address this, researchers have turned to Vision-Language Models (VLMs) like CLIP, leveraging their robust pre-trained feature spaces. While recent works have explored adapting VLMs via linear probing or fine-tuning ( 39 ; 35 ) , these methods either fail to fully unlock the model’s potential or suffer from overfitting. We hypothesize that a synergistic approach that combines visual adaptation to capture pixel-level artifacts and textual prompt tuning to align semantic anomalies can yield a more robust detector. Furthermore, the choice of training data is pivotal. Existing detectors trained on GAN-generated images fail to capture the high-frequency noise residuals characteristic of diffusion models. This raises a critical question: can a detector trained on diffusion-based noise patterns generalize backward to GANs and forward to commercial image generation tools more effectively than traditional approaches? To answer this, we introduce Diff-Gen , a large-scale dataset comprising 100k diffusion-generated images that mirror the class distribution of the LSUN ( 53 ) real dataset, shown in Fig. 1 . Unlike GANs, which leave distinct periodic fingerprints, diffusion models exhibit noise-like artifacts that existing datasets fail to capture. We further propose AdaptPrompt , a parameter-efficient transfer learning framework that optimizes both the visual and textual pathways of CLIP while keeping the backbone frozen. We conduct a comprehensive empirical analysis across 25 diverse test datasets, covering GANs, diffusion models, and commercial generators. Our contributions are summarized as follows: • We introduce Diff-Gen , a novel training benchmark that captures the broad spectral characteristics of diffusion noise. We demonstrate that models trained on Diff-Gen exhibit significantly stronger cross-domain generalization compared to those trained on ProGAN ( 30 ) . • We propose AdaptPrompt , a unified transfer learning strategy that combines adapter networks and prompt tuning. We show via ablation that pruning the final transformer block of the vision encoder enhances the retention of high-frequency generative artifacts, boosting detection accuracy. • We provide a comprehensive evaluation on 25 unseen datasets, establishing a new state-of-the-art for generalizable deepfake detection. In addition, we extend our framework to the task of source attribution, demonstrating high accuracy in identifying specific generator architectures. This paper is organized as follows. Section 2 reviews related work in generative forensics and VLM adaptation. Section 3 details the AdaptPrompt methodology and the Diff-Gen dataset. Section 4 presents extensive experimental results, including spectral analysis and robustness checks. Finally, Section 5 concludes the study. 2. Related Work 2.1. Evolution of Generative Forensics The rapid evolution of generative models has necessitated a parallel evolution in forensic detection. Early synthetic media generation relied on modifying existing images, such as facial manipulation or Photoshop editing ( 51 ; 43 ; 7 ) . With the advent of Generative Adversarial Networks (GANs) ( 24 ) , research shifted toward identifying model-specific fingerprints ( 5 ) . Studies have demonstrated that GAN-generated images exhibit distinct spectral artifacts, often visible as periodic patterns in the frequency domain due to upsampling operations ( 15 ; 16 ; 22 ; 1 ; 2 ) . Consequently, many detectors focused on these visible traces ( 19 ; 20 ; 36 ) or manipulated facial regions ( 47 ) . However, the emergence of Diffusion Models ( 50 ) has complicated this landscape. Unlike GANs, diffusion models generate images via iterative denoising, resulting in fingerprints that resemble high-frequency Gaussian noise rather than structural periodicity. This shift renders frequency-based detectors less effective on modern deepfakes (e.g., Stable Diffusion ( 46 ) , DALL-E ( 41 ) ), highlighting the need for more agnostic detection features. 2.2. The Generalization Dilemma A critical challenge in deepfake detection is the "generalization dilemma" ( 34 ; 56 ; 11 ) , where classifiers perform well on seen generator architectures but fail on unseen ones. Wang et al. ( 52 ) pioneered the study of cross-generator generalization, training a ResNet-50 ( 27 ) classifier on a large-scale dataset of 720,000 images, comprising 360,000 ProGAN fakes ( 30 ) and 360,000 LSUN real images ( 53 ) . While this approach generalized effectively to other GAN variants, subsequent work by Gragnaniello et al. ( 25 ) and Corvi et al. ( 13 ) revealed that such models struggle significantly when tested on diffusion-generated content.

Read the complete text