OpenSDI: Spotting Diffusion-Generated Images in the Open World
Yabin Wang, Zhiwu Huang, Xiaopeng Hong · 2025
Read OpenSDI: Spotting Diffusion-Generated Images in the Open World by Yabin Wang, Zhiwu Huang, Xiaopeng Hong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19653.txt.
Introduction
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong tackle fake image spotting in the open world in a 2025 paper. Diffusion models can now edit real photos in ways that fool the eye. Past test sets are too narrow to match real abuse. The team defines a new task, called OpenSDI, for this wider test.
They built a dataset that mimics open-world edits with large vision-language models. Tasks cover both spotting fake images and marking the fake parts. Their model, MaskCLIP, joins two pretrained base models to share strengths. On the new dataset it beat the best prior methods by clear margins. Gains were largest on the marking task. The dataset and code are shared in public.
Catalog record
| CREATOR | Yabin Wang, Zhiwu Huang, Xiaopeng Hong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OpenSDI: Spotting Diffusion-Generated Images in the Open World Yabin Wang, Zhiwu Huang, Xiaopeng Hong Abstract This paper identifies OpenSDI, a challenge for spotting diffusion-generated images in open-world settings. In response to this challenge, we define a new benchmark, the OpenSDI dataset (OpenSDID), which stands out from existing datasets due to its diverse use of large vision-language models that simulate open-world diffusion-based manipulations. Another outstanding feature of OpenSDID is its inclusion of both detection and localization tasks for images manipulated globally and locally by diffusion models. To address the OpenSDI challenge, we propose a Synergizing Pretrained Models (SPM) scheme to build up a mixture of foundation models. This approach exploits a collaboration mechanism with multiple pretrained foundation models to enhance generalization in the OpenSDI context, moving beyond traditional training by synergizing multiple pretrained models through prompting and attending strategies. Building on this scheme, we introduce MaskCLIP, an SPM-based model that aligns Contrastive Language-Image Pre-Training (CLIP) with Masked Autoencoder (MAE). Extensive evaluations on OpenSDID show that MaskCLIP significantly outperforms current state-of-the-art methods for the OpenSDI challenge, achieving remarkable relative improvements of 14.23% in IoU (14.11% in F1) and 2.05% in accuracy (2.38% in F1) compared to the second-best model in localization and detection tasks, respectively. Our dataset and code are available at https://github.com/iamwangyabin/OpenSDI . 1 Introduction Figure 1 : Three open-world settings of the OpenSDI challenge: (1) user diversity by simulating a range of user preferences, (2) model innovation through the use of multiple advanced diffusion models, and (3) a full manipulation scope that enables both global and local generation. The rise of advanced text-to-image (T2I) generation models, such as Stable Diffusion [ 36 , 10 , 33 ] and other diffusion-based frameworks [ 21 , 47 , 61 ] , has profoundly transformed digital content creation, enabling users to produce remarkably lifelike images from simple text prompts. This capability, while expanding creative possibilities, has blurred the boundary between real and AI-generated content, making it increasingly difficult to ensure content authenticity. Addressing this challenge requires detectors that can handle three essential open-world settings (Figure 1 ): (1) the wide range of user preferences ( user diversity ), covering diverse styles, subjects, and creative intentions; (2) the rapid evolution of diffusion models ( model innovation ), introducing significant variability in generated images; and (3) the extensive T2I generation ( manipulation scope ), from a broad spectrum of global image synthesis to precise local changes, object insertions, and background manipulations [ 62 , 46 , 63 ] . We identify this challenge as Open-world Spotting of Diffusion Images (OpenSDI) . In response to the OpenSDI challenge, we introduce OpenSDID, a large-scale dataset specifically designed to address the scarcity of suitable public datasets for benchmarking OpenSDI ( Table 1 ). Our approach incorporates the three essential dimensions of OpenSDI to create a comprehensive and realistic dataset. To reflect user diversity , we use multiple large vision-language models (VLMs) [ 44 , 24 , 1 , 30 ] to generate a broad range of natural language prompts, simulating how users iteratively refine their creative ideas through diverse, open-world textual descriptions. These prompts are designed to mirror real-world user behavior and preferences, capturing the variety seen in different creative intentions and styles. To address model innovation , we incorporate state-of-the-art (SOTA) T2I diffusion models [ 36 , 33 , 10 , 21 ] , each with distinct architectures and parameter configurations. This variety in diffusion models allows our dataset to encompass the unique visual characteristics and nuances associated with advanced diffusion models, enhancing the dataset’s relevance for evaluating detection and localization across a spectrum of T2I diffusion models. Finally, for manipulation scope , we leverage the semantic understanding of VLMs in combination with Segment Anything Model (SAM) [ 16 ] and Florence 2 [ 49 ] to translate textual instructions into precise region masks. These masks enable a wide range of manipulations, from local region edits to global image synthesis, simulating the types of modifications that make detection challenging in open-world settings. By combining global and localized manipulations, OpenSDID aims at embodying the complexity required for benchmarking the OpenSDI challenge. To develop effective approaches for addressing OpenSDI, we face two main barriers: (1) unifying detection and localization frameworks, and (2) generalizing to open-world settings. The unification is challenging due to the high heterogeneity of detection and location. Detection relies on image-level semantics, whereas localization requires pixel-level precision. To tackle these issues, we suggest an approach of building up a mixture of pretrained foundation models. While employing multiple expert models holds significant potential, achieving synergy among them remains a complex challenge. To address this, we introduce a novel framework, Synergizing Pretrained Models (SPM). It is designed to optimize the collaboration between pretrained foundation models for enhanced performance and maintaining generalization capability across both detection and localization tasks in open-world settings. In particular, this approach leverages the strengths of multiple pretrained models through prompt tuning and attention mechanisms, creating a mixture of pretrained foundation models that enhances generalization across diverse real-world scenarios. Building on this SPM approach, we introduce MaskCLIP, a mixture of pretrained models that aligns the capabilities of Contrastive Language-Image Pre-Training (CLIP) with a Masked Autoencoder (MAE) model, outperforming SOTA techniques with considerable relative improvements in both detecting and localizing diffusion-generated image content on the suggested OpenSDI dataset. In summary, our main contributions are as follows: • We define the OpenSDI challenge, and introduce a large-scale dataset (OpenSDID) designed to address the lack of suitable public datasets for OpenSDI. • We introduce an SPM framework and its derived MaskCLIP to effectively address the OpenSDI challenge. • Through extensive experiments, MaskCLIP demonstrates markedly superior performance compared to SOTA methods for OpenSDI. 2 Related Works Dataset # Real # Fake User Model Full DiffusionDB [ 46 ] – 14M ✓ ✗ ✗ GenImage [ 52 ] 1.3M 1.4M ✗ ✓ ✗ AutoSplice [ 14 ] 2.3K 3.6K ✗ ✗ ✓ CocoGlide [ 12 ] – 512 ✗ ✗ ✗ HiFi-IFDL [ 13 ] – 1M* ✓ ✗ ✓ GIM [ 5 ] 1.1M 1.1M ✗ ✓ ✗ TGIF [ 29 ] 3.1K 75K ✗ ✓ ✗ OpenSDID 300K 450K ✓ ✓ ✓ Table 1 : Overview of existing diffusion image datasets and the proposed OpenSDID. User: Simulates diverse preferences. Model: Leverages multiple advanced diffusion models published after 2023. Full: Includes both global and local generation. Note: The asterisk (*) indicates HiFi-IFDL is built from multiple prior datasets. A more comprehensive version is in the suppl. material. Diffusion Image Benchmark Datasets. Existing datasets fall short in addressing the OpenSDI challenge due to their significant limitations. Many focus on traditional manipulation techniques such as splicing and copy-move operations. Widely used datasets like Columbia [ 18 ] , NIST MFC [ 11 ] , and CASIA [ 9 ] concentrate on these conventional methods. HiFi-IFDL [ 13 ] created a million-scale dataset that combines various generative models and datasets. However, the manipulation types in HiFi-IFDL are mainly centered around traditional methods and facial editing, and do not involve recent T2I diffusion generators. To the best of our knowledge, only a few datasets incorporate text-guided editing using diffusion models such as DiffusionDB [ 46 ] , GenImage [ 52 ] , CocoGlide [