Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline
Jingchun Lian, Lingyu Liu, Yaxiong Wang, Yujiao Wu, Lianwei Wu, Li Zhu, Zhedong Zheng · 2024
Read Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline by Jingchun Lian, Lingyu Liu, Yaxiong Wang, Yujiao Wu, Lianwei Wu, Li Zhu, Zhedong Zheng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19685.txt.
Introduction
Jingchun Lian and co-authors set a new task for face forgery research: writing a report on how a face was faked, in a 2024 paper. Most detectors just answer yes or no, or mark the altered pixels. That says little about what was done to the image or why it looks wrong.
Their task joins two halves: locate the forged regions and explain the edit in words. To support it they built a data set of over 152,000 samples, each with a mask from the true editing process and a human-written description. Their baseline system, ForgeryTalker, attempts both halves in one model. Written attribution could help journalists, courts, and platforms judge fakes with evidence, not just a score.
Catalog record
| CREATOR | Jingchun Lian, Lingyu Liu, Yaxiong Wang, Yujiao Wu, Lianwei Wu, Li Zhu, Zhedong Zheng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline Jingchun Lian, Lingyu Liu, Yaxiong Wang, Yujiao Wu, Lianwei Wu, Li Zhu, Zhedong Zheng Abstract Existing facial forgery detection methods typically focus on binary classification or pixel-level localization, providing little semantic insight into the nature of the manipulation. To address this, we introduce Forgery Attribution Report Generation , a new multimodal task designed to provide post-hoc forensic evidence for manipulated images. This task jointly localizes forged regions (“Where”) and generates natural language explanations grounded in the editing process (“Why”). This dual-focus approach goes beyond traditional binary forensics, providing a comprehensive, interpretable understanding of the manipulation. To enable research in this domain, we present Multi-Modal Tamper Tracing (MMTT) , a large-scale dataset of 152,217 samples. Each sample features a process-derived ground-truth mask and a human-authored textual description, ensuring high annotation precision and linguistic richness. We further propose ForgeryTalker , a unified end-to-end baseline that integrates vision and language via a shared encoder and dual decoders for mask and text generation. Experiments show that ForgeryTalker achieves competitive performance on both subtasks, i.e., 59.3 CIDEr and 73.67 IoU, establishing a strong baseline for explainable multimedia forensics. Our dataset and code are available at: https://github.com/NattyLianJc/Generating-Attribution-Reports . 1 Introduction The rapid evolution of advanced generative models, notably diffusion models ( Ho et al., 2020 ; Song et al., 2020 ; Zhang et al., 2025 ) , has significantly enhanced the realism of synthesized images. While promising for creative domains ( Dhariwal and Nichol, 2021 ; Liu et al., 2025a ) , these technologies raise critical concerns regarding their misuse in misinformation and privacy violations ( Rana et al., 2022 ; Liu et al., 2023b ; Zhu et al., 2025b ; Liu et al., 2025b ; Ma et al., 2024 ; Zeng et al., 2024 ) , particularly concerning facial manipulation. In response, detection research has rapidly shifted from binary classification to fine-grained forgery localization to address the growing complexity of modern attacks ( Verdoliva, 2020 ; Rossler et al., 2019 ; Wu et al., 2023 ; Yu et al., 2021 ) . Unlike binary classifiers that merely output a simple decision, forgery localization aims to pinpoint specific tampered regions ( Verdoliva, 2020 ) . However, binary masks alone provide limited interpretability ( Rossler et al., 2019 ) . They treat all manipulated pixels equally, failing to differentiate between subtle and significant alterations or explain the underlying rationale. Furthermore, as modern forgeries become visually indistinguishable from reality, binary masks offer insufficient guidance for human reviewers. Subtle artifacts, such as minute distortions in facial features, are often overlooked, leaving observers without descriptive evidence to verify the detected anomalies and trust the recognition results. To address these limitations, we introduce the novel task of Forgery Attribution Report Generation , aiming to produce a comprehensive report consisting of a pixel-level mask and a textual explanation. To support this, we construct the M ulti- M odal T amper T racing ( MMTT ) dataset, the first large-scale benchmark with 152,217 samples. A key strength of MMTT lies in its high-quality ground truth: pixel-level masks are programmatically derived from the forgery process to ensure perfect alignment, while textual descriptions are crafted through a rigorous human-in-the-loop pipeline to capture subtle artifacts. Building on this, we propose ForgeryTalker , a unified baseline designed to generate these reports end-to-end. At its core, ForgeryTalker utilizes a shared encoder to learn a common, forgery-aware representation, forcing a deep fusion of visual and semantic features. This representation is processed by specialized dual decoders to concurrently generate the localization mask and the textual report, ensuring the explanation is semantically grounded in the visual evidence. Notably, our framework operates under the premise that the input image has already been flagged as suspicious by an upstream binary detector ( Livernoche et al., 2025 ; Anan et al., 2025 ) . By focusing exclusively on post-hoc attribution (localization and explanation), we shift the forensic objective from merely delivering a binary verdict’ to providing the interpretable evidence’ behind it. This specialized scope ensures that computational resources are dedicated to generating meaningful semantic insights for manipulated images rather than redundant processing of pristine ones. Our primary contributions are summarized as follows: Dataset Tasks Modality Source Samples Unique Forgeries Manipulation Type GT Type FaceForensics++ ( Rossler et al., 2019 ) Class. / Seg. Video 1,000 4,000 Multi-Face Mods Label + Mask Celeb-DF ( Li et al., 2020 ) Classification Video 590 5,639 DeepFake Image Label DeeperForensics-1.0 ( Jiang et al., 2020 ) Classification Video 50,000 10,000 GAN Image Label DFDC ( Dolhansky et al., 2020 ) Classification Video 23,654 104,500 DeepFake Image Label FaceShifter ( Li et al., 2019 ) Classification Video N/A 5,000 GAN Image Label ForgeryNet ( He et al., 2021 ) Class. / Seg. Image, Video 116,321 221,247 DeepFake, GAN Label + Mask OpenForensics ( Le et al., 2021 ) Detection / Seg. Image, Video 45,473 70,325 GAN, Inpainting BBox, Mask DF40 ( Yan et al., 2024 ) Class. / Seg. Image, Video N/A >\text{1,000,000} Multi-Face Mods Label + Mask DiffusionFace ( Chen et al., 2024 ) Generation Image N/A 50,000 Diffusion Image Label GenFace ( Zhang et al., 2024a ) Generation Image 10,000 10,000 GAN, Inpainting Mask MMTT (Ours) Seg. / Caption Text, Image 100,000 152,217 Face Swap, Inpainting, Attribute Edit Text + Mask Table 1: Comparison of Face Manipulation Datasets. Our MMTT dataset is highlighted and provides rich text annotations for the “why” problem, a unique feature among existing resources. • A New Task and Dataset. We introduce Forgery Attribution Report Generation , a multimodal task combining forgery localization and natural language explanation. To support this, we present M ulti- M odal T amper T racing ( MMTT ), the first large-scale dataset with 152,217 samples, each annotated with a precise ground-truth mask from the editing process and a human-written attribution report. • A Unified and Effective Baseline. We propose ForgeryTalker, a unified framework that jointly performs forgery localization and report generation. It is designed to facilitate coherent cross-modal reasoning through a shared encoder (image encoder + Q-former) and dual decoders (mask decoder and a Large Language Model). • Comprehensive Benchmarking. We conduct extensive experiments on the proposed dataset for both report generation and forgery localization. The results validate that our baseline achieves competitive performance (59.3 CIDEr and 73.67 IoU) and demonstrates the complementary benefits of jointly addressing both tasks. 2 Multi-Modal Tamper Tracing Dataset As a comparison with other major forgery datasets in Table 1 highlights, the proposed MMTT is the first to provide detailed textual explanations alongside forgery masks. The dataset contains 152,217 samples distributed across four manipulation paradigms, with diffusion-based inpainting and face swapping being the most prevalent (Figure 3 a). Our statistical analysis reveals several key properties: (1) Eyebrows, eyes, and lips are the most common targets for manipulation across all localized editing methods (Figure 3 b). (2) A significant portion of images feature multiple alterations, with 2-5 concurrent modifications being common (Figure 3 c). (3) The textual annotations form a rich corpus of over 4 million words, with an average description length of 27.4 words . The content of these descriptions aligns closely with the visual forgeries, frequently referencing the manipulated facial parts