DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder

Peipeng Yu, Hui Gao, Jianwei Fei, Zhitao Huang, Zhihua Xia, Chip-Hong Chang · 2024

Read DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder by Peipeng Yu, Hui Gao, Jianwei Fei, Zhitao Huang, Zhihua Xia, Chip-Hong Chang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07260.txt.

Introduction

Peipeng Yu and co-authors present DFREC, a method that recovers both faces hidden inside a deepfake image. Most forensic tools only say whether an image is fake, which gives investigators little to trace. Recovering the source face and the target face could help identify who was used and who was impersonated.

The system first splits the fake face into source and target information. It then rebuilds the source face, extracts hidden target traits, and fuses them with background context through a masked autoencoder to rebuild the target. Tested on six strong face-swap attacks across three data sets, it beat prior recovery methods. It is presented as the first scheme to recover both faces directly.

Catalog record

CREATORPeipeng Yu, Hui Gao, Jianwei Fei, Zhitao Huang, Zhihua Xia, Chip-Hong Chang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder Peipeng Yu, Hui Gao, Jianwei Fei, Zhitao Huang, Zhihua Xia, Chip-Hong Chang Abstract Recent advances in deepfake forensics have primarily focused on improving the classification accuracy and generalization performance. Despite enormous progress in detection accuracy across a wide variety of forgery algorithms, existing algorithms lack intuitive interpretability and identity traceability to help with forensic investigation. In this paper, we introduce a novel DeepFake Identity Recovery scheme (DFREC) to fill this gap. DFREC aims to recover the pair of source and target faces from a deepfake image to facilitate deepfake identity tracing and reduce the risk of deepfake attack. It comprises three key components: an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM). The ISM segments the input face into distinct source and target face information, and the SIRM reconstructs the source face and extracts latent target identity features with the segmented source information. The background context and latent target identity features are synergetically fused by a Masked Autoencoder in the TIRM to reconstruct the target face. We evaluate DFREC on six different high-fidelity face-swapping attacks on FaceForensics++, CelebaMegaFS and FFHQ-E4S datasets, which demonstrate its superior recovery performance over state-of-the-art deepfake recovery algorithms. In addition, DFREC is the only scheme that can recover both pristine source and target faces directly from the forgery image with high fadelity. I Introduction The emergence of deepfake technology, particularly its face-swapping capabilities, has ushered in an unprecedented era of digital media manipulation. Since its inception, deepfake has rapidly evolved into tools for disseminating misinformation, identity theft, and fraud. The growing ethical and legal concerns on the widespread dissemination of deepfake content have underscored the urgency of developing robust deepfake forensic tools. This is crucial not only for safeguarding individuals’ rights but also upholding the information integrity in the digital age. (a) Deepfake Face Swapping Model Training (b) Identity similarity between forgery and target/source faces Fig. 1: (a) The training process of deepfake face swapping models. The identity loss is used to increase the similarity between the forgery and source face identities, and the attribute loss is used to ensure the similarity between the forgery and target face attributes. (b) The probability density functions of the identity similarity (IDSim) between the forgery and source, forgery and target, and forgery and other unrelated faces. The results are calculated on the five types of forgeries on the FaceForensics++ and CelebaMegaFS datasets. Most of the existing deepfake forensic methods focus on improving the generalization detection performance across various kinds of forgery images [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . While these methods demonstrate impressive performance in classification and localization tasks, they are inherently limited in their ability to provide comprehensive forensic evidence. Specifically, they fail to establish a coherent chain of evidence linking the detection results to the identities involved in the manipulation. In other words, they cannot reliably trace the source identity (i.e., the person whose facial features are transferred) and the target identity (i.e., the individual whose face appears in the manipulated content). The inability to trace the origin of deepfake content can delay verifying the origin of deepfake content and mitigating potential damage. More importantly, as the onus of proof is borne by the victims with source identities, an accurate recovery of the source and target identities could help to collect and identify evidences for pursuing legal action against the defendant responsible for creating and disseminating deepfake content with malicious intent. Motivated by this forensic investigation gap, we propose to approach the deepfake forensic task from the perspective of pristine face recovery. While deepfake algorithms have evolved through various iterations, their core operation involves transferring the source identity onto the target face while retaining the attributes of the target face (such as expression, pose, lighting, etc.). This synthesis process blends the identities of both faces, resulting in a composite image that incorporates features from both identities [ 8 ] . As illustrated in Fig. 1 , the identity similarity between the forgery face and the target face is closely resembling the similarity between the forgery face and the source face. This observation suggests that the forgery face encompasses a portion of the target face information. Building on this insight, we propose DFREC, a DeepFake Identity Recovery scheme based on Identity-aware Masked autoencoder, which is designed to recover both the source and target faces. The main contributions of this paper are outlined as follows: • We propose a digital forensic aid DFREC to restore the identities of both target and source faces. DFREC accomplishes this task through ( i ) designing an Identity Segmentation Module (ISM) to segment information related to the source and target faces respectively, ( ii ) constructing a Source Identity Recovery Module (SIRM) to recover the source face using the segmented source information, and ( iii ) designing a Target Identity Recovery Module (TIRM) to recover the target face using the background information and latent target identity features extracted by the SIRM. Through identity segmentation and recovery, DFREC is able to restore the pristine source and target faces with high fidelity for intuitive interpretation. • We propose an Identity-aware Masked Autoencoder (MAE) to integrate latent target identity features into the face recovery process. Specifically, the MAE utilizes segmentation maps generated by the ISM to mask the input image and uses the Vision Transformer (ViT) for feature extraction. The target identity features are incorporated into the visible patch embedding to guide the MAE to restore pristine quality image with the desired target face identity. • We evaluate DFREC on the FaceForensics++, CelebaMegaFS, and FFHQ-E4S datasets, across six types of deepfake face swapping approaches. Compared to existing face inpainting methods and deepfake recovery methods, our approach achieves superior face recovery performance. By successfully recover both the pristine source and target faces with the correct identities, DEREC provides a non-repudiable intrinsic evidence of provenience that is gleaned directly from the forgery image. II Related Works In this section, we first review existing deepfake face-swapping algorithms, then discuss current detection methods, and finally introduce the development of deepfake recovery techniques. II-A Deepfake Face Swapping Face swapping is a representative algorithm in deepfake technology, which leverages generative models to encode target features and synthesize the swapped face. Convolutional neural networks (CNNs) and generative adversarial networks (GANs) are often employed to achieve identity-attribute encoding [ 9 , 10 , 11 , 12 ] . Gu et al. [ 9 ] proposed to leverage conditional GANs to generate specific swapping faces. This framework learns feature embeddings for each facial component independently, enabling precise and customizable local face editing. To further improve the visual quality of the swapped faces, Li et al. proposed FaceShifter [ 10 ] , which employs a U-Net architecture to capture the target face’s attributes. It incorporates an Adaptive Embedding Integration Network to blend source facial features with target attributes, producing photorealistic face-swapping results. Similarly, SimSwap [ 11 ] introduces a Weak Feature Matching Loss to optimize generated faces. It exploits both shallow and deep discriminator features for better consistency and fidelity. Based on these advancements, Zhu et al. [

Read the complete text