FaceMe: Robust Blind Face Restoration with Personal Identification

Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li · 2025

Read FaceMe: Robust Blind Face Restoration with Personal Identification by Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05177.txt.

Introduction

Siyu Liu and co-authors present a 2025 method, FaceMe, that restores damaged face photos while keeping the person's identity. Restoring a blurred or worn face is hard because key detail is missing. Past methods made clean faces that often stopped looking like the right person. FaceMe uses one or a few reference photos of the person. An encoder pulls identity traits from those photos to guide the repair model. Traits not tied to identity are kept out of the guidance.

The design accepts any number of reference photos at use time. Made-up photos can serve as references during training, thanks to the encoder's steadiness. The team built a training pool that mimics poses and expressions. Tests showed high-quality repairs that held identity across cases. The authors stress robustness to changes in pose and look. That makes the tool fit for old family or archive photos.

Catalog record

CREATORSiyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05177

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FaceMe: Robust Blind Face Restoration with Personal Identification Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li Abstract Blind face restoration is a highly ill-posed problem due to the lack of necessary context. Although existing methods produce high-quality outputs, they often fail to faithfully preserve the individual’s identity. In this paper, we propose a personalized face restoration method, FaceMe, based on a diffusion model. Given a single or a few reference images, we use an identity encoder to extract identity-related features, which serve as prompts to guide the diffusion model in restoring high-quality and identity-consistent facial images. By simply combining identity-related features, we effectively minimize the impact of identity-irrelevant features during training and support any number of reference image inputs during inference. Additionally, thanks to the robustness of the identity encoder, synthesized images can be used as reference images during training, and identity changing during inference does not require fine-tuning the model. We also propose a pipeline for constructing a reference image training pool that simulates the poses and expressions that may appear in real-world scenarios. Experimental results demonstrate that our FaceMe can restore high-quality facial images while maintaining identity consistency, achieving excellent performance and robustness. Figure 1: Make the people in the photo look like you and those you are familiar with. Using a single or a few reference images, we can restore realistic images without any fine-tuning for identity. Zoom in for best view. Introduction Face restoration focuses on improving the quality of facial images by removing complex degradation and enhancing details. In the real world, facial images frequently suffer from complex degradation, e.g. , blurring, noise, and compression artifacts, which can significantly impact downstream tasks like face recognition and detection. Face restoration is inherently a highly ill-posed task because a single low-quality input can correspond to many potential high-quality counterparts scattered throughout the high-quality image space. Most advanced face restoration methods cannot guarantee identity consistency, i.e. , the identity of the restored portrait may significantly deviate from the actual identity. Such deviations in identity are unacceptable in practical applications, e.g. , restoring facial images in personal photo albums. Recently, some studies have used high-quality reference images of the same identity to enhance identity consistency in restored facial images. These approaches improve results through feature alignment ( Li et al. 2018 ; Li et al. 2020 ; Li et al. 2022 ) or by fine-tuning the diffusion model to constrain the generation prior ( Varanka et al. 2024 ; Ding et al. 2024 ; Chari et al. 2023 ) . Although these methods have achieved significant results, they have obvious limitations. The feature alignment-based methods are influenced by the degradation of the input image and the pose of the reference image. If the features are not well aligned, the quality of the restoration can significantly deteriorate. The fine-tuning-based methods typically require 5 \sim 20 reference images and need to be fine-tuned again whenever the identity changes, making it a time-consuming process. Additionally, the quality of the reference images may significantly impact image restoration quality. We illustrate the differences between our proposed personalized method and the fine-tuning-based personalized method in Fig. 1 . In this paper, we proposed FaceMe, a fine-tuning-free personalized blind face restoration method based on the diffusion model. Given a low-quality input and either a single or a few high-quality reference images of the same identity, FaceMe restores high-quality facial images and maintains identity consistency within seconds. Remarkably, changing identities does not require fine-tuning, and the reference images can have any posture, expression, or illumination. Furthermore, the quality of the reference image does not significantly impact the quality of the restored image. To our knowledge, this is the first approach that leverages diffusion prior for personalized face restoration tasks, which does not require fine-tuning when changing identity. The fine-tuning-free personalized blind face restoration currently faces three significant challenges: 1) Influence of identity-irrelevant features from reference images; 2) Balancing the dependency between low-quality input and reference images; and 3) Insufficient datasets for personalized blind face restoration. To address the first challenge , we propose to use an identity encoder to extract identity-related features and a multi-reference image training approach to minimize the impact of identity-irrelevant features. In addition, we found that the second challenge of balancing dependencies between low-quality input and reference images arises from simultaneous personalization and restoration training. To address it, we propose a two-stage training approach that effectively balances these dependencies. In the training stage I, we focus on training the model’s personalization ability, i.e. , training the proposed identity encoder. In the training stage II, we fix the identity encoder and train a ControlNet, which corresponds to the image restoration capability of the model. Considering the third challenge of insufficient training data, we propose to use synthetic datasets as reference images. Specifically, to better align reference images with real-world application scenarios, we create a pose-reference pool categorized by poses and expressions. During the synthesis process, we sample several images from this pose-reference pool to serve as pose references and select one image from a publicly available facial image dataset to serve as identity control. Using a personalized generative model, we synthesize reference images that match the poses of the given pose references while maintaining the identity of the provided identity control image, thereby producing a set of images with the same identity but diverse poses. Related Work Blind Face Restoration Recently proposed blind face restoration methods rely on generating priors to achieve high-quality image restoration. Methods based on GAN priors, such as GPEN ( Yang et al. 2021 ) and GFP-GAN ( Wang et al. 2021 ) , enhance the quality of restored images by embedding rich generative priors from pre-trained face GANs into their network structures. Due to the remarkable success of VQGAN ( Esser, Rombach, and Ommer 2021 ) in the field of image generation, researchers have been inspired to propose restoration methods utilizing vector quantization. VQFR ( Gu et al. 2022 ) , CodeFormer ( Zhou et al. 2022 ) , and RestoreFormer ( Wang et al. 2022 ) all focus on learning high-quality codebooks and achieving superior image restoration by matching high-quality vectors. More recently, the powerful generation prior of diffusion models has been employed for face restoration. DR2 ( Wang et al. 2023 ) leverages the diffusion model as a robust degradation removal module and employs image restoration techniques to improve the restored image’s quality. DifFace ( Yue and Loy 2024 ) establishes a posterior distribution from observed low-quality images to high-quality images. By applying this distribution to each step of the denoising process of a pre-trained diffusion model, it gradually converts low-quality images into high-quality images. Although these blind face restoration methods achieve impressive results, restored identities often deviate from the real ones due to the lack of identity information. Reference-based Face Restoration To enhance the quality of restored images and ensure identity similarity, several methods leverage reference images of the same identity for face restoration. GFRNet ( Li et al. 2018 ) , ASFFNet ( Li et al. 2020 ) , and DMDNet ( Li et al. 2022 ) require alignment modules to align the features of low-quality images

Read the complete text