Face Inpainting with Identity Preserving Latent Diffusion Models
João Santos, Carlos Santiago, Manuel Marques · 2026
Read Face Inpainting with Identity Preserving Latent Diffusion Models by João Santos, Carlos Santiago, Manuel Marques (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16696.txt.
Introduction
Santos, Santiago and Marques propose ID-ControlNet, a face-inpainting method that fills missing or hidden facial regions while keeping the person's identity intact. It builds on latent diffusion models and conditions generation on identity embeddings from a face-recognition network. A training scheme with identity-consistency and triplet losses aligns output faces with the target identity. Tests on CelebA-HQ, FFHQ, and a new E-Mask set show clear gains over standard diffusion inpainting.
Catalog record
| CREATOR | João Santos, Carlos Santiago, Manuel Marques |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30126 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Face Inpainting with Identity Preserving Latent Diffusion Models João Santos, Carlos Santiago, Manuel Marques Abstract Face inpainting techniques recover missing or occluded facial regions in a visually realistic manner, but preserving the identity in the final output remains a fundamental challenge. Identity consistency is crucial for downstream applications such as face recognition, digital forensics, and human–computer interaction, where even subtle identity distortions can significantly degrade performance or trust. Although diffusion-based generative models have recently achieved remarkable progress in image inpainting, they often struggle to faithfully retain individual-specific facial characteristics. On the other hand, existing identity-aware methods typically rely on costly fine-tuning, auxiliary supervision, or exhibit limited robustness to diverse occlusions, poses, and facial variations. To address these limitations, we propose ID-ControlNet, an identity-preserving face inpainting framework built upon latent diffusion models. Based on ControlNet architecture, our approach conditions the diffusion process on facial identity embeddings extracted from a pretrained face recognition network. This design enables reconstruction of occluded facial regions while maintaining global facial coherence and identity fidelity. Furthermore, we introduce an identity consistency and triplet loss training strategy that explicitly enforces alignment between the generated face and the target identity representation. Extensive experiments on CelebA-HQ, FFHQ, and on a new E-Mask dataset demonstrate that ID-ControlNet significantly improves identity preservation over standard diffusion-based inpainting methods, achieving performance comparable to state-of-the-art identity-aware approaches. keywords Facial Occlusion ,Inpainting ,Diffusion , † † corresponding: Corresponding author 1 Introduction Face inpainting aims to reconstruct missing or occluded facial regions while producing visually realistic and semantically coherent results. This task plays a critical role in numerous real-world applications, including photo restoration, digital content editing, and virtual or augmented reality. In high-stakes domains such as law enforcement and surveillance analysis, identity-consistent inpainting is particularly important, as reconstructed faces must preserve recognizable features that are essential for accurate identification. Consequently, an effective face inpainting system must go beyond visual plausibility and ensure faithful preservation of an individual’s identity. Diffusion Models (DM) and their variants ( 10 ) have demonstrated strong capability in the generation of real images. Despite these advances, keeping the identity in face inpainting remains a persistent challenge. Standard diffusion models lack explicit mechanisms to encode subject-specific identity information, often causing reconstructed regions to drift toward an average facial appearance. As a result, visually convincing outputs may still fail to maintain the unique identity of the person depicted. To address this issue, recent identity-preserving inpainting methods have explored both GAN-based and diffusion-based solutions. GAN-based approaches ( 7 ) , such as MyStyle ( 21 ) , achieve strong identity reconstruction by leveraging subject-specific fine-tuning. More recently, diffusion-based personalization frameworks, including Custom Diffusion ( 15 ) and PVA ( 27 ) , have demonstrated that high-quality and identity-consistent face inpainting can be achieved using only a small number of reference images. These works establish diffusion models as the dominant paradigm for identity-aware face generation. However, they still depend on identity-specific adaptation or fine-tuning during inference, which limits scalability and hinders generalization to unseen identities. Moreover, identity consistency is typically enforced only at the output level, without explicitly guiding the diffusion process to maintain alignment with identity representations throughout denoising. In contrast, we propose an identity-conditioned inpainting framework that integrates facial identity cues directly into the diffusion process. Specifically, we introduce ID-ControlNet, a novel approach that extends a pretrained Latent Diffusion Model ( 23 ) by injecting identity embeddings, extracted from a face recognition approach ( 4 ) via a ControlNet-inspired conditioning architecture ( 30 ) . This design enables subject-specific identity information to condition the denoising trajectory, allowing faithful reconstruction of occluded facial regions while preserving global identity coherence. Furthermore, we introduce an identity consistency and triplet loss training strategy that explicitly reinforces alignment between generated faces and their corresponding identity embeddings in a self-supervised manner, eliminating the need for additional identity-specific fine-tuning. In addition to methodological challenges, evaluating identity preservation in face inpainting remains nontrivial. Existing datasets such as CelebA-HQ ( 11 ) and FFHQ ( 12 ) provide high-quality facial images but typically employ random or large-area occlusions, which emphasize overall realism rather than precise recovery of identity-defining features. Such settings make it difficult to assess whether a model truly reconstructs missing identity cues or simply generates plausible facial content. To address this limitation, we introduce the E-Mask dataset, a controlled benchmark where occlusions are restricted to the eye region, one of the most identity-informative facial areas. Each image is paired with a predefined eye-region mask, enabling consistent and focused evaluation of identity restoration under controlled occlusions. We evaluate our approach on CelebA-HQ, FFHQ, and the proposed E-Mask dataset, demonstrating that ID-ControlNet achieves superior identity preservation compared to standard diffusion-based inpainting methods, while maintaining competitive visual quality and generalization across identities and occlusion patterns. 2 Related Work Face Inpainting Early approaches based on encoder and decoder or Generative Adversarial Networks (GANs) frameworks, such as Context Encoder ( 22 ) and Partial Convolutions ( 17 ) , introduced contextual reasoning to fill missing regions. Later improvements like EdgeConnect ( 20 ) and Gated Convolution ( 29 ) enhanced structural consistency and mask adaptability. While these models achieve visually convincing completions, they often disregard subject-specific features, resulting in reconstructions that deviate from the individual’s identity when identity-critical regions (e.g., the eyes or nose) are occluded. Identity and Reference-Guided Inpainting To improve identity consistency, several subsequent works integrated explicit identity priors or reference images. ID-Inpainter ( 32 ) incorporated embeddings from pretrained face recognition networks to guide identity recovery. MyStyle ( 21 ) fine-tuned a StyleGAN ( 12 ) model per individual, learning a personalized latent prior for identity-accurate synthesis, but at high computational cost. Reference-Guided Inpainting ( 25 ) introduced modular control for texture and structure using adaptive normalization, while PATMAT ( 19 ) employed person-specific transformer anchors for large-occlusion restoration. These approaches improved fine-grained fidelity yet rely heavily on per-identity adaptation or multiple references. E2F-Net ( 8 ) specifically leveraged the eye region to reconstruct full faces, revealing the discriminative power of periocular cues for identity recovery. However, most of these GAN-based methods struggle to scale to unseen identities without retraining or fine-tuning. Diffusion-Based Personalization and Inpainting Diffusion models have recently surpassed GANs in generative fidelity and stability. Latent Diffusion Inpainting ( 23 ) and RePaint ( 18 ) established strong baselines for structural and texture restoration through iterative denoising and personalized diffusion models then extended this paradigm by conditioning generation on specific identity features. DreamBooth ( 24 ) and Custom Diffusion ( 15 ) enabled few-shot personalization but still required fine-tuning per subject. Parallel Visual Attention (PVA) ( 27 ) introduced reference-conditioned attention layers to encode identity directly during the diffusion process, achieving identity-consistent inpainting with minimal adaptation. More recently, IDiff-Face ( 3 ) adapted diffusion modeling for synthetic face generation and recognition, employing identity-conditioned latent diffusion with contextual dropout to balance identity discrimination and intra-class variation. These advances position diffusion-based frameworks as the new state of the art for personalized face generation, yet challenges remain in integrating identity conditioning efficiently without subject-specific optimization. 3 Background This section provides a brief overview of Diffusion Models and ControlNet ( 30 ) , which form the foundational components of our approach. Diffusion Probabilistic Models (DPMs) (