Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction

Xiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu · 2024

Read Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction by Xiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18149.txt.

Introduction

Xiao Guo, Manh Tran, Jiaxin Cheng, and Xiaoming Liu wrote this computer vision paper, published in two thousand twenty-four, on making face images of a specific person from text. A user gives one reference photo and a written prompt, and the model should draw that same face doing or wearing what the text says.

Older methods either need fresh tuning for each person or drift away from the prompt. The team's model, Dense-Face, adds a pose-controlling adapter to a standard image generator, so the face can turn as asked while text editing still works. It also learns to predict dense face annotations from the generator's own inner features, which teaches it face structure. Tests show top or near-top scores on text match, identity keeping, and pose control. The authors aim at custom portraits that stay true to both the person and the prompt.

Catalog record

CREATORXiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction Xiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu Abstract The text-to-image (T2I) personalization diffusion model can generate images of the novel concept based on the user input text caption. However, existing T2I personalized methods either require test-time fine-tuning or fail to generate images that align well with the given text caption. In this work, we propose a new T2I personalization diffusion model, Dense-Face, which can generate face images with a consistent identity as the given reference subject and align well with the text caption. Specifically, we introduce a pose-controllable adapter for the high-fidelity image generation while maintaining the text-based editing ability of the pre-trained stable diffusion (SD). Additionally, we use internal features of the SD UNet to predict dense face annotations, enabling the proposed method to gain domain knowledge in face generation. Empirically, our method achieves state-of-the-art or competitive generation performance in image-text alignment, identity preservation, and pose control. [ ] [ ] [ ] [ ] Figure 1 : Performance comparison of different personalized methods. 1 Introduction The Text-to-image (T2I) personalized generation, or customized T2I diffusion models, has obtained significant attention for its ability to generate visually compelling images for novel concepts unseen in training [ 16 , 50 , 33 , 37 , 67 ] . One of the essential personalized T2I generation applications is face generation [ 30 , 29 , 7 , 12 , 36 ] , a classic and challenging computer vision task, requiring the generative method to both model complicated non-rigid face characteristics ( e.g. , subject identity) and produce high-quality images. Prior research works have successfully adapted the Stable Diffusion (SD) model for face generation, producing high-fidelity and identity-preserved images with high text-controllability. First, test-time fine-tuning (TTFT) is common to model the subject appearance [ 33 , 16 , 33 , 17 ] , generating realistic customized images in various contexts. Based upon that, recent works [ 16 , 50 , 33 , 67 , 55 ] introduce a new identity preservation face generation scheme without TTFT. Specifically, they first integrate the deterministic face identity embedding with the text-embedding input to the pre-trained SD. Then, they fine-tune the pre-trained SD on face samples from LAION 5 B [ 51 ] or other complicated collection procedures, for learning the face generation domain knowledge. This work focuses on the second generation paradigm, since multiple face images of the test subject can often be unavailable for TTFT. Specifically, we empirically notice the previous work is limited in achieving optimal performance on both face domain generation and text-based editing. This is because they uniformly fine-tune the pre-trained SD on the face-specific dataset. This fine-tuning step may decrease the text-editing ability of the pre-trained SD, and the resultant model’s text controllability depends on dataset qualities that vary from one to another. To address this limitation, we propose Dense-Face, a new personalization model, leveraging a novel pose-controllable adapter (PC-adapter) and the additional pose branch to help the proposed method obtain two generation modes: text-editing mode and face-generation mode (as depicted in Fig. 2(a) ). Then, these two generation modes are jointly used via a latent space blending technique for the personalized image generation, which achieves the domain-specific generation while maintaining the premium pre-trained SD’s text controllability. Moreover, we utilize the internal SD-UNet feature for predicting dense face annotations, enhancing the face generation domain knowledge further. Specifically, our proposed Dense-Face, as depicted in Fig. 3 , introduces a pose branch, the PC-adapter, and the annotation prediction module on the top of the pre-trained SD. The PC-adapter and pose branch alter the forward propagation of cross-attention modules, adapting the pre-trained SD for the face generation domain. We keep the pre-trained SD frozen and only update additional components, which are plug-in modules that enable Dense-Face to have two generation modes: text-editing mode and face-generation mode . The former mode generates the base image with diverse contexts based on the text caption; then the latter generation mode updates the face region of the base image to preserve the subject identity. Such a two-stage design relies on the generation ability of face-generation mode in generating faces at different poses given the head pose ( i.e. , Euler angles consisting of yaw, pitch and roll) as the condition (Fig. 2(b) ). Also, this procedure involves a commonly used latent space blending technique [ 3 , 2 , 40 , 69 ] that effectively preserves the background contexts (Sec. 3.6 ). In this way, our Dense-Face improves the specific face domain generation while maintaining the original T2I-SD text-based editing ability. (a) (b) Figure 2 : (a) Our proposed Dense-Face introduces additional components, including a pose branch and PC-adapter, on the top of the pre-trained SD. These two components enable Dense-Face to have two generation modes: text-editing mode and face-generation mode . These two modes are jointly used via the latent space blending (Sec. 3.6 ) for the personalized generation. For example, given one of reference subject images, the text-editing mode generates a base image, and face-generation mode updates the face region for the identity-preservation in the final result. (b) Dense-Face in face-generation mode generates realistic face images at different pose views. Secondly, given the sparse head pose condition ( i.e. , Euler angles), Dense-Face employs the annotation prediction module that leverages the internal SD feature for predicting different dense face annotations to improve further the face generation domain knowledge (Sec. 3.3 ). This is inspired by the fact that regularizing such internal SD features assists the domain-specific generation in prior works [ 19 , 44 , 32 ] . These features contain rich and detailed object semantics (including human face) and already benefit various applications [ 62 , 4 , 39 , 54 , 66 ] . Such dense annotation prediction also aligns with observations of predicting dense landmarks, helping construct complex non-rigid face patterns [ 61 , 15 ] . Also, we adopt the coarse-to-fine granularity learning scheme, predicting a series of dense annotations given the sparse head pose coordinates, to better learn the structural geometry of the human face, similar to full body T2I works [ 41 , 28 ] . To support this dense annotation prediction, we collect a large-scale T2I face generation dataset with high-quality images and rich, dense annotations, named T2I-Dense-Face dataset, which also contributes to future face generation research. In summary, our contributions are: \diamond We propose a text-to-image diffusion model, named Dense-Face, for personalized image generation. We use the pose-controllable adapter with the pose branch to help the model generate high-fidelity images while maintaining high text controllability. \diamond To facilitate Dense-Face to learn domain-specific knowledge for face generation, we devise an auxiliary training objective that predicts a series of dense face annotations. This is supported by our proposed identity-centric text-to-image face dataset, termed T2I-Dense-Face. \diamond Our proposed method is evaluated on various generation tasks and achieves promising performance in personalization generation and face swapping. 2 Related Works Personalization Diffusion Model The personalization T2I diffusion models [ 16 , 50 , 33 , 37 , 67 , 59 , 19 ] attract increasing attentions in these two years. FaceX [ 25 ] achieves head pose control and identity preservation generation, whereas

Read the complete text