Turn That Frown Upside Down: FaceID Customization via Cross-Training Data
Shuhe Wang, Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, Eduard Hovy · 2025
Read Turn That Frown Upside Down: FaceID Customization via Cross-Training Data by Shuhe Wang, Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, Eduard Hovy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15407.txt.
Introduction
Face generators can clone a portrait faithfully but freeze the expression; ask for the same person laughing or in profile and they falter. Shuhe Wang and six co-authors built a training set in 2025, CrossFaceID, to teach identity models how to vary a face without losing the person.
The set holds forty thousand text-image pairs across about two thousand people, twenty shots each spanning poses, expressions, angles, and adornments. Training shows the model one face and demands another of the same person with altered features. Fine-tuned models kept their identity fidelity while gaining real editing range. The authors release the data as a public fix for a known blind spot.
Catalog record
| CREATOR | Shuhe Wang, Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, Eduard Hovy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15407
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Turn That Frown Upside Down: FaceID Customization via Cross-Training Data Shuhe Wang, Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, Eduard Hovy Abstract † † footnotetext: Email: [email protected] † † footnotetext: ♠ The University of Melbourne, ♣ Zhejiang University, ⧫ University of Washington, ▲ 01.AI, ▼ Nanyang Technological University Existing face identity (FaceID) customization methods perform well but are limited to generating identical faces as the input, while in real-world applications, users often desire images of the same person but with variations, such as different expressions (e.g., smiling, angry) or angles (e.g., side profile). This limitation arises from the lack of datasets with controlled input-output facial variations, restricting models’ ability to learn effective modifications. To address this issue, we propose CrossFaceID, the first large-scale, high-quality, and publicly available dataset specifically designed to improve the facial modification capabilities of FaceID customization models. Specifically, CrossFaceID consists of 40,000 text-image pairs from approximately 2,000 persons, with each person represented by around 20 images showcasing diverse facial attributes such as poses, expressions, angles, and adornments. During the training stage, a specific face of a person is used as input, and the FaceID customization model is forced to generate another image of the same person but with altered facial features. This allows the FaceID customization model to acquire the ability to personalize and modify known facial features during the inference stage. Experiments show that models fine-tuned on the CrossFaceID dataset retain its performance in preserving FaceID fidelity while significantly improving its face customization capabilities. To facilitate further advancements in the FaceID customization field, our code, constructed datasets, and trained models are fully available to the public. † † Our codes, models, and datasets are available at: https://github.com/ShuheSH/CrossFaceID . 1 Introduction Face identity (FaceID) customization is an important image generation task [ 18 , 23 , 27 , 25 , 8 , 16 , 26 , 36 ] , allowing users to achieve personalized facial customization using a pre-trained text-to-image diffusion model. Although existing methods demonstrate effectiveness, they exhibit a significant limitation: they can only generate images with the exactly same face as the input, while in real-world applications, users often desire images of the same individual but with variations, such as different facial expressions (e.g., smiling) or angles (e.g., side profile), shown in Figure 1 . Figure 1: An example illustrating the limitations of existing FaceID customization models in generating images of the same individual with variations. Here, we use the same image of a girl as the input, paired with two different text prompts. In Case 1 , the desired output is an image of the person with smiling, and in Case 2 , the goal is an image of the person facing right and wearing sunglasses. The results clearly show that the two leading FaceID customization models, IP-Adapter [ 37 ] and InstantID [ 34 ] , perform well in preserving the exact same face as the input image but fail to customize the input face as specified—such as adding a smile for Case 1 or generating a right-facing person with sunglasses for Case 2. In contrast, the model trained on our proposed CrossFaceID dataset effectively addresses these shortcomings, successfully generating a smiling face for Case 1 and a right-facing person wearing sunglasses for Case 2. This issue is primarily due to the lack of such a FaceID customization dataset where input and output faces exhibit controlled variations. In current datasets for training face customization models, the input and output faces in the dataset are often identical [ 15 , 3 , 40 , 19 ] . This setup restricts the model’s ability to learn effective facial feature modifications during training, instead reinforcing its focus on maintaining face consistency. Consequently, the model struggles during inference when it is required to modify facial features while preserving FaceID consistency due to the lack of relevant training experience. To address the above issue, in this paper, we propose CrossFaceID, the first large-scale, high-quality, and publicly available dataset specifically designed to improve the facial modification capabilities of FaceID customization models. Specifically, to obtain multiple public images of the same person, CrossFaceID gathered 40,000 images from approximately 2,000 celebrities, with each celebrity represented by around 20 images showcasing diverse facial attributes such as poses, expressions, angles, and adornments. To annotate these images, we use GPT-4 to generate detailed descriptions for these 40,000 images, particularly focusing on the facial features of the individuals, resulting in a set of 40,000 one-to-one text-image pairs. In this way, during the training stage, we start with a pre-trained FaceID customization model to ensure a solid foundation for preserving FaceID fidelity. Then, we employ a cross-training method, where a specific face of a person is used as input, and the pre-trained model is forced to generate another image of the same person but with altered facial features. This allows the pre-trained FaceID customization model to acquire the ability to personalize and modify known facial features during the fine-tuning process. Trained on our CrossFaceID dataset, we achieve comparable performance in preserving FaceID fidelity when compared to the widely used FaceID customization frameworks, InstantID [ 34 ] and IP-Adapter [ 37 ] , while also significantly enhancing their ability to customize FaceID. The contribution of this work can be summarized as follows: 1 We collect CrossFaceID, a high-quality, publicly available dataset designed to improve the facial modification capabilities of FaceID customization models. 2 The model fine-tuned on the CrossFaceID dataset retains its performance in preserving FaceID fidelity while significantly improving its face customization capabilities, showing the effectiveness of the dataset. 3 Our code, models, and datasets are fully available to the public, supporting further advancements in the FaceID customization field. The rest of this paper is organized as follows: • Section 2 reviews studies related to text-to-image models and FaceID customization. • Section 3 details on the construction of CrossFaceID. • Section 4 focuses on the training and inference processes utilizing our constructed CrossFaceID dataset. • Section 5 presents experimental results evaluating the effectiveness of CrossFaceID. • Section 6 concludes this work. 2 Related Work 2.1 Text-to-image Models Text-to-image refers to the process of generating images from textual descriptions using pre-trained image generation models [ 24 , 6 , 7 , 23 , 25 , 27 , 13 ] . These models are trained to understand the relationship between textual input and visual content, enabling them to create images that match the given description. Thanks to the success of the transformer model [ 33 ] , most early text-to-image approaches can be broken down into two stages: (1) using an image encoder, such as DARN [ 10 ] , PixelCNN [ 31 ] , PixelVAE [ 11 ] , or VQ-VAE [ 32 ] , to convert an image into several tokens; and (2) training the model to predict these image tokens based on the provided text input within the transformer framework [ 33 ] . Recently, diffusion models [ 28 , 29 , 18 , 5 , 23 , 27 , 25 , 2 , 13 ] have emerged as the new state-of-the-art approach for image generation, offering innovative solutions for the text-to-image task. In this approach, the text prompt is first encoded