Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID
Koray Ulusan, Benjamin Kiefer · 2025
Read Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID by Koray Ulusan, Benjamin Kiefer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03557.txt.
Introduction
Koray Ulusan and Benjamin Kiefer study fake training photos for portrait AI in a 2025 paper. Tools that personalize image generators need a few real photos of a person. Keeping the face true to life is the hard part. The pair test whether extra synthetic images help or hurt.
Classic tweaks like flips and crops were compared with generated extras. Two personalization methods, DreamBooth and InstantID, were the test beds. A face distance score based on a known network graded likeness. Classic tweaks sometimes added artifacts that eroded identity. Generated images helped when balanced with real ones, but hurt alone. A user study of ninety seven people split on polish versus likeness. The results guide data recipes for portrait tools.
Catalog record
| CREATOR | Koray Ulusan, Benjamin Kiefer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID Koray Ulusan, Benjamin Kiefer Abstract Personalizing Stable Diffusion for professional portrait generation from amateur photos faces challenges in maintaining facial resemblance. This paper evaluates the impact of augmentation strategies on two personalization methods: DreamBooth and InstantID . We compare classical augmentations (flipping, cropping, color adjustments) with generative augmentation using InstantID’s synthetic images to enrich training data. Using SDXL and a new FaceDistance metric based on FaceNet, we quantitatively assess facial similarity. Results show classical augmentations can cause artifacts harming identity retention, while InstantID improves fidelity when balanced with real images to avoid overfitting. A user study with 97 participants confirms high photorealism and preferences for InstantID’s polished look versus DreamBooth’s identity accuracy. Our findings inform effective augmentation strategies for personalized text-to-image generation. 1 Introduction Personalized text-to-image generation has gained traction with the rise of models like Stable Diffusion (SD). However, training SD on small, user-specific datasets presents challenges, such as identity retention, overfitting, and artifact generation. Augmentation techniques are widely used in deep learning to improve generalization, but their role in personalized text-to-image generation is underexplored. In this work, we analyze how augmentations affect personalized SD models trained via few-shot and zero-shot methods, focusing on DreamBooth and InstantID. We examine their impact on model performance and whether they improve the realism and consistency of generated images. In particular, we analyze both classical and generative augmentation strategies to bridge the gap between limited real data and high-fidelity synthetic outputs. By refining facial features and preserving identity through targeted GenAI-based augmentations, such as InstantID, we aim to improve the applicability of personalized generation in scenarios where synthetic data must closely mirror real-world characteristics. We analyze under which conditions we can ensure that “GenAI outputs improve GenAI outputs”, avoiding a data quality collapse, providing best practices and heuristics. Our contributions include: • Analysis of classical augmentation techniques such as flipping, cropping, color enhancement, and background modifications. • Using InstantID as a fast way of enhancing the user-specific dataset using the diffusion model itself. • We conduct a survey to evaluate how white-collar workers perceive personalized generations from DreamBooth and InstantID under various augmentation strategies. 2 Background and Related Work In this section, we present the foundational concepts and prior research relevant to our work on augmentation techniques for few-shot personalization in diffusion models. 2.1 Text-to-Image Diffusion Models Text-to-image diffusion models generate high-quality images from natural language descriptions by gradually denoising random Gaussian noise guided by text embeddings [ 19 , 16 ] . Stable Diffusion [ 19 ] employs a latent diffusion approach that operates in a compressed latent space rather than pixel space, reducing computational requirements while maintaining generative quality. 2.2 Subject-Driven Image Generation Subject-driven image generation creates images featuring specific subjects with high fidelity while maintaining their identity across contexts [ 20 ] . Key approaches include: DreamBooth [ 20 ] fine-tunes the U-Net of Stable Diffusion using 3-5 images of a specific subject. It preserves the semantic prior through class-specific prior preservation loss and uses a rare token with weak prior to refer to the subject with the prompt format “a [ V ] [class noun]”. InstantID [ 24 ] is a zero-shot method combining facial feature extraction and text conditioning. It uses 5 facial landmarks to control the face’s position and orientation, providing greater control over the output. We standardize experiments by using the same SDXL model for both methods to ensure a fair comparison. 2.3 Image Augmentation Techniques 2.3.1 Classical Image Augmentations Classical image augmentations include geometric transformations (flipping, rotation, scaling, cropping), photometric adjustments (brightness, contrast, saturation, hue), and noise injections (Gaussian, salt-and-pepper). These predefined techniques preserve semantic integrity while adding controlled diversity to expand limited training data. 2.3.2 Augmentations in Diffusion Models Data augmentation enhances diffusion model performance while reducing computational demands [ 23 ] . Key approaches include mixing-based augmentations that interpolate between existing samples [ 10 ] and consistency regularization techniques that enforce invariance to specific transformations [ 9 , 15 ] . Our work investigates these techniques specifically for few-shot personalization applications. 2.4 Face Processing Approaches FaceNet [ 21 ] maps facial images to a 128-dimensional embedding space, where similar faces are positioned closely together. The standard pipeline uses MTCNN [ 27 ] for face detection before embedding generation, with cosine distance metrics for similarity assessment [ 22 ] . Alternative approaches include faceswapping methods [ 14 , 6 ] and augmented reality techniques for virtual try-on applications [ 8 ] . While these provide real-time capabilities, they often lack the flexibility and integration capabilities of diffusion-based approaches. Our research builds on these foundations to investigate how strategic data augmentation can improve few-shot personalization in diffusion models, focusing on identity preservation and recontextualization. (a) Real Images . (b) DreamBooth results with classical augmentations (crop, resize, and color) (c) DreamBooth results with GenAI augmentations & without classical augmentations Figure 2 : Example Improvement: DreamBooth Results Comparing Classical vs. Generative Augmentations for Subject Dataset Vacation-Anna . The model is prompted with default prompt with batchsize 4. The results are not cherry-picked to resemble the downstream application use. Although (b) is visually more interesting, the method in (c) is more consistent across many subject datasets. 3 Methodology We use augmentations across various Subject Datasets to see if there is an overall improvement in generated pictures. 3.1 Subject Datasets Our internal dataset includes 3 to 15 images per participant, with 10 participants. To maintain naturalistic data collection, we instructed them: “Can you send me portrait/selfie-style photos of your face in different places? The more different places, the better.” By avoiding strict guidelines, we ensured the images reflect realistic user behavior. Consequently, our findings align well with real-world data distributions, improving transferability and applicability. The names used to refer to subjects in this dataset are pseudonyms and do not correspond to real individuals. Our internal dataset exhibits diverse environmental conditions, facial orientations, and image qualities, ensuring variability that reflects real-world scenarios. The images include varied backgrounds, lighting, and subject behaviors, making the dataset both representative and robust. For example, some images feature cluttered or irregular backgrounds (e.g., Baker-Zoe , Bottle-Hugo ), while others are from controlled settings ( Biometric-Kora ). Gaze variation is present: Doctor-Nina looks away from the camera, and 3D-Gary shows dynamic head movements from video. Differences in appearance and accessories also appear, such as Farmer-Lisa wearing a helmet and Staircase-Judy wearing makeup. Lighting varies from well-lit ( Vacation-Anna ) to suboptimal ( 2024-Kora ), enhancing realism. These factors make our dataset valuable for evaluating model performance in unconstrained, real-world conditions. 3.2 Dataset Augmentations We apply augmentations individually to evaluate each technique’s performance improvement independently. Classical Augmentations Standard techniques include: (i) Random Horizontal Flip with p\in\{0,\frac{1}{2},1\} , and (ii) Color Jitter varying brightness, saturation (\pm 5,\pm 15) and hue (\pm 5\degree,\pm 15\degree) . Background Augmentation We use U 2 -Net [ 18 ] for subject isolation, testing both base and human segmentation models. Backgrounds include flat colors, Wikimedia Patterns [ 5 ] , and Studio Backdrops (handpicked simple pictures from BG-20k [ 13 ] , combined with our internal dataset). Blending Techniques We separately evaluate Alpha Blending and Poison Blending through both automated and