Styleclone: Face Stylization with Diffusion Based Data Augmentation

Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma · 2025

Read Styleclone: Face Stylization with Diffusion Based Data Augmentation by Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17045.txt.

Introduction

Neeraj Matiyali and co-authors teach fast networks to paint faces in a chosen style in this 2025 paper. Style datasets are often tiny, which starves training. Their system, StyleClone, grows the dataset first, then trains the painter.

They use a diffusion model, guided by text tokens tuned to the style and by real face photos, to make many varied style samples. A fast translation network then learns from that richer set. In tests across styles, the networks beat diffusion tools on speed and quality. They also kept the source face's content better. The work suits apps that restyle portraits in real time.

Catalog record

CREATORNeeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17045

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Styleclone: Face Stylization with Diffusion Based Data Augmentation Neeraj Matiyali, Siddharth Shrivastava, Gaurav Sharma Abstract We present StyleClone , a method for training image-to-image translation networks to stylize faces in a specific style, even with limited style images. Our approach leverages textual inversion and diffusion-based guided image generation to augment small style datasets. By systematically generating diverse style samples guided by both the original style images and real face images, we significantly enhance the diversity of the style dataset. Using this augmented dataset, we train fast image-to-image translation networks that outperform diffusion-based methods in speed and quality. Experiments on multiple styles demonstrate that our method improves stylization quality, better preserves source image content, and significantly accelerates inference. Additionally, we provide a systematic evaluation of the augmentation techniques and their impact on stylization performance. 1 Introduction Image stylization has become a very popular area of research and application. Almost all image editing applications have some form of stylization built in. Stylizing faces is specially interesting for consumption in social media, as well as for use in creative applications, media and marketing. The task has seen rapid progress with two popular approaches, (i) that take a style image as reference ( Gatys, Ecker, and Bethge 2016 ; Huang and Belongie 2017 ) , and (ii) that are conditioned by text, e.g . “style this image as anime”. These approaches are primarily based on conditional diffusion models ( Ho, Jain, and Abbeel 2020 ; Rombach et al. 2022 ) and their derivatives. Such stylization methods are generic, i.e . they are made to do potentially any kind of stylization, based on the reference image or text prompt. They have also been adapted to learn new styles as concepts by giving a set of images of a style of interest that the user would like to learn to use techniques such as textual inversion ( Gal et al. 2022 ) , LoRA ( Hu et al. 2021 ) and DreamBooth ( Ruiz et al. 2023 ) . In such a style cloning scenario, the aim is to make a stylization method adapted to the particular style. Image synthesis with diffusion models is relatively slow due to multiple evaluations of a denoising neural network. While this is suitable for general-purpose tasks, it is inefficient for style-specific face stylization. We propose prioritizing faster inference by leveraging diffusion models for extensive training dataset augmentation and training a lightweight, style-specific image translation model. Figure 1: StyleClone takes a small set of reference style images and augments them by generating similar styled images using textual inversion and diffusion models. It then trains a lightweight network with the augmented images which can mimic the style in the input reference images. We propose a method to train lightweight image translation network which is able to convert an input image into a particular style defined by a small set of images in that style. We propose to train such a network with adversarial training such that it learns to match the style of the output images to that of the style images. However, training it directly with the small set of reference style image does not give satisfactory results. Further, images generated by such a method carry strong distortions and deviations from the input image, and thus cannot be used directly as reasonable outputs. Moreover, diffusion based methods are heavy and slow, and thus not suitable for many applications that prioritize fast inference. Therefore, we propose an augmentation method, which takes the set of style images and learns to generate similar styled images using textual inversion ( Gal et al. 2022 ) and image-guided diffusion ( Meng et al. 2021 ) . We show that when such a set is curated with a carefully designed augmentation strategy, it can be used to train lightweight image translation network with adversarial loss. We curate such a set by first using textual inversion to learn the concept represented by the style images. We then use it with diffusion model for guided image synthesis with the original input style images (self augmentation), as well as real face images (cross augmentation). While the augmentations generated with self augmentation closely match the input style images, and are more stable and less distorted, cross augmentation generates more varied and diverse images, albeit with some distortions. Specifically, cross augmentation helps increase the coverage of style images in the augmented training data in terms of facial identity, head pose, colors, and background appearance. It makes the trained translation network more robust to variations in the input face images and helps preserve the identity and structure of input faces better during stylization. On the other hand, self augmentation, complements it by enforcing the constraint that the style of the final augmented training set is close to that of the input style images. Finally, we train the image translation network on the source domain consisting of real faces and the target domain consisting of augmented style images. The network is trained with an adversarial loss, without assuming any pairing between real and styled images. To summarize, our contributions are as follows. (i) We propose to use lightweight models for face stylization based on a small set of reference images in the style of interest. (ii) We use textual inversion and guided diffusion based data augmentation to generate examples that enable the training of the lightweight models. (iii) We propose two modes of augmentation (self and cross) based on the choice of guiding images and study their effect on the stylization results. (iii) We compare our method with diffusion based image-to-image approaches, and perform various ablation experiments studying different aspects of the proposed method. 2 Related Work Image-to-Image Translation Image-to-image translation maps images from one domain to another, typically using adversarial training ( Goodfellow et al. 2014 ) . Paired methods rely on aligned training data and combine adversarial and reconstruction losses ( Isola et al. 2017 ; Wang et al. 2017 ) , while unpaired methods like CycleGAN ( Zhu et al. 2017 ) introduce cycle-consistency. CUT ( Park et al. 2020 ) improves efficiency with contrastive loss. While fast at inference, these models require large training datasets. We mitigate this by first augmenting with diffusion models and then training a CUT-based stylization network. Portrait Stylization using StyleGAN StyleGAN ( Karras, Laine, and Aila 2019 ; Karras et al. 2020 ) is widely used for portrait generation and stylization, often pretrained on FFHQ ( Karras, Laine, and Aila 2019 ) . Toonify ( Pinkney and Adler 2020 ) adapts it for cartoon-like styles. StyleAlign ( Wu et al. 2021 ) exploits latent space alignment, while Deceive D ( Jiang et al. 2021 ) and ( Ojha et al. 2021 ) propose data-efficient fine-tuning strategies. Image inversion is key for generalization ( Richardson et al. 2021 ; Song et al. 2021 ) , enabling tasks like sketch-to-face or exemplar-based stylization ( Yang et al. 2022 ) . However, inversion can limit robustness to out-of-distribution inputs. DCT-Net ( Men et al. 2022 ) addresses this by avoiding inversion during inference. Diffusion-Based Stylization Diffusion models ( Ho, Jain, and Abbeel 2020 ; Rombach et al. 2022 ) have shown exceptional generative capabilities due to large-scale image-text

Read the complete text