StyleYourSmile : Cross-Domain Face Retargeting Without Paired Multi-Style Data

Avirup Dey, Vinay Namboodiri · 2025

Read StyleYourSmile : Cross-Domain Face Retargeting Without Paired Multi-Style Data by Avirup Dey, Vinay Namboodiri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01895.txt.

Introduction

Moving a facial expression from one face to another is easy within photos of real people and much harder across styles, say from a photograph to a cartoon. Avirup Dey and Vinay Namboodiri present StyleYourSmile in 2025, a method for retargeting faces across such domains without paired training examples in each style.

The hard part is keeping identity, expression, and style separate, so one can change without dragging the others along. Earlier methods often failed to generalize across domains, or needed slow test-time tuning and carefully built multi-style datasets. A method that skips those demands would make stylized animation and avatar tools far easier to build.

Catalog record

CREATORAvirup Dey, Vinay Namboodiri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01895

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StyleYourSmile : Cross-Domain Face Retargeting Without Paired Multi-Style Data Avirup Dey, Vinay Namboodiri Abstract Cross-domain face retargeting requires disentangled control over identity, expressions, and domain-specific stylistic attributes. Existing methods, typically trained on real-world faces, either fail to generalize across domains, need test-time optimizations, or require fine-tuning with carefully curated multi-style datasets to achieve domain-invariant identity representations. In this work, we introduce StyleYourSmile , a novel one-shot cross-domain face retargeting method that eliminates the need for curated multi-style paired data. We propose an efficient data augmentation strategy alongside a dual-encoder framework, for extracting domain-invariant identity cues and capturing domain-specific stylistic variations. Leveraging these disentangled control signals, we condition a diffusion model to retarget facial expressions across domains. Extensive experiments demonstrate that StyleYourSmile achieves superior identity preservation and retargeting fidelity across a wide range of visual domains. Figure 1 : StyleYourSmile can preserve fine-grained identity features as well as domain-specific attributes while retargeting facial expressions. Our model achieves disentanglement between identity and domain style without using any curated multi-style pairs. 1 Introduction Imagine taking an old photograph of your loved one and transforming it to match your artistic vision—making the portrait smile exactly as you once sketched in your notebook! As trivial as it seems, this is a particularly challenging problem, even for state-of-the-art generative models. Entanglement between identity and transient attributes like expression, pose, lighting, is the core challenge in facial image synthesis. Previous works [ 26 , 1 , 14 ] leverage parametric face models [ 28 , 15 ] to decouple identity from transient attributes. However, there is little research on preserving this disentanglement across domains (like photo \rightarrow painting). Adding style as another orthogonal axis of variation poses an unconventional difficulty for existing methods, because (i) such domain shifts introduce structural inconsistencies— a dimple in a photograph may manifest as an intensity gradient, while in a painting, it could be a few abstract strokes and (ii) lack of multi-style datasets makes training nearly intractable. In this paper, we propose an image-based cross-domain face retargeting approach that can provide one-shot inference abilities and can be trained without curated multi-style paired data. Cross-Domain Retargeting One-shot HyperReenact [ 1 ] ✗ ✓ ✓ DiffusionRig [ 10 ] ✗ ✓ ✗ Arc2Face [ 35 ] ✗ ✗ ✓ Ours ✓ ✓ ✓ Table 1 : Scope of various face synthesis models: Unlike previous models, StyleYourSmile is trained for one-shot high fidelity cross-domain face retargeting. Recent works based on video diffusion [ 11 , 48 ] demonstrate strong identity retention across disparate visual domains however they rely on a single ‘ReferenceNet’ [ 19 ] to encode identity and appearance directly from the source image. Despite the simplicty of this approach, these models are often trained at scale with very large datasets spanning multiple domains, including various animated, CGI clips [ 31 ] . Methods that use compressed representations like facial recognition embeddings [ 35 , 47 ] achieve strong identity retention with smalled compute budget but discard non-identity details (e.g., style, lighting, accessories) due to the highly discriminative nature of these embeddings, as illustrated in Fig 2 . Figure 2 : Arc2Face has strong identity retention but it cannot preserve the source style as the underlying face recognition encoder discards all information that is not relevant to a person’s identity. While such stripped-down representations is useful for identity retention alone, it becomes a bottleneck for face retargeting, which demands preserving more nuanced attributes. In cross-domain scenarios preservation of domain-specific cues is critical. Hence, we propose two complementary encoders - a face recognition-based encoder that learns identity features agnostic to domain style, and a style encoder that complements it by learning representations for domain-specific cues. The spatial conditioning needed for accurate retargeting is generated by a 3DMM and routed to the diffusion model via ControlNet. Previous methods like HyperReenact [ 1 ] and GIF [ 14 ] trains StyleGAN with outputs of parametric face models [ 28 ] such that the latent is ‘pulled’ towards the desirable pose without identity drift. However, being trained on real-world images only these cannot preserve fine-grained details in other domains and often produces over-smoothed images as shown in Fig. 6 . DiffusionRig [ 10 ] too is limited by its single-frame real-world dataset [ 23 ] Hence, we also propose a light-weight style augmentation module that multiplexes standard video datasets like VoxCeleb [ 33 ] with visual styles of different domains, as shown in Fig. 4 . We jointly fine-tune the image encoders and the diffusion model on the augmented dataset. Since our primary goal is to adapt the diffusion model to abstract styles, low rank fine-tuning [ 18 ] of the UNet suffices. The architecture choice and training formulation helps us achieve one-shot cross-domain face retargeting . In summary, our key contributions are: • We propose StyleYourSmile , a unified framework for one-shot cross-domain face retargeting with image-based diffusion models. • We propose a novel training pipeline that (a) sidesteps curated multi-style data by augmenting real-world images using a fast, training-free technique and (b) disentangles identity and domain-specific style with two encoders that capture complementary representations. Our model can be trained using significantly lower compute budget compared to video-diffusion models (4x NVIDIA A5000 GPUs). • Through extensive comparisons on VoxCeleb1 [ 33 ] , a large scale face dataset, we demonstrate our model’s superior performance against existing baselines for cross-domain retargeting. 2 Related Works 2.1 Identity-Aware Image Generation Identity-aware generation benefits significantly from face recognition models, which extract detailed identity features from facial images. These models are designed to capture comprehensive facial embeddings for identity similarity assessment [ 45 , 8 , 2 ] . Consequently, inverting such models in a black-box manner has demonstrated the ability to generate facial images directly from identity embeddings [ 32 , 37 , 43 ] , including applications in zero-shot generation using both GAN and diffusion architectures [ 12 , 22 ] . However, existing inversion methods often rely on low-resolution datasets [ 12 , 41 ] tailored for face recognition or employ high-quality but limited datasets, restricting their overall generalization capability. Recent advancements have showcased impressive results through extensions of Stable Diffusion [ 38 ] . These include works based on Textual Inversion [ 13 ] and DreamBooth [ 39 ] , where diffusion models are fine-tuned on a few subject-specific images to learn a subject identifier, enabling accurate reproduction of that individual. Subsequent works have focused on reducing optimization time, such as HyperDreamBooth [ 40 ] , which integrates LoRA [ 18 , 53 ] and a hypernetwork for fine-tuning from a single image. Other methods, like Celeb-Basis [ 57 ] and StableIdentity [ 46 ] , learn an embedding basis from a celebrity dataset to condition text-based models. More broadly, Kosmos-G [ 34 ] introduced a multimodal perception model that processes diverse inputs, including facial images. Related to face retargeting, there are advancements that condition diffusion models directly on facial features, enabling tuning-free personalization. Approaches such as FastComposer [ 50 ] , PhotoVerse [ 4 ] , and PhotoMaker [ 29 ] utilize CLIP image features to represent the input subject, although these methods are inherently limited by CLIP’s facial encoding capabilities. To enhance fidelity, models such

Read the complete text