AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition

Parsa Rahimi Noshanagh, Damien Teney, Sebastien Marcel · 2025

Read AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition by Parsa Rahimi Noshanagh, Damien Teney, Sebastien Marcel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11544.txt.

Introduction

Parsa Rahimi and co-authors introduce AugGen in a 2025 paper on face matching. Large data sets raise real fears about privacy, yet most fake data tools lean on outside data or ready made models. AugGen skips those crutches and trains only on the target data set.

The method draws new samples from a class based image model to boost the real data. Tests on eight face matching bench marks showed gains over real data alone. The gains beat prior fake data tools while using less real data. For teams short on data, the work shows fake samples can lift scores and ease privacy strain.

Catalog record

CREATORParsa Rahimi Noshanagh, Damien Teney, Sebastien Marcel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition Parsa Rahimi Noshanagh, Damien Teney, Sebastien Marcel Abstract The increasing reliance on large-scale datasets in machine learning poses significant privacy and ethical challenges, particularly in sensitive domains such as face recognition. Synthetic data generation offers a promising alternative; however, most existing methods depend heavily on external datasets or pre-trained models, increasing complexity and resource demands. In this paper, we introduce AugGen , a self-contained synthetic augmentation technique. AugGen strategically samples from a class-conditional generative model trained exclusively on the target FR dataset, eliminating the need for external resources. Evaluated across 8 FR benchmarks, including IJB-C and IJB-B, our method achieves 1–12% performance improvements , outperforming models trained solely on real data and surpassing state-of-the-art synthetic data generation approaches, while using less real data. Notably, these gains often exceed those from architectural enhancements, underscoring the value of synthetic augmentation in data-limited scenarios. Our findings demonstrate that carefully integrated synthetic data can both mitigate privacy constraints and substantially enhance recognition performance. Paper website: https://parsa-ra.github.io/auggen/ . Figure 1: Core idea of AugGen. AugGen boosts the model’s overall discriminative capabilities without requiring external datasets or pre-trained networks. To achieve this, we propose a novel sampling strategy using a conditional diffusion model—trained exclusively on the discriminator’s original data —this enables the generation of synthetic “mixes” of source classes. Incorporating these synthetic samples into the discriminator’s training, results in higher intra-class compactness and greater inter-class separation ( \theta_{\mathrm{ours}}>\theta_{\mathrm{baseline}} ) than models trained solely on the original data . 1 Introduction As machine learning increasingly relies on application-specific data, the demand for high-quality, accurately labeled datasets poses significant challenges. Privacy, legal, and ethical concerns amplify these difficulties, particularly in sensitive areas like human face images. A popular solution is synthetic data generation [ 54 , 2 , 38 ] , [ 3 ] , which leverages methods such as 3D-rendering graphics and generative models ( e.g. , GANs and diffusion models). Notably, synthetic data can surpass real data in model performance, as shown by [ 54 ] , where 3D-rendered face models with precise labels outperformed real-data-based models in tasks like face landmark localization and segmentation, highlighting the advantages of data synthesis, especially for tasks requiring dense annotations. Image generative models remain underutilized despite rapid advances in VAEs [ 27 ] , GANs [ 12 , 22 , 20 ] , and Diffusion models [ 47 , 19 , 21 , 17 , 13 ] . Comparisons of generative models often use metrics like Fréchet Distance (FD) [ 48 , 15 ] , which measure similarity to training data, or subjective user preferences for text-to-image tasks [ 10 ] . Figure 2: Unlike prior methods that depend on external data or pretrained generators, our self-contained synthetic augmentation framework improves recognition purely through its own generative process. As depicted in Figure 2 , currently, synthetic data generation involves training large-scale generative models [ 39 ] on datasets such as LAION-5B [ 43 ] , then refining them via fine-tuning, prompt engineering, or textual inversion [ 2 , 52 ] . This trend also applies to Face Recognition (FR), where synthetic data aims to mitigate privacy and ethical concerns. However, most methods still rely on large face datasets (which carry their own privacy issues) and auxiliary models, offering no clear advantage over existing real datasets. For instance, DCFace [ 25 ] generates diverse face images from multiple identities and uses robust FR systems and auxiliary networks to filter and balance samples. It remains unclear whether performance gains stem from the datasets, the generative models, or other factors—though larger, more diverse data typically improves results. Contrary to current trends, we advocate using generative models as an augmentation tool for FR training rather than replacing real datasets. Two key factors motivate this stance: 1. Synthetic datasets generated by diffusion models often leak training data [ 29 , 6 , 46 ] , offering no clear benefit over existing priors [ 25 , 5 , 32 , 49 , 55 ] . 2. Since responsible FR datasets are scarce and difficult to collect, we aim to boost performance with limited real data, thereby narrowing the gap between small-scale and large-scale training sets. In this paper, we focus on scenarios involving limited data, demonstrating how we can increase discriminative power by generating synthetic samples while only using a single labeled dataset . As illustrated in Figure 1 , we generate mixed classes that combine features from two or more source classes while preserving their distinct identities. We choose face recognition (FR) as our primary benchmark due to its unique difficulty, as it requires distinguishing between hundreds of thousands of identities within a highly structured input space. Moreover, FR is a privacy-sensitive task where responsible labeled data are scarce, making it an ideal setting for studying augmentation. Finally, it benefits from a range of well‐established benchmarks that enable consistent and meaningful evaluation. To enhance the effect of margin‐based losses used by state‐of‐the‐art discriminators in FR systems, given a dataset of \{({\bm{\mathsfit{X}}},{\bm{y}})\} , where {\bm{\mathsfit{X}}} is an image and {\bm{y}} is its corresponding label, we train a generative model, p({\bm{\mathsfit{X}}}\mid{\bm{y}}) , and a discriminator, p({\bm{y}}\mid{\bm{\mathsfit{X}}}) , on the same real dataset from scratch. We then introduce a simple yet novel sampling strategy to synthesize new examples. Empirically, we demonstrate that augmenting real data with these carefully generated synthetic samples leads to substantial improvements in the discriminator’s performance. Our main contribution is to validate this hypothesis in the context of face recognition (FR): H1 : A generative model can boost the performance of a downstream discriminator with an appropriate informed sampling, and augmenting the resulting data with the original data that was used for training the generative and discriminative models. Our contributions are summarized as follows: • We propose a simple yet effective sampling technique that strategically conditions a generative model to produce beneficial samples, enhancing the discriminator’s training process ( Subsection 3.1 ) without relying on any auxiliary models/data. • We show that mixing our AugGen data with real samples often surpasses even architectural-level improvements , underscoring that synthetic dataset generation can be as impactful as architectural advances ( Section 4 ). • We demonstrate that AugGen training can be as effective as adding up to 1.7 \times real samples, reducing the need for more face images while preserving performance ( Subsection 4.3 ). • We show that current generative metrics (e.g., FD, KD) are poorly correlated with downstream discriminative performance, emphasizing the need for improved proxy metrics ( Appendix F ). To the best of our knowledge, this is the first demonstration of generative image models effectively enhancing augmentation at this scale without relying on auxiliary models or external datasets. 2 Related Work Synthetic Data in Computer Vision. For a smaller number of class variations, ( e.g. , 2 or 3 classes for classification target), authors in [ 11 ] train separate generative models. This approach is not scalable for a higher number of classes and variations of our target ( e.g. , we have thousands of classes for training an FR system). In [ 2 ] , the authors fine-tuned pre-trained diffusion models on ImageNet classes after training on large text-image datasets, demonstrating improved

Read the complete text