Representation-Conditioned Diffusion Models for Guided Training Data Generation
Nithesh Chandher Karthikeyan, Jonas Unger, Gabriel Eilertsen · 2026
Read Representation-Conditioned Diffusion Models for Guided Training Data Generation by Nithesh Chandher Karthikeyan, Jonas Unger, Gabriel Eilertsen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27495.txt.
Introduction
Nithesh Chandher Karthikeyan, Jonas Unger, and Gabriel Eilertsen train diffusion models on synthetic images and test them as classifiers. Images made from DINOv2, DINOv3, and CLIP features beat class-prompted images by about 11 points on ImageNet100. Growing the synthetic set further beats a model trained on real data, and the generated images also improve standard data augmentation.
Catalog record
| CREATOR | Nithesh Chandher Karthikeyan, Jonas Unger, Gabriel Eilertsen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16561 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Representation-Conditioned Diffusion Models for Guided Training Data Generation Nithesh Chandher Karthikeyan, Jonas Unger, Gabriel Eilertsen Abstract Data availability remains a critical bottleneck in many deep learning applications. Large-scale datasets are often expensive to collect, curate and annotate, which can limit the scalability and applicability of supervised learning methods. In this work, we evaluate the classification performance of models trained on synthetic image datasets produced by generative deep learning. In particular, we use latent diffusion models conditioned on learned representations from DINOv2, DINOv3, and CLIP. Our results demonstrates that this representation-conditioned formulation significantly outperforms class-conditioned generation by a large margin (+10.76 p.p. top-1 accuracy on ImageNet100), by improving sample quality and mode coverage. Furthermore, by scaling the size of the synthetic dataset, we are able to outperform a classifier trained on the real data (+2.0 p.p. top-1 accuracy). We also demonstrate how generated images can be used for augmentation purposes, outperforming classical augmentation methods, and how the conditioning space can be used for sample filtering to further improve training value. Collectively, these findings highlight that representation-conditioned diffusion models provide a promising approach for augmenting, complementing, or potentially replacing real-world datasets in large-scale visual learning tasks. 1 Introduction Figure 1 : Overall pipeline of the methodology, which consist of two parts: (a) Synthetic Data Generation using Representation Conditioned Diffusion Model (RCDM) and (b) Classifier Based Evaluation for Comparative analysis. With continuous advancements in deep learning, there is a constant demand for larger and diverse training datasets. At the same time, generative modeling has experienced significant progress over the last decade, particularly with the emergence of diffusion models 17 ; 7 . These developments have sparked a growing interest in using these models to expand training datasets via synthetic data generation, e.g., by steering or fine-tuning a general pre-trained text-to-image model 1 ; 6 ; 18 . In this paper, we focus on the most challenging scenario for generative dataset synthesis. Specifically, we assume that (1) only the target dataset is available, preventing the use of existing text-to-image models, (2) the downstream model is trained exclusively on the generated dataset (without access to real images), and (3) no additional annotations, such as text prompts, are available to guide generation. Unlike conventional data augmentation, where augmented images only introduce variations to the real samples, training solely on synthetic dataset requires the generative model to capture the full diversity of the real data distribution. Further, this scenario is also relevant in situations where sharing of real data are restricted due to privacy concerns, such as in medical imaging. Text-to-image diffusion models can produce high quality and diverse samples, with their performance largely benefiting from guidance provided by text prompts, in addition to large models and training dataset. To increase the quality and diversity without text guidance, we consider representation-conditioned diffusion models (RCDMs) 2 ; 10 ; 8 . In RCDMs, guidance is provided from extracted representations of each image, which has been demonstrated to significantly increase the quality of generation as compared to unconditioned or class-conditioned generation 10 . In this work, we evaluate how these improvements translate to the case of training data generation. We provide experiments with a representation-conditioned latent diffusion model 14 trained on ImageNet100, which is a challenging dataset due to its diversity and relatively limited size. We demonstrate that RCDMs show great promise for synthetic dataset generation, surpassing the class-conditioned generation by a large margin. Furthermore, by scaling the size of the synthetic dataset, we are able to outperform the model trained only on the real dataset. We see this as quite remarkable evidence for the promise of RCDMs and using generative deep learning for training data synthesis. In summary, we present the following contributions: • We evaluate the classification performance of models trained on synthetic datasets generated by RCDMs, investigating the impact of dataset size and representation used for conditioning (DINOv2 11 , DINOv3 16 , and CLIP 12 ). • We show that, by scaling the synthetic dataset, RCDMs can surpass the performance of models trained solely on real data, highlighting their potential for large-scale dataset generation. • We further demonstrate that RCDM generated images are effective for data augmentation, outperforming classical techniques such as RandAugment 4 and MixUp 19 , and how the representation space used for conditioning allows simple sample filtering. While this work demonstrates a first case study of using RCDMs for dataset generation, the results open up for many possibilities in future research, for improving or replacing datasets by means of generative synthesis. 2 Method In this section, we describe the pipeline used for generation and validation of synthetic training data. The pipeline consists of two main stages: synthetic data generation using Representation Conditioned Diffusion Models (RCDMs) and evaluation using a classifier. Figure 1 illustrates the overall workflow, which is described in detail in the following subsections. 2.1 Synthetic Data Generation using RCDM Training the RCDM involves two stages. First, input images are projected into a lower-dimensional representation space using a pre-trained encoder. Second, a latent diffusion model (LDM) 14 is trained using these representations as conditioning signals 9 . For the experiments, we train three RCDM models conditioned on representations extracted from the image encoder of CLIP 12 (also called unCLIP 13 ), DINOv2 11 , and DINOv3 16 . Specifically, we use the CLS token embeddings from the DINOv2 ViT-B/14 and DINOv3 ViT-B/16 architectures, each producing 768-dimensional representations, and embeddings from CLIP ViT-B/32 with a dimensionality of 512. Training is performed on the ImageNet-100 dataset, a subset of ImageNet containing 100 classes 15 . The dataset contains 130k images, all resized to a resolution of 256×256. Each model was trained for 125 epochs with a batch size of 16, resulting in approximately 1.16 million training steps. For sampling, we use the Denoising Diffusion Probabilistic Model (DDPM) 7 with 100 inference steps. To increase dataset diversity beyond the original 130k images, images are generated using different random seeds, each corresponding to a distinct latent noise initialization. This produces diverse images while preserving semantic content through representation conditioning 9 . 2.2 Classifier Based Evaluation To evaluate the quality of the synthetic ImageNet-100 data in relation to the real dataset, we train separate classifiers on the real and synthetic datasets using the ResNet-50 5 architecture. This evaluation measures how effectively models trained on synthetic data can learn discriminative features comparable to those learned from real data. The training procedure follows the standard configuration used for training ResNet-50 on ImageNet in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Specifically, the classifier is trained using the same architectural setup, data preprocessing pipeline, and optimization strategy adopted in ILSVRC. By maintaining an identical training configuration across both datasets, the comparison provides a consistent and fair assessment of the representational quality and class-discriminative structure present in the generated synthetic images. Figure 2 : Scaling synthetic datasets for classifier training. Top-1 (left) and Top-5 (right) accuracy of a ResNet-50 trained on synthetic ImageNet-100 datasets generated using representation-conditioned diffusion models with DINOv2, DINOv3, and CLIP representations, compared to a class-conditioned LDM baseline and ResNet-50 trained on real ImageNet100 baseline. 3 Results This section presents the experimental evaluation of the proposed framework. We assess the quality and usefulness of the generated