Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification

William Yang, Xindi Wu, Zhiwei Deng, Esin Tureci, Olga Russakovsky, Google DeepMind · 2025

Read Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification by William Yang, Xindi Wu, Zhiwei Deng, Esin Tureci, Olga Russakovsky, Google DeepMind (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24078.txt.

Introduction

William Yang and four co-authors propose a better way to make fake training images for fine grained image sorting. Tasks like telling bird species apart need many labeled images, and text to image models can help fill the gap. Tuning those models on a few real images can improve quality but also cause copying and reduce variety.

Their method, BOB, first pulls out traits that are not tied to the class, such as background and pose. It then trains with those traits stated clearly and averages them out when making new images. Broad tests show cleaner, more varied data and better sorting results, with less copying of the small real set.

Catalog record

CREATORWilliam Yang, Xindi Wu, Zhiwei Deng, Esin Tureci, Olga Russakovsky, Google DeepMind
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24078

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained Classification William Yang, Xindi Wu, Zhiwei Deng, Esin Tureci, Olga Russakovsky, Google DeepMind Abstract Text-to-image (T2I) models are increasingly used for synthetic dataset generation, but generating effective synthetic training data for classification remains challenging. Fine-tuning a T2I model with a few real examples can help improve the quality of synthetic training data; however, it may also cause overfitting and reduce diversity in the generated samples. We propose a fine-tuning strategy BOB ( B eyond OB jects) to mitigate these concerns for fine-grained classification. Given a small set of real examples, we first extract class-agnostic attributes such as scene background and object pose. We then explicitly condition on these attributes during fine-tuning of the T2I model and marginalize them out during generation. This design mitigates overfitting, preserves the T2I model’s generative prior, reduces estimation errors, and further minimizes unintended inter-class associations. Extensive experiments across multiple T2I models, backbones, and datasets show that our method achieves state-of-the-art performance in low-shot fine-grained classification when augmented with synthetic data. Concretely, BOB outperforms DataDream by 7.4% on the Aircraft dataset (from 50.0% to 57.4% when fine-tuning a CLIP classifier with five real images augmented with 100 synthetic images). In three of the four benchmarks, fine-tuning downstream models with 5 real images augmented with BOB achieves better performance than fine-tuning with 10 real images. Collectively, BOB outperforms prior art in 18 of 24 experimental settings, with 2+% accuracy improvements in 14 of these settings. Code is available at https://github.com/princetonvisualai/BeyondObjects . 1 Introduction Powerful text-to-image (T2I) models trained on internet-scale datasets ( Schuhmann et al. 2022 ; Rombach et al. 2022 ) have shown promise in the creation of synthetic data for representation learning ( Tian et al. 2023 ; Tian et al. 2024 ) , 3D synthesis ( Poole et al. 2023 ) , and image editing ( Hertz et al. 2022 ; Mokady et al. 2022 ; Brooks et al. 2023 ) . However, there remains a considerable performance gap when using synthetic data as training data for downstream tasks such as classification ( Burg et al. 2023 ; Fan et al. 2024 ; Geng et al. 2024 ) . Ideally, given a target classification task described in language (e.g., “train an aircraft classifier to distinguish a 747-300 from a 747-400”), a T2I model can be directly used to generate training images of the desired classes. A key challenge limiting T2I models from generating informative images is the model estimation errors caused by a misalignment between the T2I model’s learned distribution and the target task ( Geng et al. 2024 ) . For example, the visual difference between 747-300 and 747-400 is subtle: the presence of a winglet on a slightly longer wing. In consequence, the introduction of low level artifacts and incorrect visual compositions introduce a challenge for the task of fine-grained recognition. One approach to mitigate model estimation errors for these classes is to provide a few real images to fine-tune the T2I model ( Wang et al. 2024 ; Kim et al. 2024 ) . The underlying T2I model needs to rely heavily on additional guidance from provided examples to generate not only accurate samples but also diverse enough samples to augment training of downstream classifier. However, operating in the few-shot regime requires special considerations ( Yue et al. 2020 ) . The increased expressivity from fine-tuning can introduce a trade-off where the T2I model starts to overfit to the few examples, losing its strong world prior and hurting the diversity of the synthetic dataset. In this work, we tackle fine-grained classification with synthetic data generation by introducing BOB ( B eyond OB jects). BOB mitigates overfitting during T2I fine-tuning by obtaining the background and pose for each example via a captioning model and incorporating them into the text condition. During data generation, background and pose pairs are sampled across the dataset, effectively marginalizing out any unintended associations across classes. We provide a comprehensive evaluation across three backbones, two T2I models, four datasets, two data scales, and seven existing methods, to demonstrate the effectiveness of BOB . We observe the most considerable gain on Aircraft, a dataset where T2I perform poorly and fine-tuning benefits the most. Training the CLIP model ( Radford et al. 2021 ) using 5 real images augmented with 100 synthetic images per class result in 7.4% increase in classification accuracy from 50.0% when augmented with the previous fine-tuning method DataDream ( Kim et al. 2024 ) to 57.4% with BOB . Further, across three of the four dataset (Aircraft, Cars, and CUB), using 5 real images augmented with BOB generated images results in better classification performance than using 10 real images: e.g., CLIP fine-tuned on CUB achieves accuracy of 75.8% with 5 real images augmented with BOB generated images and only 74.6% with 10 real images without augmentation. Overall, BOB outperforms existing state-of-the-art methods by at least 2% on 18 of the 24 experimental settings (backbone, dataset source, and dataset size). On the six remaining settings (on the Pets ( Parkhi et al. 2012 ) dataset), BOB offers competitive performance within 1% of state-of-the-art. To summarize, we make the following contributions: 1. We introduce stronger supervision with more detailed captioning during T2I fine-tuning to mitigate model overfitting and enhancing prior preservation (§ 3.1 ). 2. We marginalize out unintended inter-class associations by randomly sampling class-agnostic features (background, pose) across the whole dataset (§ 3.2 ). 3. We provide a comprehensive evaluation (§ 4 ) across seven previous methods, two T2I models, and 24 different experimental settings to demonstrate our methods outperforms previous methods in 18 of the 24 settings with competitive performance in the rest. 2 Related Work Personalization. Many personalization methods serve as inspiration for approaches aimed at synthetic data generation for classification. Personalization methods seek to guide and control T2I models beyond text descriptions, typically using a few image exemplars. These approaches have proven effective at tailoring T2I models to reproduce highly specific visual concepts ( Gal et al. 2023 ; Ruiz et al. 2023 ; Kumari et al. 2023 ; Ye et al. 2023 ; Zhang et al. 2023 ; Li et al. 2023b ; Zong et al. 2024 ; Zhao et al. 2025 ) . However, while they enable strong concept-level control, their objectives differ from those required for classification-oriented synthetic data. They emphasize concept fidelity over diversity, often limiting intra-class variation and inter-class separability—both essential for robust classifier training. Therefore, such personalization methods are insufficient for addressing the challenges of synthetic data generation for classification. Synthetic data for classification. The field of synthetic data generation for classification has initiated a significant shift, moving toward leveraging powerful T2I models. This paradigm shift contrasts sharply with traditional data augmentation methods, such as CutMix ( Yun et al. 2019 ) and Mixup ( Zhang et al. 2018 ) . These techniques interpolate between existing data, which helps smooth the decision function but is limited in sample diversity and fidelity. In contrast, a pre-trained T2I model provides a ”world prior” that significantly enhances both sample diversity and fidelity. Early works demonstrated the utility of T2I models for classification. Real Guidance ( He et al. 2023 ) demonstrated that utilizing

Read the complete text