Generate Any Scene: Scene graph driven data synthesis for Visual Generation Training

Ziqi Gao 1 , Weikai Huang 1 , Jieyu Zhang 1 , Aniruddha Kembhavi 2 , Ranjay Krishna 1,2 1 University of Washington, 2 Allen Institute for Artificial Intelligence Code: https://github.com/RAIVNLab/GenerateAnyScene Dataset: GenerateAnyScene Dataset · 2024

Read Generate Any Scene: Scene graph driven data synthesis for Visual Generation Training by Ziqi Gao 1 , Weikai Huang 1 , Jieyu Zhang 1 , Aniruddha Kembhavi 2 , Ranjay Krishna 1,2 1 University of Washington, 2 Allen Institute for Artificial Intelligence Code: https://github.com/RAIVNLab/GenerateAnyScene Dataset: GenerateAnyScene Dataset (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08221.txt.

Introduction

Ziqi Gao and co-authors build a data engine for image and video models in 2024. Current training sets are noisy and weak on complex scenes with many linked objects. Models trained on them struggle to place objects in the right relations.

Their engine lists scene graphs from a structured list of objects, traits, and links. Each graph becomes a caption and a set of test questions with answers. Models trained on the made-up data improved on hard scene tasks. A small set of under eight hundred captions lifted one model's score by a tenth, showing the data can be potent in small doses.

Catalog record

CREATORZiqi Gao 1 , Weikai Huang 1 , Jieyu Zhang 1 , Aniruddha Kembhavi 2 , Ranjay Krishna 1,2 1 University of Washington, 2 Allen Institute for Artificial Intelligence Code: https://github.com/RAIVNLab/GenerateAnyScene Dataset: GenerateAnyScene Dataset
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generate Any Scene: Scene graph driven data synthesis for Visual Generation Training Ziqi Gao 1 , Weikai Huang 1 , Jieyu Zhang 1 , Aniruddha Kembhavi 2 , Ranjay Krishna 1,2 1 University of Washington, 2 Allen Institute for Artificial Intelligence Code: https://github.com/RAIVNLab/GenerateAnyScene Dataset: GenerateAnyScene Dataset Abstract Recent advances in text-to-vision generation excel in visual fidelity but struggle with compositional generalization and semantic alignment. Existing datasets are noisy and weakly compositional, limiting models’ understanding of complex scenes, while scalable solutions for dense, high-quality annotations remain a challenge. We introduce Generate Any Scene , a data engine that systematically enumerates scene graphs representing the combinatorial array of possible visual scenes. Generate Any Scene dynamically constructs scene graphs of varying complexity from a structured taxonomy of objects, attributes, and relations. Given a sampled scene graph, Generate Any Scene translates it into a caption for text-to-image or text-to-video generation; it also translates it into a set of visual question answers that allow automatic evaluation and reward modeling of semantic alignment. Using Generate Any Scene , we first design a self-improving framework where models iteratively enhance their performance using generated data. SDv1.5 achieves an average 4% improvement over baselines and surpasses fine-tuning on CC3M. Second, we also design a distillation algorithm to transfer specific strengths from proprietary models to their open-source counterparts. Using fewer than 800 synthetic captions, we fine-tune SDv1.5 and achieve a 10% increase in TIFA score for compositional and hard-concept generation. Third, we create a reward model to align model generation with semantic accuracy at a low cost. Using GRPO algorithm, we fine-tune SimpleAR-0.5B-SFT and surpass CLIP-based methods by +5% on DPG-Bench. Finally, we apply these ideas to the downstream task of content moderation where we train models to identify challenging cases by learning from synthetic data. 1 Introduction Despite the high-fidelity of modern generative models (text-to-image and text-to-video), we are yet to witness wide-spread adoption Brooks et al. (2024) ; Ramesh et al. (2022) ; Betker et al. (2023) ; Wang et al. (2024c) ; Chen et al. (2023b) . Controllability remains out of reach Peng et al. (2024) . Generated content appears realistic but often falls short of semantic alignment Huang et al. (2023) ; Sun et al. (2024) ; Ghosh et al. (2023) ; Hu et al. (2024a) . Users prompt models with a specific concept in mind. For example, when prompted to generate a scene of “a black dog chasing after a rabbit that is eating grass, in Van Gogh’s style, with starlight lightning”, some models are likely to generate an image of a dog but might miss the rabbit or get the style incorrect. We hypothesize that these limitations stem not only from architectural bottlenecks but more fundamentally from the lack of structured, compositionally rich training data Betker et al. (2023) , especially those with uncommon compositions. Popular datasets such as LAION Schuhmann et al. (2022) and CC3M Sharma et al. (2018) predominantly consist of web-crawled image-caption pairs, which are inherently noisy, weakly compositional, and biased toward single-object, coarse-grained descriptions. Such datasets lack explicit grounding of object-attribute relations and multi-object interactions, restricting models’ ability to generalize to complex visual scenes. Efforts to enhance caption quality Betker et al. (2023) ; Li et al. (2024d) have demonstrated that enhancing the compositional density and semantic richness of captions can significantly improve generative performance. Nevertheless, manual curation of such dense compositional annotations is labor-intensive, while automatic annotation methods (e.g., via MLMs) suffer from hallucination and semantic noise. Constructing a compositional dataset requires that we first define the space of the visual content . Scene graphs are one such representation of the visual space Krishna et al. (2017) ; Johnson et al. (2015) ; Ost et al. (2021) ; Johnson et al. (2018) ; Ji et al. (2020) , grounded in cognitive science Biederman (1987) . A scene graph represents objects in a scene as individual nodes in a graph. Each object is modified by attributes, which describe its properties. For example, attributes can describe the material, color, size, and location of the object in the scene. Finally, relationships are edges that connect the nodes. They define the spatial, functional, social, and interactive relationships between objects Lu et al. (2016) . For example, in a living room scene, a “table” node might have attributes like “wooden” or “rectangular” and be connected to a “lamp” node through a relation: “on top of”. This systematic scene graph structure provides simple yet effective ways to define and model the scene. As such, scene graphs are an ideal foundation for systematically defining the compositional space of visual content in text-to-vision generation. We introduce Generate Any Scene , a system capable of efficiently enumerating the space of scene graphs representing a wide range of visual scenes. Generate Any Scene composes scene graphs of any structure using a rich taxonomy of visual elements, translating each scene graph into an input caption and visual question answers to evaluate the output image or video. In particular, we first construct a rich taxonomy of visual concepts consisting of 28,787 objects, 1,494 attributes, 10,492 relations, 2,193 scene attributes from various sources. Based on these assets, Generate Any Scene can synthesize an almost infinite number of scene graphs of varying complexity Zhang et al. (2024) . Besides, Generate Any Scene allows configurable scene graph generation. For example, evaluators can specify the complexity level of the scene graph to be generated or provide a seed scene graph to be expanded. By automating these steps, our system ensures both scalability and adaptability, providing researchers and developers with diverse, richly detailed scene graphs and corresponding captions tailored to their specific needs. We also conduct comprehensive text-to-vision evaluations using our generated captions, as detailed in Appendix A . We show that Generate Any Scene can allow generation models to self-improve. Our diverse captions can facilitate a framework to iteratively improve Text-to-Vision generation models using their own generations. Given a model, we generate multiple images, identify the highest-scoring one, and use it as new fine-tuning data to improve the model itself. We fine-tune SDv1.5 Rombach et al. (2022b) and achieve an average of 4% performance boost compared with original models, and this method is even better than fine-tuning with the same amount of real images and captions from the Conceptual Captions CC3M over different benchmarks. We also use Generate Any Scene to design targeted distillation algorithms. Using our evaluations, we identify limitations in open-sourced models that their proprietary counterparts excel at. Next, we distill these specific capabilities from proprietary models. For example, DaLL-E 3 Betker et al. (2023) excels particularly in generating composite images with multiple parts. We distill this capability into SDv1.5 , effectively bridging the gap between DaLL-E 3 and SDv1.5 . After targeted fine-tuning, SDv1.5 achieves a 10% increase in TIFA score Hu et al. (2023) for compositional tasks and hard concept generation. Then we propose a low-cost scene graph-based reward model for RLHF Ouyang et al. (2022) in text-to-image generation. By leveraging synthetic scene graphs generated by Generate Any Scene , we generate exhaustive question-answer pairs that cover all objects, attributes, and relationships in the caption. Our method enables fine-grained, compositional reward modeling without manual annotation or heavy LLM inference. With GRPO

Read the complete text