Neuro-Symbolic Scene Graph Conditioning for Synthetic Image Dataset Generation

Giacomo Savazzi, Eugenio Lomurno, Cristian Sbrolli, Agnese Chiatti, Matteo Matteucci · 2025

Read Neuro-Symbolic Scene Graph Conditioning for Synthetic Image Dataset Generation by Giacomo Savazzi, Eugenio Lomurno, Cristian Sbrolli, Agnese Chiatti, Matteo Matteucci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17224.txt.

Introduction

Giacomo Savazzi and co-authors test, in 2025, whether structured scene descriptions can improve made-up training images for visual models. Training data is costly and scarce in special fields, and synthetic data still lags real data as tasks grow harder. Scene graphs record objects and their relations in a formal way.

The team used those graphs to guide image generation for a model that itself predicts scene graphs. Adding the synthetic data raised standard recall scores by up to a few points. The gain suggests the graphs add structural information that real data alone did not supply in the same form. The approach offers a route around data scarcity.

Catalog record

CREATORGiacomo Savazzi, Eugenio Lomurno, Cristian Sbrolli, Agnese Chiatti, Matteo Matteucci
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17224

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neuro-Symbolic Scene Graph Conditioning for Synthetic Image Dataset Generation Giacomo Savazzi, Eugenio Lomurno, Cristian Sbrolli, Agnese Chiatti, Matteo Matteucci Abstract As machine learning models increase in scale and complexity, obtaining sufficient training data has become a critical bottleneck due to acquisition costs, privacy constraints, and data scarcity in specialised domains. While synthetic data generation has emerged as a promising alternative, a notable performance gap remains compared to models trained on real data, particularly as task complexity grows. Concurrently, Neuro-Symbolic methods, which combine neural networks’ learning strengths with symbolic reasoning’s structured representations, have demonstrated significant potential across various cognitive tasks. This paper explores the utility of Neuro-Symbolic conditioning for synthetic image dataset generation, focusing specifically on improving the performance of Scene Graph Generation models. The research investigates whether structured symbolic representations in the form of scene graphs can enhance synthetic data quality through explicit encoding of relational constraints. The results demonstrate that Neuro-Symbolic conditioning yields significant improvements of up to +2.59% in standard Recall metrics and +2.83% in No Graph Constraint Recall metrics when used for dataset augmentation. These findings establish that merging Neuro-Symbolic and generative approaches produces synthetic data with complementary structural information that enhances model performance when combined with real data, providing a novel approach to overcome data scarcity limitations even for complex visual reasoning tasks. K eywords Synthetic Data Generation \cdot Neuro-Symbolic AI \cdot Scene Graph Generation \cdot Data Augmentation \cdot Visual Reasoning. 1 Introduction Recently, research on synthetic dataset generation has emerged as a critical direction for advancing deep learning architectures. As models continue to scale in complexity and parameter count, acquiring and annotating large-scale real-world datasets remains prohibitively resource-intensive due to costs, privacy regulations, and domain-specific data scarcity. Synthetic datasets offer a scalable alternative with reduced annotation overhead and controllable attribute distribution. Despite these advantages, models trained exclusively on synthetic data consistently underperform compared to those trained on real datasets, particularly on high-level reasoning tasks. This performance gap widens further as task complexity increases, even when scaling dataset size. To address these limitations, various conditioning techniques for generative models have been explored. Simultaneously, Neuro-Symbolic (NeSy) approaches have gained traction in the machine learning community [ 1 ] . These methods integrate neural representation learning with symbolic reasoning frameworks, combining data-driven optimization with structured knowledge. While NeSy architectures have demonstrated enhanced generalization in reasoning tasks, their potential for improving synthetic data generation remains largely unexplored. Scene graphs—structured representations where nodes represent objects and edges represent relations—offer a promising symbolic framework for conditioning image generation. This work explores the integration of such NeSy approaches into dataset generation to improve performance on complex tasks, particularly Scene Graph Generation (SGG) from images. The hypothesis posits that scene graphs encode useful background knowledge that can guide the generation process, ensuring synthetic data adhere to structural and semantic constraints. To the best of current knowledge, this represents the first systematic exploration of NeSy conditioning for synthetic dataset generation in the context of SGG tasks. The experimental framework employs SGAdapter [ 2 ] for Scene Graph-to-Image generation, with Stable Diffusion 2.0 [ 3 ] as the baseline model. CausalTDE [ 4 ] serves as the SGG evaluation model. The key contributions of this work are threefold: (1) proposing a novel framework that integrates scene graph-based symbolic knowledge into synthetic dataset generation; (2) demonstrating that structurally-guided generation produces complementary training signals that enhance augmentation efficacy despite lower perceptual fidelity; and (3) providing empirical evidence that NeSy conditioning addresses specific limitations in conventional synthetic data generation for complex visual reasoning tasks. 2 Related Works The creation of effective synthetic datasets for augmentation and model training necessitates generative architectures capable of high-quality image synthesis. Among these, Generative Adversarial Networks (GAN) and Denoising Diffusion Probabilistic Models (DDPM) have emerged as predominant families of methods [ 5 , 3 ] . Despite their distinct operational principles, both approaches can be conditioned in various ways to produce images that correspond to specific domains or visual concepts. 2.1 Conditioning Generative Models The evolution of GANs quickly led to conditional formulations, allowing the generation process to be guided by supplementary information provided to both generator and discriminator components [ 6 ] . Particularly promising has been the conditioning on class labels, which offers benefits through training set expansion and noise input variance truncation, ultimately enhancing the quality of generated images [ 7 ] . The landscape of generative models has been further enriched by the exploration of text-prompt conditioning. Ku et al. advanced this field by developing a regressor that produces refined text-conditioned vectors, enabling nuanced control over subtle features in generated images [ 8 ] . Complementary approaches harnessed CLIP’s comprehensive understanding of visual scenes, simultaneously reducing training requirements while elevating synthesis quality [ 9 , 10 ] . In the realm of diffusion models, textual prompts are similarly employed, often alongside guidance strategies that carefully balance fidelity with diversity. Nichol et al. demonstrated the superiority of classifier-free guidance—achieved by blending predictions with and without text conditioning—over approaches relying on CLIP guidance [ 11 ] . The widely-used Stable Diffusion implements this concept through its Guidance Scale parameter, which regulates how closely generated images adhere to provided text prompts [ 12 ] . 2.2 Learning from Synthetic Data The research trajectory has recently shifted from generating individual synthetic samples toward producing comprehensive synthetic datasets, prioritising collective diversity over the visual quality of individual images. This transition has sparked numerous studies focused on enhancing the diversity of synthetic datasets. Researchers have discovered that minimal prompt engineering combined with reduced guidance scale in Stable Diffusion can yield performance comparable to that achieved with real data [ 13 ] . The critical importance of the guidance scale parameter for synthetic image generation has been found to benefit downstream classification tasks [ 14 ] . A parallel finding revealed that expanding input noise variance enhances the informativeness of GAN-generated datasets, even when this approach might compromise the visual fidelity of individual samples [ 15 ] . Shipard et al. introduced an innovative approach based on multiple textual variations, amalgamated into a "Bag of Tricks" that effectively broadens synthetic dataset diversity. Taking a different approach, Lei et al. employed captioning models to craft detailed prompts, combining image descriptions with class labels to enhance the distinction between foreground and background elements [ 16 ] . Complementing these generation-focused methods, researchers have explored post-generation enrichment techniques. The Generative Knowledge Distillation technique elegantly leverages models trained on real data (Teacher) to produce nuanced soft labels for synthetic datasets, which then serve to train new models (Student) [ 17 ] . This approach has demonstrated remarkable effectiveness in enhancing classification performance while increasing resilience against inference attacks in both local and federated learning environments [ 18 ] . 2.3 Neuro-Symbolic Artificial Intelligence Neuro-Symbolic (NeSy) AI represents an innovative approach that integrates neural and symbolic paradigms in Artificial Intelligence. This integration strategically combines the perceptual strengths of neural systems with the reasoning capabilities of symbolic approaches, creating a synergy that aims at addressing the limitations of both paradigms. Neural systems excel at processing unstructured data and learning patterns through approximation via deep neural networks. Their strength lies in perception tasks such as image recognition and automatic information extraction from vast

Read the complete text