SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing
Thanh-Nhan Vo, Trong-Thuan Nguyen ♠, Tam V. Nguyen, Minh-Triet Tran · 2026
Read SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing by Thanh-Nhan Vo, Trong-Thuan Nguyen ♠, Tam V. Nguyen, Minh-Triet Tran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21498.txt.
Introduction
Thanh-Nhan Vo, Trong-Thuan Nguyen, Tam V. Nguyen, and Minh-Triet Tran unify image creation and image editing under one frame. A scene graph describes the objects in a picture and how they relate. Most tools handle making an image and changing it as separate jobs, which breeds inconsistency.
Edits can drift away from the layout of the original scene. A graph gives both jobs the same structured plan to follow. It also gives users control over where objects sit and how they connect. The unified design aims for images that stay coherent from first draft through every edit.
Catalog record
| CREATOR | Thanh-Nhan Vo, Trong-Thuan Nguyen ♠, Tam V. Nguyen, Minh-Triet Tran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing Thanh-Nhan Vo, Trong-Thuan Nguyen ♠, Tam V. Nguyen, Minh-Triet Tran Abstract Recent advancements in Generative Artificial Intelligence (GenAI) have significantly enhanced the capabilities of both image generation and editing. However, current approaches often treat these tasks separately, leading to inefficiencies and challenges in maintaining spatial consistency and semantic coherence between generated content and edits. Moreover, a major obstacle is the lack of structured control over object relationships and spatial arrangements. Scene graph-based methods, which represent objects and their interrelationships in a structured format, offer a solution by providing greater control over composition and interactions in both image generation and editing. To address this, we introduce SimGraph, a unified framework that integrates scene graph-based image generation and editing, enabling precise control over object interactions, layouts, and spatial coherence. In particular, our framework integrates token-based generation and diffusion-based editing within a single scene graph-driven model, ensuring high-quality and consistent results. Through extensive experiments, we empirically demonstrate that our approach outperforms existing state-of-the-art methods. ${}^{\spadesuit}$ ${}^{\spadesuit}$ footnotetext: Corresponding Author 1 Introduction Recent advances in Generative Artificial Intelligence (GenAI) have revolutionized image creation, demonstrating remarkable capabilities in both generation and editing. In particular, models like Stable Diffusion [ 20 ] and DALL·E [ 19 ] have demonstrated the ability to generate high-quality, diverse images from textual descriptions. By leveraging diffusion-based models and transformers, these models can synthesize realistic images conditioned on natural language prompts, providing users with significant creative freedom. Similarly, image editing has made substantial strides, with models now capable of manipulating images to meet specific criteria, such as altering objects, interactions, backgrounds, or other attributes. Thus, these advances in image generation and editing have broadened the potential applications of GenAI across entertainment, design, and education. However, despite these impressive developments of GenAI, significant challenges remain in unifying image generation and editing into a single, seamless framework. While text-driven generation models offer flexibility and creativity, they frequently struggle to preserve background details and maintain spatial consistency when edits are introduced, often resulting in artifacts and unrealistic compositions. On the other hand, scene graph-based methods, which utilize structured representations of objects and their relationships, offer enhanced control over spatial layouts and object interactions [ 9 ] [ 27 ] . Nevertheless, these methods often require additional fine-tuning or training to handle editing tasks, thereby increasing computational costs and inference time [ 11 ] . Therefore, the current challenge lies in integrating both image generation and editing into a unified framework that is efficient and capable of producing quality, coherent outputs. In this work, we aim to introduce a unified framework that can seamlessly combine the strengths of both image generation and editing. Existing systems often treat these tasks separately, resulting in inefficiencies, particularly when dealing with complex scene graphs that require precise manipulation. Furthermore, our framework maintains semantic consistency between generated content and modified elements, which remains a persistent issue in current methods [ 29 ] [ 4 ] . By addressing these gaps, our approach aims to provide a more flexible, efficient, and high-quality solution, enabling seamless transitions between generation and editing while ensuring high fidelity and spatial coherence in the final outputs. Our primary contribution is the introduction of SimGraph, which simultaneously integrates scene graph-based image generation and editing into a single pipeline. Specifically, we propose a unified approach that leverages a single scene graph-driven model to handle both tasks, allowing for fine-grained control over object interactions and spatial relationships. In addition, our framework incorporates token-based generation (Section 4.2 ) and diffusion-based editing (Section 4.3 ), which are conditioned on scene graph-derived captions and edited prompts to ensure consistency and high-quality results across tasks. Moreover, through extensive experiments, we show that our approach outperforms existing scene graph-based models in image generation [ 9 ] [ 27 ] [ 28 ] and editing tasks [ 4 ] [ 27 ] , delivering significant improvements in fidelity, efficiency, and semantic alignment. The remainder of this paper is organized as follows. Sec. 2 reviews related work on scene graph-based image generation and editing. In Sec. 3 formulates the problem of integrating image generation and editing from scene graphs. Additionally, Sec. 4 , we present the unified framework, detailing the scene graph extraction, token-based image generation, and diffusion-based image editing processes. The experimental setup and results are discussed in Sec. 5 , where we present both quantitative and qualitative evaluations of our approach. Finally, we conclude the paper in Sec. 6 and discuss potential directions for future research. 2 Related Work Scene graphs provide a structured representation of visual scenes by modeling objects as nodes and their interrelationships as edges. In particular, this graph-based approach provides a rich, semantic understanding of the scene, enabling more effective manipulation and control over the visual content [ 26 ] [ 17 ] . By explicitly capturing object-level attributes and the spatial and relational context between them, scene graphs have become a powerful tool for a variety of image generation and editing tasks. In addition, scene graphs can be extended to the video domain [ 16 ] [ 15 ] [ 14 ] , enabling dynamic, temporally coherent scene generation and editing. The following subsections investigate the application of scene graphs in both generating realistic images and performing sophisticated image edits. 2.1 Image Generation from Scene Graphs The task of generating images from scene graphs, which represent the relationships between objects in a scene, has attracted considerable attention in recent years due to its potential for controlled, realistic image synthesis. Early works, such as SG2IM [ 9 ] , leveraged graph convolutional networks to process scene graphs and generate images by predicting bounding boxes and segmentation masks, which are then refined through cascaded networks. However, previous methods often struggled with the complexities of large graphs and the semantic consistency of object relationships. In response, Herzig et al. [ 6 ] address these limitations by learning canonical graph representations that improve performance with complex scene graphs, providing better robustness to noise and improved generalization across semantically equivalent graphs. In addition, these methods are further complemented by advancements such as PasteGAN [ 11 ] , which introduces a semi-parametric approach for controlling object appearance and interactions via external object crops, ensuring high-quality, semantically consistent results. Recent progress has also seen the adoption of diffusion models. In particular, SGDiff [ 27 ] enhances alignment between scene graphs and images by optimizing graph embeddings via masked autoencoding and a contrastive loss. Specifically, SGDiff enables more accurate image generation and manipulation from scene graphs, outperforming earlier methods in image quality and alignment. In addition, SG-Adapter [ 21 ] enhances text-to-image generation by incorporating scene graph guidance, which involves multiple objects and relationships. Specifically, SG-Adapter shows the potential to improve alignment between textual descriptions and generated images by refining text embeddings with structured representations. 2.2 Image Editing from Scene Graphs Recent advancements in scene graph-based image generation and manipulation have leveraged the structured, hierarchical nature of scene graphs, which significantly improves image synthesis and editing. SGEdit [ 29 ] integrates large language models with generative models, which facilitates precise image modifications at the object level. In particular, SGEdit outperforms existing image editing methods in terms