Graph-PiT: Enhancing Structural Coherence in Part-Based Image Synthesis via Graph Priors

Junbin Zhang 1 , Meng Cao 2 , Feng Tan 1 , Yikai Lin 1 , Yuexian Zou 1,* · 2026

Read Graph-PiT: Enhancing Structural Coherence in Part-Based Image Synthesis via Graph Priors by Junbin Zhang 1 , Meng Cao 2 , Feng Tan 1 , Yikai Lin 1 , Yuexian Zou 1,* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06074.txt.

Introduction

Zhang and colleagues' 2026 paper presents Graph-PiT for part-based image synthesis, linking parts instead of treating them as a loose set. Parts become nodes and their links become edges. A hierarchical graph neural network passes messages both ways between part-level super-nodes and IP+ token sub-nodes, tuning embeddings early. A Laplacian smoothness loss and an edge-reconstruction loss give nearby parts matching embeddings. Tests on synthetic and real images show better part layouts than vanilla PiT. The setup stays compatible with IP-Prior.

Catalog record

CREATORJunbin Zhang 1 , Meng Cao 2 , Feng Tan 1 , Yikai Lin 1 , Yuexian Zou 1,*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43978
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06074

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Graph-PiT: Enhancing Structural Coherence in Part-Based Image Synthesis via Graph Priors Junbin Zhang 1 , Meng Cao 2 , Feng Tan 1 , Yikai Lin 1 , Yuexian Zou 1,* Abstract Abstract— Achieving fine-grained and structurally sound controllability is a cornerstone of advanced visual generation. Existing part-based frameworks treat user-provided parts as an unordered set and therefore ignore their intrinsic spatial and semantic relationships, which often results in compositions that lack structural integrity. To bridge this gap, we propose Graph-PiT, a framework that explicitly models the structural dependencies of visual components using a graph prior. Specifically, we represent visual parts as nodes and their spatial-semantic relationships as edges. At the heart of our method is a Hierarchical Graph Neural Network (HGNN) module that performs bidirectional message passing between coarse-grained part-level super-nodes and fine-grained IP+ token sub-nodes, refining part embeddings before they enter the generative pipeline. We also introduce a graph Laplacian smoothness loss and an edge-reconstruction loss so that adjacent parts acquire compatible, relation-aware embeddings. Quantitative experiments on controlled synthetic domains (character, product, indoor layout, and jigsaw), together with qualitative transfer to real web images, show that Graph-PiT improves structural coherence over vanilla PiT while remaining compatible with the original IP-Prior pipeline. Ablation experiments confirm that explicit relational reasoning is crucial for enforcing user-specified adjacency constraints. Our approach not only enhances the plausibility of generated concepts but also offers a scalable and interpretable mechanism for complex, multi-part image synthesis. The code is available at https://github.com/wolf-bailang/Graph-PiT. I Introduction Controllable image synthesis has advanced dramatically with the advent of large-scale diffusion models and rich conditioning mechanisms [ 19 ] , [ 14 ] , [ 7 ] , [ 5 ] . However, fine-grained compositional control remains challenging. In industrial design, 3D modeling, and creative AI, designers often assemble new concepts by arranging existing parts or sub-components (e.g., prototyping a robot from limbs, designing furniture layouts, or crafting novel characters). However, most diffusion-based generators assume text or global image prompts, ignoring explicit part relationships. For example, PiT (Piece it Together) [ 18 ] encodes each part into an IP^{+} token grid and composes the set through an IP-Prior diffusion model, but it still treats the parts as an unordered set, allowing components to be placed in implausible configurations. In contrast, many real-world domains require structured assemblies: mechanical parts must join at specific hinges, furniture components must align according to room geometry, and product elements, such as wheels and chassis, have fixed spatial adjacency. As noted by Chen et al. [ 20 ] on 3D generation, assets typically consist of a single, fused representation without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts. To address this, we propose Graph-PiT, which endows part-based generative modeling with explicit structural priors. We represent each input part as a node in a graph and connect edges to encode spatial or semantic relationships (e.g., adjacency, attachment points, or user-specified constraints). A hierarchical Graph Neural Network (HGNN) propagates information along this graph so that inter-part constraints inform generation. The refined part embedding then conditions a latent diffusion model (e.g., a latent SDXL generator [ 13 ] ), ensuring that the output image not only matches each part’s appearance but also respects the desired layout. By design, Graph-PiT can take both image patches and a skeleton graph (or layout) as inputs and synthesize images that are globally coherent and physically plausible in the 2D part-layout sense considered here. In this paper, structural coherence refers to respecting adjacency and topology among visible parts rather than enforcing full 3D geometry, viewpoint, or occlusion consistency. In creative workflows, this allows, for instance, specifying that a chair’s legs attach at the seat and that robot arms connect to the torso, yielding more realistic and controllable results. As PartComposer [ 15 ] demonstrates, artists need part-level control; current models often “ignore the attribute details” at the part level, and our graph-based approach closes this gap by encoding structural priors. Our contributions are threefold: • Graph prior. We introduce a two-tiered graph over input parts whose edges encode spatial/semantic constraints, derived either automatically during training or directly from user-specified layouts. • Hierarchical graph aggregator. We design an HGNN that couples part-level super-nodes and token-level IP^{+} sub-nodes token features via bidirectional top-down/bottom-up message passing and two structural regularizers (Laplacian smoothness and edge reconstruction), going beyond treating the graph simply as an extra token encoder. • Graph-conditioned diffusion prior compatible. We condition the IP-Prior on the graph-aggregated tokens in a way that degenerates exactly to the original PiT model when graph tokens are removed, enabling a clean comparison, ablations, and drop-in replacement in existing PiT pipelines. II Methodology In this section, we present Graph-PiT, a framework that elevates part-based image synthesis from an unordered collection of components to a structurally coherent generation process guided by relational priors. For related work, please refer to Supplementary Section V. For preliminary knowledge on IP^{+} representation and GNN background, please refer to Supplementary Section VI. II-A Problem Formulation Part-Based Concept Generation. Let \mathcal{I}=\{I_{1},I_{2},\ldots,I_{N}\} be an unordered collection of N part images (e.g., a chair’s leg, backrest, and seat cushion). Our objective is to generate a complete concept x (an embedding or rendered image) that plausibly integrates all provided parts while satisfying structural constraints encoded by a graph G . Formally, we seek a generative distribution \textbf{x}\sim\textbf{p}_{\theta}(\textbf{x}|\{I_{i}\}_{i=1}^{N},\textbf{G}) (1) where \mathbf{G}=(\textbf{V},\textbf{E}) is a part-relationship graph whose nodes V correspond to the N parts and whose edge set E encodes spatial or semantic adjacency. Motivation for Graph Prior. PiT treats user-provided parts as an unordered set and therefore ignores the relations that naturally exist between components. We instead augment generation with an explicit graph prior in which each part is a node and edges encode spatial or semantic adjacency, as illustrated in Figure 1 . Fig. 1: Graph Prior Visualization. To inject structural awareness, we condition generation on an explicit graph prior G . Each part I_{i} is mapped to a deterministic IP^{+} embedding \textbf{h}_{i} , and the resulting graph-conditioned distribution factorizes as: \textbf{p}_{\theta}(\textbf{x}|I_{i},\textbf{G})=\int\textbf{p}_{\theta}(\textbf{x}|\{\textbf{h}_{i}\},A)\prod_{j=1}^{N}\delta(\textbf{h}_{j}-IP^{+}(I_{i}))d\textbf{h}_{i} (2) where the Dirac delta \delta(\cdot) simply expresses that each part embedding \textbf{h}_{i} is a deterministic output of the pre-trained IP^{+} encoder rather than a free latent variable; this is the standard encoder–prior factorization. The inner conditional distribution \textbf{p}_{\theta}(\textbf{x}\,|\,\{\textbf{h}_{i}\},\textbf{A}) is instantiated by our graph-conditioned diffusion prior. The adjacency matrix \textbf{A}\in\{0,1\}^{N\times N} encodes the part relationship graph G . Our approach, Graph-PiT, first constructs A from the parts spatial layout (during training) or from user-specified constraints (at inference), then learns a diffusion prior \textbf{p}_{\theta} conditioned on both \textbf{h}_{i} and A . This mechanism ensures that generated concepts x adhere to the intended part relationships, resolving the unordered-input and coherence challenges inherent in part-based concept composition. II-B Graph Relation Construction from Visual Parts During training, we automatically construct a two‑tiered graph \textbf{G}=(\textbf{V},\textbf{E},\textbf{A}) from each full image and its N segmented visual parts, so that the structural relationships among parts are explicitly encoded. Each visual part i is represented by a super-node v_{i}^{super}\in\textbf{V} , whose feature is its IP^{+} embedding Mean(\textbf{h}_{i})\in\mathbb{R}^{1\times D} . We further expand each super-node into d sub-nodes v_{i,k}^{sub}\in\textbf{V} corresponding to each token of \textbf{h}_{i} , yielding a hierarchical representation. For

Read the complete text