POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion

Andrea Rigo, Luca Stornaiuolo, Weijie Wang, Mauro Martino, Bruno Lepri, Nicu Sebe · 2026

Read POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion by Andrea Rigo, Luca Stornaiuolo, Weijie Wang, Mauro Martino, Bruno Lepri, Nicu Sebe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14056.txt.

Introduction

Andrea Rigo and co-authors give image generation consistent 3D layout control in 2026. Text-to-image diffusion models can place objects in odd spots or warp their shape. Prior methods use 2D cues or copy-warp-paste steps to fix placement. Those often distort object geometry and fail to preserve consistency across edits.

Their POCI-Diff uses 3D-layout guided diffusion for placement. Objects can be positioned consistently and interactively, then edited in place. A scene that stays stable across edits suits design and story work. Users gain control that plain text prompts cannot give. The user can move an object and see the scene stay in shape. That kind of live control is what most text prompts lack.

Catalog record

CREATORAndrea Rigo, Luca Stornaiuolo, Weijie Wang, Mauro Martino, Bruno Lepri, Nicu Sebe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14056

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion Andrea Rigo, Luca Stornaiuolo, Weijie Wang, Mauro Martino, Bruno Lepri, Nicu Sebe Abstract We propose a diffusion-based approach for Text-to-Image (T2I) generation with consistent and interactive 3D layout control and editing. While prior methods improve spatial adherence using 2D cues or iterative copy-warp-paste strategies, they often distort object geometry and fail to preserve consistency across edits. To address these limitations, we introduce a framework for P ositioning O bjects C onsistently and I nteractively (POCI-Diff), a novel formulation for jointly enforcing 3D geometric constraints and instance-level semantic binding within a unified diffusion process. Our method enables explicit per-object semantic control by binding individual text descriptions to specific 3D bounding boxes through Blended Latent Diffusion, allowing one-shot synthesis of complex multi-object scenes. We further propose a warping-free generative editing pipeline that supports object insertion, removal, and transformation via regeneration rather than pixel deformation. To preserve object identity and consistency across edits, we condition the diffusion process on reference images using IP-Adapter, enabling coherent object appearance throughout interactive 3D editing while maintaining global scene coherence. Experimental results demonstrate that POCI-Diff produces high-quality images consistent with the specified 3D layouts and edits, outperforming state-of-the-art methods in both visual fidelity and layout adherence while eliminating warping-induced geometric artifacts. 1 Introduction Figure 1 : POCI-Diff introduces an interactive diffusion-based system for image generation from 3D layouts, enabling precise control over object type, position, scale, and orientation, as well as camera movement, while supporting layout-driven image editing with seamless integration of new or modified objects. Precise and semantically grounded spatial control remains a fundamental challenge in text-to-image diffusion models, particularly when users wish to specify or modify scenes using 3D layouts. While recent advances in text-to-image (T2I) diffusion models have significantly improved visual realism and text–image alignment [ 30 , 21 , 3 ] , explicit control over object placement, orientation, and depth consistency remains limited. Most existing approaches rely on 2D spatial cues [ 34 ] , including keypoints [ 13 ] , rough sketches [ 38 ] , bounding boxes [ 32 ] , or segmentation masks [ 13 ] , to guide the diffusion process. While effective for enforcing planar layouts, these 2D representations lack explicit 3D constraints, making it difficult to reliably manipulate object position and depth ordering within a geometrically consistent 3D scene [ 12 ] . Recently, to introduce explicit geometric awareness into diffusion models, LooseControl [ 4 ] conditions generation on 3D scene layouts by rendering depth maps from 3D bounding boxes. This strategy improves geometric coherence compared to purely 2D layouts. However, LooseControl [ 4 ] relies on a single global text prompt, and therefore cannot associate object-specific semantic descriptions with individual 3D bounding boxes. As a consequence, while the overall scene geometry may be constrained, the semantic identity of objects at specific locations remains ambiguous, preventing explicit control over which object should appear where. The state-of-the-art Build-a-Scene (BAS) [ 8 ] partially addresses this limitation by introducing per-object semantics through an iterative generation pipeline. In BAS, a background image is synthesized first, after which individual objects are generated one at a time using 3D layout guidance and incrementally inserted into the scene. While effective, this design introduces two intrinsic limitations. First, BAS relies on a copy-warp-paste strategy during editing, in which objects are cropped from the image, geometrically warped to a new 3D location, and lightly harmonized via diffusion. Such warping-based manipulation often distorts object geometry and degrades visual realism, particularly under large spatial transformations. Second, the pipeline follows a sequential generation paradigm, requiring objects to be inserted one at a time, which is computationally inefficient and prone to error accumulation. Moreover, since harmonization is treated as a lightweight post-processing step, inconsistencies often arise between each edited object and its surrounding context, particularly in lighting, shading, and fine-grained structural details. Method Per-object One-shot 3D Warping-free Computational Semantic Control Scene Generation Editing Efficiency LooseControl [ 4 ] ✗ ✓ ✓ ✓ Build-A-Scene [ 8 ] ✓ ✗ (iterative only) ✗ (warp distortion) ✗ POCI-Diff (Ours) ✓ ✓ ✓ ✓ Table 1 : Comparison of key capabilities across methods. POCI-Diff is the only method that simultaneously supports one-shot scene generation from 3D layouts, per-object semantic control, and warping-free editing with high efficiency. These limitations motivate the need for a unified approach that (1) enables explicit per-object semantic control, (2) synthesizes complete scenes in a single diffusion pass, and (3) supports 3D objects manipulation without compromising visual quality or global scene coherence. To achieve this, we propose POCI-Diff, a diffusion framework for P ositioning O bjects C onsistently and I nteractively with 3D-guided image generation, a one-shot generation paradigm that jointly enforces explicit 3D geometric constraints and per-instance semantic binding, enabling a previously unaddressed capability for diffusion-based image generation. Our key insight is to integrate Blended Latent Diffusion (BLD) [ 2 ] with 3D layout conditioning to explicitly bind text prompts to individual 3D bounding boxes. This design enables one-shot synthesis of complete scenes with both accurate geometry and fine-grained, per-object semantics, directly addressing the limitations of LooseControl [ 4 ] and eliminating the iterative insertion required by BAS [ 8 ] . For interactive editing, we further introduce a warping-free manipulation pipeline that supports object insertion, removal, replacement, translation, and scaling. Instead of deforming pixel data, POCI-Diff regenerates objects at their target locations while inpainting them at the source, thereby avoiding geometric artifacts. To preserve object identity and maintain global scene harmonization, we condition the diffusion process on the original object image using IP-Adapter [ 35 ] , enabling consistent repositioning without compromising visual appearance. While we leverage prior diffusion-based controls, naively combining techniques like depth control and latent blending fails to maintain geometric consistency and object identity. Our distinct formulation resolves these conflicts by jointly enforcing geometry and instance-level semantics, achieving capabilities that exceed those of isolated components. Previous methods address either 3D geometric control or instance-level semantic control in isolation, but not both within a single diffusion process. Techniques that enable per-object semantics rely on iterative copy–warp–paste pipelines, leading to geometric distortion and poor scene coherence, whereas one-shot layout-guided methods enforce global structure but lack explicit semantic binding to individual instances. Table 1 summarizes the key capabilities of existing approaches. POCI-Diff is the only method that simultaneously supports one-shot scene generation from 3D layouts, explicit per-object semantic control, and warping-free editing while maintaining high computational efficiency. Finally, both LooseControl and BAS employ a lightweight LoRA-based [ 9 ] adaptation to ControlNet [ 38 ] , which often leads to box-like geometric artifacts. To mitigate this, we fully fine-tune a depth-conditioned ControlNet [ 38 ] using a curated dataset of high-resolution synthetic images generated by Flux.1 [ 11 ] , allowing the model to interpret 3D bounding boxes as volumetric constraints and avoid distortions, while maintaining high visual quality. Our contributions are as follows: • We introduce POCI-Diff, a depth-guided diffusion framework for one-shot scene generation from 3D layouts with fine-grained, per-object semantic control. • POCI-Diff is a warping-free editing pipeline that leverages IP-Adapter to perform consistent object transformations, including translation and scaling, while preserving object identity and scene coherence. • Through extensive quantitative and qualitative evaluations, we

Read the complete text