MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

Zehuan Huang Yuan-Chen Guo Xingqiao An Yunhan Yang Yangguang Li Zi-Xin Zou, Ding Liang Xihui Liu Yan-Pei Cao Lu Sheng · 2024

Read MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation by Zehuan Huang Yuan-Chen Guo Xingqiao An Yunhan Yang Yangguang Li Zi-Xin Zou, Ding Liang Xihui Liu Yan-Pei Cao Lu Sheng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03558.txt.

Introduction

Zehuan Huang and co-authors introduce MIDI, a method to generate a full three dimensional scene from one image in a 2024 paper. Older routes rebuild scenes, fetch ready objects, or generate objects one by one in stages. MIDI instead extends a trained image to object model into a multi instance diffusion model, which creates many objects at once with correct spatial relations.

A new attention mechanism captures how objects interact and cohere, inside the generation step itself. Inputs are partial object images plus global scene context, and object completion is modeled directly. Training uses a small set of scene level data plus single object data to keep general skill. Tests on synthetic, real, and stylized scenes show top results.

Catalog record

CREATORZehuan Huang Yuan-Chen Guo Xingqiao An Yunhan Yang Yangguang Li Zi-Xin Zou, Ding Liang Xihui Liu Yan-Pei Cao Lu Sheng
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation Zehuan Huang Yuan-Chen Guo Xingqiao An Yunhan Yang Yangguang Li Zi-Xin Zou, Ding Liang Xihui Liu Yan-Pei Cao Lu Sheng Abstract This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models. Figure 1 : MIDI generates compositional 3D scenes from a single image by extending pre-trained image-to-3D object generation models to multi-instance diffusion models, incorporating a novel multi-instance attention mechanism that captures inter-object interactions. (a) shows our generated scenes compared with those reconstructed by existing methods. (b) presents our generated results on synthetic data, real-world images, and stylized images. 1 Introduction Generating compositional 3D scenes from a single image is challenging due to the limited spatial clues captured from a partial point of view. In fact, accurately inferring the 3D geometry of each instance and the spatial relationships of multiple instances within a scene, requires extensive prior knowledge of the 3D visual world. Existing methods can be broadly categorized into two classes, according to how the prior knowledge is processed. The former class [ 48 , 77 , 50 , 7 , 38 , 18 , 79 , 6 , 4 ] encodes 3D geometry by neural networks that are trained from scene-level 3D datasets, and then inferences the geometry in a new image with a feed-forward pass. Due to the scarcity of supervised data, these methods often suffer from poor reconstruction quality in unseen scenarios. The other class [ 19 , 28 , 32 , 33 , 34 , 17 ] stores 3D models in a database, then retrieve and assemble similar 3D models to match the input image. However, the limited geometric clues from a single image make it difficult to precisely identify and arrange the correct models. Moreover, since it is impractical for a 3D database to contain every possible model that exactly corresponds to the input image, the retrieved models can only approximately align with the objects, leading to inconsistencies. Therefore, methods in both classes lack accuracy and sufficient out-of-domain generalizability, in terms of novel object shapes and unseen scene layouts. Recent image-to-3D object generation models [ 29 , 40 , 43 , 44 , 24 , 64 , 27 , 78 , 70 , 35 , 69 , 73 , 66 , 68 , 39 , 80 , 71 , 65 , 20 , 75 ] , with powerful 3d prior and generalization capabilities, can generate high-quality geometry from a single object image. Building upon these pre-trained models, a common approach for scene generation involves using them as tools within a multi-step compositional generation process, which includes segmenting the scene image, completing individual object images, generating each object, and optimizing their spatial relationships [ 81 , 5 , 21 ] , as shown in Fig. 2 . While these methods leverage the priors of 3D object generation models, the generation process is inherently lengthy and prone to error accumulation – errors in intermediate steps can significantly distort the final result. Moreover, the optimization of spatial relationships cannot directly optimize 3D objects generated one by one by the previous stage that lacks global scene context, leading to misalignments between the generated instances and the overall scene. Therefore, if inter-object spatial relationships can be modeled directly within the 3D generation model, it is possible to construct an end-to-end pipeline that addresses these issues by generating all instances simultaneously with coherent spatial arrangements. We propose MIDI, which extends pre-trained 3D object generation models to multi-instance diffusion models, establishing a new paradigm for compositional 3D scene generation. Our approach enables the simultaneous creation of multiple 3D instances with accurate spatial relationships from a single scene image, moving beyond independent object generation to a holistic understanding of the scene. Building upon large-scale pre-trained image-to-3D object generation models [ 78 , 71 , 35 , 80 ] , MIDI employs a novel multi-instance attention mechanism that effectively captures complex inter-object interactions and spatial coherence directly within the generation process, eliminating the need for complex multi-step procedures. This advanced design allows for the direct generation of cohesive 3D scenes, significantly enhancing both efficiency and accuracy. Due to the universal nature of spatial relationships between objects, we effectively supervise the interactions between 3D instances using a limited amount of scene-level datasets [ 15 , 16 ] during training. Additionally, we incorporate single-object data for regularization, thereby maintaining the generalization ability of the pre-trained model. Figure 2 : Comparison between our scene generation pipeline with multi-instance diffusion and existing compositional generation methods. To validate the effectiveness of our proposed paradigm, we conduct experiments on synthetic datasets [ 15 , 16 ] , real-world scenes [ 8 , 62 ] , and various stylized scene images generated by text-to-image diffusion models [ 57 , 52 ] . Results demonstrate that MIDI significantly advances the field of 3D scene generation by effectively modeling inter-object interactions through our multi-instance attention mechanism in the pre-trained 3D generation model. MIDI produces high-quality 3D scenes with accurate geometries and spatial layouts, while exhibiting strong generalization capabilities. In summary, our main contributions are as follows: • We establish a new paradigm for compositional 3D scene generation by proposing a multi-instance diffusion model, which extends pre-trained image-to-3D object generation models to generate spatially correlated 3D instances. • We introduce a novel multi-instance attention mechanism that effectively models cross-instance interactions, ensuring the coherence and accurate spatial relationships. • Experiments demonstrate MIDI achieves state-of-the-art performance, significantly improving the generation of 3D scenes by accurately capturing inter-object relationships and providing better alignment with the input. Figure 3 : Method overview. Based on 3D object generation models, MIDI denoises the latent representations of multiple 3D instances simultaneously using a weight-shared DiT module. The multi-instance attention layers are introduced to learn cross-instance interaction and enable global awareness, while cross-attention layers integrate the information of object images and global scene context. 2 Related Work 2.1 Scene Reconstruction from a Single Image Recovering the 3D structure of a scene from a single image is a fundamental challenge in computer vision. Existing methods can be categorized into feed-forward reconstruction methods [ 48 , 77 , 50 , 7 , 38 , 18 , 79 , 6 , 4 ] , retrieval-based methods [ 19 , 28 , 32 , 33 , 34 , 17 ] , and

Read the complete text