Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
Wei Deng, Mengshi QiHuadong MaState Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications, China · 2025
Read Global-Local Tree Search in VLMs for 3D Indoor Scene Generation by Wei Deng, Mengshi QiHuadong MaState Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18476.txt.
Introduction
Wei Deng, Mengshi Qi, and Huadong Ma generate 3D indoor scenes with a vision language model guided by global local tree search. Large vision language models excel in many fields, yet few studies apply them to indoor scene generation. The task is a planning problem bound by spatial and layout common sense.
Globally, objects are placed one by one, with multiple placements explored in a tree search. The scene is decomposed by room, region, floor objects, and supported objects to cut tree depth. Locally, each placement splits into steps. The model reads a top down grid whose cells carry distinct emojis and names a cell as the object location. Experiments show more plausible scenes than state of the art approaches.
Catalog record
| CREATOR | Wei Deng, Mengshi QiHuadong MaState Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications, China |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Global-Local Tree Search in VLMs for 3D Indoor Scene Generation Wei Deng, Mengshi QiHuadong MaState Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications, China Abstract Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e . room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at https://github.com/dw-dengwei/TreeSearchGen . 1 Introduction Figure 1 : Illustration of chain-like (left) and tree-like (right) reasoning in VLMs on 3D scene generation. Each node represents a token or language sequence. The red dashed nodes indicate the VLM produces an inappropriate output. The chain-like method cannot correct the prior errors and the subsequent process reasons based on the errors, leading to a non-realistic layout, such as exceeding the room. In contrast, the tree-like method can modify the output if a mistake occurs, resulting in a more realistic layout. 3D indoor scene generation [ 11 , 21 , 39 , 30 , 7 , 14 , 8 , 36 , 12 , 34 , 37 ] refers to the process of automatically producing realistic 3D indoor scenes with a computer program. In recent years, some studies [ 14 , 8 , 37 , 11 , 30 ] have explored on using natural language for scene generation, as the textual prompts offer a user-friendly interface. Realistic 3D indoor scene generation is crucial in fields such as interior design, 3D gaming, virtual/augmented reality, and embodied AI. The primary challenge in generating high-quality scenes is modeling plausible and realistic spatial relationships ( i.e . , 3D layouts) between objects. To achieve this, early studies [ 14 , 30 ] adopt a data-driven fashion. They train generative models on 3D scene datasets to learn the joint distribution of real indoor scenes and their textual descriptions. However, collecting 3D scene datasets is challenging and costly, resulting in limited dataset scales and less robust models. Recently, researchers have focused on using large Vision-Language Models (VLMs) [ 11 , 8 , 37 , 40 ] to achieve this task. VLMs are trained on large-scale datasets, enabling them to understand complex user instructions and infer 3D layouts from language based on common knowledge of indoor furnishing. For example, Feng et al . [ 8 ] propose LayoutGPT to generate the dimension, location, and orientation parameters for each object within a scene from a textual prompt. However, as shown in Figure 1 , the current VLMs perform a token-level, left-to-right decision-making process during inference time [ 38 ] , which is not suited for 3D layout reasoning. Specifically, such an auto-regressive reasoning method cannot modify previous outputs. If a previous object is placed at an inappropriate location, this error will accumulate. Consequently, it remains a challenge to improve VLM’s reasoning ability for 3D scene generation. 3D scene generation needs to search with a tree-like approach rather than chain-like. When furnishing a room, humans usually place objects sequentially. In addition, each object has multiple candidate positions. We pick one of them for the current object and continue to put the next objects. This process is iteratively applied until we place all the objects. If an object is put in an inappropriate location, it stops the next object from being put in the room and we will adjust the previous decision. Consequently, in Figure 1 , the problem space can be represented as a tree. Each layer in the tree denotes an individual object and each node in the same layer represents the candidate placement for the object. The chain from the root node to the last layer is a layout solution. Our objective is to search on the tree to find such a chain, subjected to space range, the placement common sense, non-overlapping, and non-floating constraints. Based on the above analysis, we propose a novel global-local tree search method to enhance VLM reasoning for 3D indoor scene generation. Complex scenes with numerous objects make searching a deep tree challenging. Our key insight is that the scene structures can be represented hierarchically. We first construct a hierarchical scene representation from user input. A scene is decomposed as room level, region level, floor object level, and supported object level. This representation serves as a proxy between the textual input and the scene output and we can place the objects region by region, which reduces the computational cost. Subsequently, we propose a global-local tree search method to generate the layout for the object within the same region. The global tree search method places objects sequentially, mimicking human behavior. Specifically, it starts from the root node (the region) and leverages the local tree search method to generate a node in the next layer. Then, the global tree search method turns to the next layer and iteratively applies such a process until it generates all objects. The local tree search method generates the position of an individual object. At each layer, it determines one position parameter by textually and visually prompting the VLM. In our approach, each layer has multiple alternatives. Thus, our method performs a tree search in the problem space. To summarize, our contributions are three-fold: (1) We propose a novel global-local tree search method to enhance VLM reasoning for generating realistic 3D indoor scenes; (2) We design a hierarchical scene representation as a commonsense bridge between textual input and 3D indoor scenes, further reducing computational costs. (3) Our quantitative and qualitative studies demonstrate that our method generates more realistic 3D indoor scenes than state-of-the-art approaches. User studies suggest that our approach ranks the best among the three approaches. 2 Related Work 3D Scene Generation is not as trivial as directly using a 3D object generation model to generate multiple objects. The primary challenge lies in layout generation, which requires modeling spatial and semantic relationships among objects. In the early stage, some works train generative models ( e.g . GANs [ 19 , 18 ] , VAEs [ 7 , 39 , 12 ] , diffusion models [ 39 , 30 , 14 , 36 ] ,