SpatialLM : Training Large Language Models for Structured Indoor Modeling

Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou · 2025

Read SpatialLM : Training Large Language Models for Structured Indoor Modeling by Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07491.txt.

Introduction

Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, and Zihan Zhou teach language models to sketch indoor spaces in 3D in a 2025 paper. Their model, SpatialLM, eats raw point clouds of rooms. It outputs walls, doors, windows, and labeled object boxes. Standard multimodal architectures are kept, with no task-specific design.

A synthetic set of over twelve thousand scenes trains it. Those scenes hold ground-truth layouts for supervision. Choices in training design were studied with care. Layout estimates topped public benchmarks in their runs. Object detection scores stayed competitive with specialists. Augmented reality and robots are target users.

Catalog record

CREATORYongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07491

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpatialLM : Training Large Language Models for Structured Indoor Modeling Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou Abstract SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM , we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more. 1 Introduction 3D indoor environments are ubiquitous in our daily lives. We spend a lot of time and perform various activities in such environments every day. Therefore, a long-standing goal of artificial intelligence is to teach machines to perceive, reason about, and interact with 3D indoor scenes as humans do. In this work, we focus on the task of structured indoor modeling , which aims to extract structured indoor scene descriptions from raw sensorial inputs ( i.e . , RGBD scans). Specifically, the scene description includes both the architectural layout ( i.e . , walls, doors, and windows) and 3D object bounding boxes in the indoor environment. Such 3D structure information has been shown to benefit a wide range of real-world applications such as scene editing [ 40 ] , augmented reality [ 4 , 2 ] , and robot navigation [ 17 ] . Compared to 3D formats such as meshes, voxels, or implicit functions, the structured scene description provides a highly compact yet flexible scene representation. In this paper, our first contribution is to regard the structured descriptions as scripts of a general-purpose language ( i.e . , Python) and propose to predict the language in text form, as illustrated in Figure 1 . This design choice has several advantages: (i) it is human interpretable and editable; (ii) it can be easily extended to incorporate any new classes without affecting existing content in the scripts; and (iii) it allows us to leverage the strong built-in coding capability of pre-trained large language models. Therefore, our goal is to implement our method by directly fine-tuning open-source LLMs. While modern (multimodal) LLMs have revolutionized fields like natural language processing, 2D image understanding and generation, few work has attempted to use LLMs for 3D structured scene modeling. This is partly due to the lack of large-scale, high-quality datasets for the task. In this paper, as our second contribution , we collect a new synthetic dataset, which consists of the point clouds of 12,328 distinct scenes (54,778 rooms) paired with 3D structure information, and conduct the first empirical study on the best strategy of aligning the point cloud input with LLMs for structured scene modeling. Building upon our findings, we introduce SpatialLM , a large language model which is capable of processing point cloud input and generating structured scene descriptions. As our third contribution , we show that, by first training on our large-scale dataset and then on smaller downstream task data, our model gives competitive performance on public benchmarks for layout estimation and 3D object detection, respectively. Additionally, we provide proof-of-concept zero-shot experiments which show that our model can handle point clouds from diverse sources such as monocular video sequences. These results suggest that SpatialLM may serve as a base for developing future solutions for scenarios where enhanced spatial understanding and reasoning is required ( e.g . , embodied AI). Figure 1 : The overall pipeline of SpatialLM . Given the point cloud input, it employs a standard “Encoder-MLP-LLM” architecture for multimodal feature alignment (left) , and generates structured scene descriptions in pure text form as output (middle) . The reconstructed 3D structure is further overlaid on the point cloud for visualization (right) . 2 An Empirical Study on Point Cloud Feature Alignment ⬇ @dataclass class Wall : a_x : int a_y : int a_z : int b_x : int b_y : int b_z : int height : int ⬇ @dataclass class Door : wall_id : str position_x : int position_y : int position_z : int width : int height : int ⬇ @dataclass class Window : wall_id : str position_x : int position_y : int position_z : int width : int height : int ⬇ @dataclass class Bbox : class : str position_x : int position_y : int position_z : int angle_z : int scale_x : int scale_y : int scale_z : int Figure 2 : Definition of our structured representation for layouts and objects. Given the point cloud \mathbf{P} (typically derived from RGBD scans), our goal is to recover the 3D layout \mathbf{L} , which consists of all walls, doors and windows, and a set of objects \mathbf{O} in the scene. As discussed before, we represent \mathbf{L} and \mathbf{O} as scripts of a general-purpose language, and fine-tune LLMs to predict the language in text form. Fig. 2 shows all the parameters we use for the layout and objects. To the best of our knowledge, we are the first to employ LLMs for structured scene reconstruction from point clouds. Thus, we first propose a new dataset suitable for training the LLMs on (Section 2.1 ). Then, with the new dataset, we set out to investigate the following two key questions: (i) What point cloud features to use for the structured indoor modeling task ( Section 2.2 )? (ii) How to integrate the features into LLMs ( Section 2.3 )? 2.1 Dataset and Metrics Table 1 provides a list of popular indoor RGBD scan datasets. We can make a few observations: First , the number of scenes in real datasets is relatively small, ranging from about one hundred to a few thousands. To collect such a dataset, one needs to take RGBD scans in real scenes and manually annotate 3D information, which is a highly laborious process. Further, most real datasets provide annotations on the objects ( i.e . , semantic labels and 3D boxes) only. One exception is SceneCAD [ 1 ] , which manually adds wall annotations to ScanNet dataset. But it only contains 1,151 rooms. Table 1: Quantitative statistics of indoor RGBD scan datasets. Dataset (year) source #scenes annotations layouts objects ScanNet (2017) [ 13 ] real 1,513 \bullet Matterport3D (2017) [ 7 ] real 90 \bullet 3RScan (2019) [ 50 ] real 1,482 \bullet SceneCAD (2020) [ 1 ] real 1,151 \bullet \bullet ARKitScenes (2021) [ 4 ] real 5,048 \bullet MultiScan (2022) [ 34 ] real 273 \bullet ScanNet++ (2023) [ 59 ] real 1,006 \bullet HM3DSem (2023) [ 58 ] real 181 \bullet Structured3D (2020) [ 64 ] syn. 3,500 \bullet Hypersim (2021) [ 45 ] syn. 461 \bullet ProcTHOR (2022) [ 14 ] syn. 12,000 \bullet \bullet HSSD (2024) [ 14 ] syn. 211 \bullet \bullet ASE

Read the complete text