3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

Haoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan · 2026

Read 3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing by Haoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22279.txt.

Introduction

Zhen and co-authors teach language models to edit 3D layouts from text instructions. The input is a scene graph plus a plain language request. The model reasons over the graph and returns an updated graph. The update must meet the request and keep the layout coherent. The team tests the method on sorting, alignment, and room editing tasks. Against chain-of-thought fine-tuning and plain GRPO, it lifts IoU by about 15 percent on average. Center-distance error drops by about 25 percent.

Catalog record

CREATORHaoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41527
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22279

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing Haoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan Abstract Large Language Models (LLMs) and Vision Language Models (VLMs) have shown impressive reasoning abilities, yet they struggle with spatial understanding and layout consistency when performing fine-grained visual editing. We introduce a Structured Reasoning framework that performs text-conditioned spatial layout editing via scene-graph reasoning. Given an input scene graph and a natural-language instruction, the model reasons over the graph to generate an updated scene graph that satisfies the text condition while maintaining spatial coherence. By explicitly guiding the reasoning process through structured relational representations, our approach improves both interpretability and control over spatial relationships. We evaluate our method on a new text-guided layout editing benchmark encompassing sorting, spatial alignment, and room-editing tasks. Our training paradigm yields an average 15% improvement in IoU and 25% reduction in center-distance error compared to Chain of thought Fine-tuning (CoT-SFT) and vanilla GRPO baselines. Compared to SOTA zero-shot LLMs, our best models achieve up to 20% higher mIoU, demonstrating markedly improved spatial precision. Figure 1 : We introduce 3D-Layout-R1 , which performs multi-step language-guided 3D layout editing, iteratively updating an initially randomized scene into a sequence of spatially consistent intermediate layouts. 0 0 footnotetext: Equal contribution. 0 0 footnotetext: This work was done while Haoyu Zhen was an intern at NVIDIA. 1 Introduction Understanding and manipulating 3D scenes through natural language is a fundamental capability for agents and content creation systems. Beyond passive perception, agents must be able to rearrange their surroundings, e.g., “move the chair from the desk to align with the sofa”, which requires a sophisticated blend of capabilities: comprehending compositional spatial relationships, understanding semantic intent, and adhering to strict physical constraints to produce a plausible layout. While recent progress in 3D perception and multimodal foundation models has advanced the ability of VLMs to answer spatial questions, far fewer methods can execute structured and multi-step 3D layout edits in response to natural language instructions. Existing VLM-based spatial reasoning frameworks have primarily focused on passive 3D understanding. Models like SpatialRGPT ( 6 ) , SpatialLLM ( 28 ) , SpatialReasoner ( 27 ) , and 3D-R1 ( 19 ) enhance spatial VQA through depth cues, implicit 3D representations, or coordinate-based reasoning. Despite these gains, such systems do not modify the underlying 3D scene and lack the structured mechanisms needed for long-horizon editing. This gap motivates a shift from answering spatial queries to acting upon 3D layouts in an interpretable and physically consistent manner. Language-driven 3D layout editing has recently emerged as a promising direction. A recent line of works use LLMs as high-level planners to edit existing layouts. In this dominant paradigm, the VLM generates a high-level plan or a set of spatial constraints, which is then passed to a separate, external module, such as a constraint solver ( 37 ) or a differentiable optimizer ( 33 ; 10 ) , to compute the final 3D poses. While effective for ensuring physical feasibility, these pipelines are limited by manually specified rules or objectives, reduced flexibility for diverse instructions, difficulty handling long-horizon compositional edits, and a largely opaque reasoning process that offers little interpretability or opportunity for correction. Other methods perform one-shot layout prediction or single-object placement 1 , but these do not generalize to multi-object rearrangement or sequential editing of existing scenes. To the best of our knowledge, no existing system supports fully integrated, multi-step 3D layout editing that directly reasons over structured spatial representations. In this work, we introduce 3D-Layout-R1 , a framework that performs structured, interpretable, and language-directed 3D layout editing by reasoning directly over a 3D bounding-box based scene graph as an iterative canvas. Instead of generating a vague, free-form chain-of-thought (CoT), our model, 3D-Layout-R1, produces a structured trace of scene-graph transformations (Fig 1 ). Each reasoning step is an explicit, verifiable graph edit that directly updates the scene’s state. This approach embeds the 3D spatial logic directly within the model’s generation process. This allows 3D-Layout-R1 to plan and execute complex, multi-step rearrangements (e.g., ”first move the box, then place the lamp next to the book”) while ensuring each intermediate step is interpretable and geometrically coherent. Figure 2 : Example from the synthetic 3D sorting benchmark. Given a instruction to group and sort objects by shape and height, our model generates a concise structured reasoning trace with JSON scene-graph updates that transforms the initial scene into the final layout, in contrast to a long, ambiguous free-form thinking path. To achieve this, we integrate a GRPO-based reinforcement learning stage that optimizes layout accuracy using a dense 3D IoU reward and collision-aware penalties. By jointly leveraging structured scene-graph reasoning and RL-driven refinement, the model learns to generate precise, physically consistent layout edits that reliably satisfy complex textual instructions. Our contributions are summarized as follows: 1. We introduce 3D-Layout-R1 , a framework that directly performs multi-step 3D scene editing over a 3D-Layout using language-guided, interpretable chain-of-graph-edits reasoning across three tasks: Sorting, Spatial Alignment, and Room Editing. 2. We release a dataset of 15k 3D scenes with natural language instructions, intermediate chain-of-thought graph edits, and target layouts, providing the first benchmark dedicated to multi-step editing of existing 3D scenes. 3. We design geometric reward functions based on 3D IoU for policy optimization and evaluation, offering continuous and geometrically faithful supervision than distance-based or binary success metrics. 4. 3D-Layout-R1 improves mean IoU by 15% and reduces center-distance errors by 25–30% compared to CoT-SFT and GRPO baselines. It achieves up to 20% higher IoU compared to leading zero-shot LLMs and prior spatial reasoning or layout editing systems. 2 Related Works Spatial Reasoning. Recent efforts to endow VLMs with spatial intelligence have explored diverse modalities and representations. Several approaches enhance 2D VLMs with 3D positional cues from RGB-D inputs or multi-view images ( 6 ; 3 ; 44 ; 5 ; 8 ) , while others learn implicit “cognitive maps” from video ( 40 ; 35 ) or ground reasoning in 2D coordinates and large-scale embodied datasets ( 25 ; 32 ; 42 ; 30 ) . The most proximate works leverage explicit 3D representations for spatial reasoning. SpatialVLM ( 4 ) taught metric relationships by extracting 3D properties from 2D images, while SpatialRGPT ( 6 ) generated region-aware Visual Question Answering (VQA) using implicit 3D scene graphs. Furthermore, ( 28 ) and ( 26 ) incorporated estimated depth, distances, and explicit 3D coordinates to enable multi-step reasoning. In contrast to these works, which primarily focus on spatial VQA using implicit or coordinate-based representations, we investigate whether LLMs/VLMs can reason and update over the structure of a 3D scene graphs. Language-Driven 3D Layout Editing. Traditional 3D scene editing requires expert knowledge and manual operation. Recent language-driven approaches aim to automate spatial manipulation using LLMs/VLMs. Some methods operate via intermediate representations, such as 43 ; 22 , which either edit in 2D before lifting to 3D or construct semantic graph priors for scene decoding. Others model 3D structure directly, including diffusion- or placement-based techniques 41 ; 1 , though these are often restricted to simple object-level edits. A complementary line of work uses LLMs as high-level planners that produce spatial constraints or initial poses for external

Read the complete text