FirePlace: Geometric Re fi nements of LLM Common Sense Re asoning for 3D Object Place ment

Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi · 2025

Read FirePlace: Geometric Re fi nements of LLM Common Sense Re asoning for 3D Object Place ment by Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04919.txt.

Introduction

Ian Huang and co-authors teach a vision-language model to place furniture by checking its instincts against geometry. Their 2025 paper, FirePlace, addresses a split talent: multimodal models understand scenes semantically but fumble the three-dimensional details. Placing an object in a room needs both common sense and exact fit.

The framework first extracts the relevant geometry from the scene, then builds and solves explicit geometric constraints, and finally prunes the candidate spots that violate common sense. The model's judgment proposes; geometry disposes. The authors argue this division of labor is the reliable way to use such models in 3D tasks, rather than trusting their raw spatial guesses.

Catalog record

CREATORIan Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FirePlace: Geometric Re fi nements of LLM Common Sense Re asoning for 3D Object Place ment Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi Abstract Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work. FirePlace enables multi-modal large language models (MLLMs)to place new 3D objects into complex, preexisting 3D scenes, given (1) a 3D scene, (2) a 3D object, and (3) a language prompt. It uses a combination of MLLM common sense and low-level geometry constraints through the process described in this paper. Object placements generated by FirePlace are shown in Red . 1 Introduction Generating scenes with preexisting 3D assets is crucial for diverse applications like architecture, game development, and virtual reality. While constructing a scene can be seen as an iterative process of object placement, doing so even for a single object requires complex understanding of both high-level considerations for why an object should be placed at a particular location, as well as low-level understanding of the geometry of that object and its environment. Constructing such scenes requires placing objects in a way that is both physically feasible and contextually appropriate. This involves understanding high-level concepts like object relationships and aesthetics, as well as low-level geometric constraints. Multi-modal Large Language Models (MLLMs) [ 43 , 33 ] offer a promising approach due to their real-world knowledge and common-sense reasoning capabilities [ 58 , 29 ] , which have been leveraged for tasks such as graphical editing [ 20 , 52 , 24 , 47 , 19 , 10 , 14 ] , scene understanding [ 3 ] , and question answering [ 53 , 25 ] . However, applying them to object placement leads to many failure cases, for the reason that MLLMs struggle with precise 3D spatial reasoning and fine-grained geometry [ 9 , 55 ] . Existing efforts often resort to training or fine-tuning these models on extensive 3D datasets to enhance their spatial capabilities [ 5 , 30 , 17 , 59 , 21 , 27 ] . This paper introduces FirePlace, a novel approach for placing new 3D objects into existing scenes using off-the-shelf MLLMs, guided by natural language instructions. More concretely, our method takes as input a Universal Scene Descriptor (USD) of a 3D scene (including 3D assets within the scene), a camera angle capturing the scene, a new object mesh to be placed, and a language instruction describing the desired placement. FirePlace then outputs an edited USD file, representing the scene with the object inserted. Our system iteratively translates abstract constraint descriptions ( e.g . , “a book should be on the shelf”) into lower-level grounded 3D constraints ( i.e . Which part of the book should be related to which part of the shelf, and in what way?). To accurately determine the parts of the scene that are relevant to the placement task, FirePlace introduces a way for the MLLM to interrogate, visualize, and reason about relevant 3D information from the scene, by giving it access to a range of 3D processing tools. FirePlace is able to combine the 3D processing capabilities of such tools with the common-sense reasoning capabilities of MLLMs to produce placements of objects that are feasible ( i.e . , satisfy geometric constraints) and plausible ( i.e . , satisfy common-sense reasoning about aesthetics, function and accessibility of object placements). FirePlace: Geometric Re fi nements of LLM Common Sense Re asoning for 3D Object Place ment shows some object placements generated by FirePlace. While recent works have introduced systems that generate object placements in scenes from scratch [ 52 , 18 , 1 , 34 , 47 , 51 , 10 , 42 , 46 ] , we note that they would perform poorly on the task of object placement in complex preexisting scenes since they lack the following three capabilities: Reasoning with Fine-grained 3D Geometry. Prior works in scene-generation propose methods that act upon bounding boxes of objects [ 52 , 20 , 18 , 1 , 34 , 51 ] , attending to coarse physical relationships between objects to be synthesized. However, such representations fail when the key surfaces for the placement task lie within the bounding box, as with shelves in FirePlace: Geometric Re fi nements of LLM Common Sense Re asoning for 3D Object Place ment . For example, placing a book on a shelf requires reasoning about the specific geometric features of the shelf’s surface, such as its top plane and its depth, rather than simply its overall bounding box. To overcome the limitations of bounding-box-based approaches, FirePlace reasons with fine-grained 3D geometry from an explicit 3D scene representation by extracting, visualizing, and reasoning about object surfaces. This allows it to generate scenes that are more realistic and physically plausible. Reasoning about object instances. Existing scene generation methods lack the ability to reference specific object instances ( e.g . which chair, which wall). These methods rely on pre-defined assumptions about wall layouts [ 52 , 1 , 51 , 10 ] and assume either that the specific instance choice is irrelevant or predetermined by construction [ 1 , 52 , 18 ] . This assumption fails when inserting objects into complex scenes, where the correct instance choice may be context-dependent. For example, hanging a picture in a room with multiple walls requires understanding the specific context to identify the intended wall. To tackle this, FirePlace enables MLLMs to precisely reference object instances through visual selection. Common-sense placement. Existing methods [ 52 , 18 , 10 , 51 , 34 , 46 ] often neglect the final visual result and common-sense considerations such as aesthetics, functionality, and accessibility during object placement. FirePlace addresses this limitation by leveraging the knowledge embedded in MLLMs, selecting among potential placements based on a wider range of criteria, including aesthetic appeal, functional appropriateness, and accessibility considerations. This results in more realistic and plausible scene configurations that go beyond mere geometric feasibility. In order to enable MLLMs to reason about fine-grained 3D geometry and object instances , the MLLM must select among many discrete choices (of objects, and of surfaces) to choose the right ones for the placement task. However, MLLMs become substantially more prone to error as the set of options gets larger. We introduce a visual selection

Read the complete text