PlaceIt3D : Language-Guided Object Placement in Real 3D Scenes
Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, Guillermo Garcia-Hernando · 2025
Read PlaceIt3D : Language-Guided Object Placement in Real 3D Scenes by Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, Guillermo Garcia-Hernando (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05288.txt.
Introduction
Ahmed Abdelreheem and co-authors set a new task: placing an object in a real 3D scene by words alone. Their 2025 paper hands a model a room scan, a 3D asset, and a loose text hint about where it belongs. Many placements can satisfy one hint, so the task is open-ended by design.
Judging needs sense of free space and of how objects relate in three dimensions. The team builds a benchmark and scoring rules for the task. They also release training data for 3D language models and a first working method. The authors hope the task joins the standard suite for testing general 3D models.
Catalog record
| CREATOR | Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, Guillermo Garcia-Hernando |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05288
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PlaceIt3D : Language-Guided Object Placement in Real 3D Scenes Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, Guillermo Garcia-Hernando Abstract We introduce the task of Language‑Guided Object Placement in Real 3D Scenes. Given a 3D reconstructed point‑cloud scene, a 3D asset, and a natural‑language instruction, the goal is to place the asset so that the instruction is satisfied. The task demands tackling four intertwined challenges: (a) one‑to‑many ambiguity in valid placements; (b) precise geometric and physical reasoning; (c) joint understanding across the scene, the asset, and language; and (d) robustness to noisy point clouds with no privileged metadata at test time. The first three challenges mirror the complexities of synthetic scene generation, while the metadata‑free, noisy‑scan scenario is inherited from language-guided 3D visual grounding. We inaugurate this task by introducing a benchmark and evaluation protocol, releasing a dataset for training multi-modal large language models (MLLMs), and establishing a first nontrivial baseline. We believe this challenging setup and benchmark will provide a foundation for evaluating and advancing MLLMs in 3D understanding. Figure 1 : Language-guided 3D Object Placement in Real 3D Scenes : Given a text prompt, the task is to find a valid placement for an asset, requiring semantic and geometric understanding of the scene, the asset’s shape, and spatial relationships. Colored dots show referenced objects (for visualization only, not given to the model), and the yellow arrow indicates the predicted frontal direction. 1 Introduction † † ∗ Work done during an internship at Niantic. At two to three years old, neurotypical children learn to follow two-step instructions like “Get your shoes and put them on the shelf” [ 42 ] . These tasks may appear simple, yet children need time to grasp basic vocabulary and to learn the physical affordances of both 3D objects and scene layout. Perhaps AIs could obtain similar capabilities. In this paper, we focus on the novel task of language-guided 3D object placement in a reconstructed real 3D scene . As in the shoe example, the goal is to find a valid placement of the object among multiple configurations that satisfy the instruction. As shown in Figure 1 , the placement must also respect the physical constraints of the space and of the 3D asset. Excelling at this task would unlock applications such as instructing a robot, through language, to move a real object to a new location. It is also relevant to augmented reality (AR). For instance, a shopper wearing an AR headset could use natural‑language commands to anchor a virtual product on a real‑world surface or reposition digital décor anywhere in the room. MLLMs have recently shown strong performance on language-guided 3D scene understanding tasks, including visual question answering [ 22 , 11 , 23 ] , visual grounding [ 25 , 62 ] , and synthetic scene generation [ 18 , 55 , 44 ] . We study language-guided 3D asset placement in reconstructed scenes, a problem closest to grounding and to synthetic scene generation, yet distinct in that it requires addressing all of the following challenges simultaneously: i. Ambiguity of solutions . 3D visual grounding typically admits a single correct match, whereas 3D placement is inherently one-to-many: multiple placements can satisfy the instruction [ 19 , 6 ] . This complicates both benchmarking and data construction, which must accommodate multiple valid answers. ii. Intrinsic 3D reasoning . Many constraints are geometric and cannot be resolved from 2D projections alone. For example, “place the asset in between the chair and the table, hidden from the window” requires reasoning about free space and spatial relationships in 3D. iii. No privileged information at test time . In contrast with synthetic scene generation, we require just the reconstructed 3D scene. We are not given layouts [ 38 , 44 ] , scene graphs [ 61 ] , object properties, or clean geometry [ 55 , 24 ] to aid with making a prediction. iv. Joint reasoning over scene, asset, and language . The asset’s size and shape restrict feasible placements; given the same scene and instruction, a large object has fewer valid locations than a small one. Among the valid options, the model must follow the user’s stated intent rather than default to common sense priors [ 56 , 36 ] . The highlight of this paper is the introduction of a challenging novel task, which we call PlaceIt3D : language-guided 3D placement on real scenes. To the best of our knowledge, no existing benchmarks, datasets, or methods directly address this problem. To advance research in this area, we make three key contributions, summarized here: • We introduce PlaceIt3D -benchmark for language-guided placement with 3,500 evaluation examples, each consisting of a real ScanNet scene [ 15 ] , a PartObjaverse-Tiny asset [ 54 ] , and a guiding prompt. Our evaluation protocol accounts for placement ambiguity, enabling fair comparison across methods. • We present PlaceIt3D -dataset, a large-scale training dataset with 100,505 training and 2,566 validation examples, each annotated with all valid placements that can be used for training MLLMs. Like the benchmark, it uses ScanNet scenes and PartObjaverse-Tiny assets. • We propose PlaceWizard , a proto-method for this task built on recent 3D LLMs [ 25 ] . It uses a modified form of spatial aggregation, an asset encoder, and rotation prediction, outperforming existing baselines. Figure 2 : PlaceIt3D -dataset creation . Given a scene and an asset as input (a) the goal is to create a prompt (f) and corresponding mask \mathcal{M} of valid placements (e) . We start by finding the set of points which are physically plausible placements, shown in red in (b) . We consider eight equally spaced rotation angles, which condition the valid placements. For this example, angle 0^{\circ} has more valid placements than 45^{\circ} . To generate the language constraints, we use the ground truth scene graph (c) . Object anchors are selected from the scene graph and combined with relationship types to create a constraint and corresponding validity mask (d) . The different placement constraints are combined in the final output by intersecting the validity masks (e) given a mask of valid dense placements. Based on each selected set of anchors and constraint relationships, a natural language prompt is created using templates (please, see supplemental for more details). 2 Related Work 3D and Large Language Models. 3D LLMs [ 35 ] are a subclass of MLLMs that jointly process point clouds/meshes and text. They typically align 3D features to the language space with lightweight projection layers or cross‑attention adapters, mirroring strategies from image-grounded MLLMs (VLMs) [ 32 , 30 , 29 , 9 , 5 , 13 , 27 ] . For grounding and QA, 3D-LLM [ 22 ] fuses point clouds with 2D image features; Reason3D [ 25 ] routes a pre-trained point encoder through a Q-Former [ 31 ] and a decoder guided by both the 3D input and the LLM output. ScanReason [ 62 ] interleaves grounding and reasoning during inference. However, none of these systems addresses language-guided object placement, which requires reasoning about free space, spatial relations, and asset properties. We introduce a 3D LLM baseline designed to model these factors. 3D object placement and