InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement

Nikita Kister * Pradyumna YM * István Sárándi Jiayi Wang, Anna Khoreva Gerard Pons-Moll, [5pt] · 2026

Read InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement by Nikita Kister * Pradyumna YM * István Sárándi Jiayi Wang, Anna Khoreva Gerard Pons-Moll, [5pt] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19673.txt.

Introduction

Kister and colleagues introduce InHabit, an auto and scalable data maker for filling 3D scenes with humans. Training agents to grasp 3D scenes needs large-scale data of people engaging with varied environments, yet such data is scarce. Real capture is costly and limited to lab settings, while current datasets rely on simple shape rules, skipping rich scene context. In contrast, 2D base models trained at web scale have gained common knowledge of human-environment interactions, which InHabit transfers to 3D via render-generate-lift.

Catalog record

CREATORNikita Kister * Pradyumna YM * István Sárándi Jiayi Wang, Anna Khoreva Gerard Pons-Moll, [5pt]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37478
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19673

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement Nikita Kister * Pradyumna YM * István Sárándi Jiayi Wang, Anna Khoreva Gerard Pons-Moll, [5pt] Abstract Training embodied agents to understand 3D scenes as humans do requires large-scale data of people meaningfully interacting with diverse environments, yet such data is scarce. Real-world motion capture is costly and limited to controlled settings, while existing synthetic datasets rely on simple geometric heuristics that ignore rich scene context. In contrast, 2D foundation models trained on internet-scale data have implicitly acquired commonsense knowledge of human–environment interactions. To transfer this knowledge into 3D, we introduce InHabit, a fully automatic and scalable data generator for populating 3D scenes with interacting humans. InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry. Applied to Habitat-Matterport3D, InHabit produces the first large-scale photorealistic 3D human–scene interaction dataset, containing 78K samples across ∼ \sim 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and RGB images. Augmenting standard training data with our samples improves RGB-based 3D human–scene reconstruction and contact estimation, and in a perceptual user study our data is preferred in 78% of cases over the state of the art. † † footnotetext: * Equal contribution. Figure 1 : InHabit generates diverse, scene-aware 3D human placement across varied environments and actions. In this craft shop, it produces realistic behaviors such as browsing, leaning, crouching, and reaching. This enables large-scale generation of interaction data for embodied 3D scene understanding. 1 Introduction Understanding 3D scenes from a human-centric perspective is essential for robotics and embodied AI [ 28 , 33 , 40 ] . Beyond geometric reconstruction and semantic labeling, this requires perceiving how humans would naturally interact with a scene, whether that is cooking at a stove, sitting on a sofa, or pausing at a window to look outside. However, progress is bottlenecked by a data scarcity problem. While we do have large collections of static 3D scenes [ 8 , 29 ] , and separately large collections of 3D human motions captured in studios [ 25 ] , there are no large datasets of 3D humans meaningfully interacting with diverse 3D environments. Capturing such interactions with motion capture inside natural scenes is logistically prohibitive and scales poorly. The handful of existing captured datasets [ 16 , 18 , 15 ] cover fewer than 30 scenes each. Synthetic generation offers an alternative, but existing approaches rely on geometric heuristics that optimize for collision avoidance and contacts without deeper consideration of what a person would be doing in a given scene [ 17 , 38 ] . Meanwhile, 2D foundation models trained on internet-scale data have implicitly acquired rich commonsense knowledge of how humans interact with their environments [ 23 ] . Image-editing models [ 5 , 12 ] can synthesize what it looks like to cook at a stove or read on a couch, because they have seen millions of such images and vision–language models [ 1 , 12 ] can process the image of a room and reason about what activities the scene affords. We introduce InHabit, a fully automated data-generation engine that brings the 2D capabilities of foundation models into the 3D domain for synthesizing large-scale 3D human–scene interaction data, as follows. InHabit follows a render–generate–lift principle: we render an existing 3D scene and using a VLM [ 12 ] we propose contextually meaningful actions tailored to the visible environment. Leveraging an image foundation model [ 12 ] we then generate one or more humans performing these actions into the rendered view. Finally, we lift the humans back to the 3D scene with an optimization-based method inspired by PhySIC [ 36 ] which reconstructs physically plausible SMPL-X [ 27 ] bodies within the known 3D scene geometry. This process requires no manual annotation and is scalable. Our method outperforms GenZI [ 24 ] and POSA [ 17 ] on quantitative placement metrics and is preferred in 78% of cases in a perceptual user study. To show scalability, we apply InHabit to the Habitat-Matterport 3D dataset [ 29 ] , producing the first large-scale photorealistic human–scene interaction dataset, which we call InHabitants . It contains over 78k samples across \sim 800 building-scale scenes, with complete 3D geometry, SMPL-X bodies, and RGB images. The dataset includes both single-person and multi-person interactions. We demonstrate the practical value of the InHabitants data on two downstream tasks. First, augmenting standard training data with our samples improves the SOTA contact estimation method DECO [ 31 ] on the DAMON benchmark [ 31 ] . This is remarkable as DECO required labor intensive contact annotations, whereas our data can be generated fully automatically. Second, existing feed-forward HSI reconstruction models [ 7 , 37 ] consistently improve when trained using our data, achieving stronger performance on PROX and RICH benchmarks than when trained on their original data alone. In summary, our contributions are: 1. A fully automated, scalable method for generating semantically meaningful human–scene interactions in 3D scenes, combining VLM-based and generative image affordance reasoning, and optimization-based 3D reconstruction. 2. The first large-scale 3D HSI dataset with over 78k samples across \sim 800 building-scale scenes, including complete scene geometry, SMPL-X humans performing a wide variety of interactions, and RGB images. 3. Quantitative evidence that our synthetic data improves both contact estimation and feed-forward HSI reconstruction, and a perceptual user study in which our interactions are strongly preferred over existing methods. We will release the dataset and code publicly. 2 Related Works 2.1 Human Placement in 3D Scenes The dominant approach to placing humans in 3D scenes relies on optimization over geometric constraints such as contact and penetration [ 16 , 38 ] . POSA [ 17 ] goes further by learning data-driven contact priors from captured interactions. These methods are effective at producing physically plausible configurations, but not necessarily semantically plausible ones. A body can be placed standing at any flat surface, yet there is no reasoning about what exactly one should be doing, or where one’s attention should be directed. Alternatively, [ 24 ] generates 3D human-scene interactions using natural language description but also requires the user to provide a coarse point location in the 3D scene where the interaction should occur. Our method addresses this limitation by leveraging the implicit interaction priors of 2D foundation models to place humans in a scene-appropriate way, without human in the loop. Leveraging 2D Image Generation for 3D Tasks. A recent line of work uses 2D image or video generation models as a source of training signal for 3D tasks. HumanWild [ 11 ] generates humans in diverse scenes using diffusion models and recovers 3D annotations from the generated images, showing that 2D generation can substitute for expensive real-world capture. Beyond the Contact [ 21 ] uses 2D inpainting diffusion to synthesize images of humans interacting with objects, then lifts these to 3D affordance maps, showing that interaction knowledge latent in 2D models can be extracted and grounded in 3D. DAViD [ 20 ] generates 2D human–object interaction videos using video diffusion and lifts them to 4D

Read the complete text