MessyKitchens: Contact-rich object-level 3D scene reconstruction

Junaid Ansari *, Ran Ding *, Fabio Pizzati, Ivan Laptev · 2026

Read MessyKitchens: Contact-rich object-level 3D scene reconstruction by Junaid Ansari *, Ran Ding *, Fabio Pizzati, Ivan Laptev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16868.txt.

Introduction

Ansari and co-authors tackle object-level 3D reconstruction of cluttered real scenes from images. Splitting a scene into single objects is hard due to varied shapes, occlusion, and contact. Robotics also needs reconstructions that respect physics, with no interpenetration. They release MessyKitchens, a dataset of messy kitchen scenes with precise object shapes, poses, and contacts as ground truth. They also extend SAM 3D with a Multi-Object Decoder for joint reconstruction. On three datasets, the authors report steady gains over prior state of the art.

Catalog record

CREATORJunaid Ansari *, Ran Ding *, Fabio Pizzati, Ivan Laptev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51990
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MessyKitchens: Contact-rich object-level 3D scene reconstruction Junaid Ansari *, Ran Ding *, Fabio Pizzati, Ivan Laptev Abstract Monocular 3D scene reconstruction has recently seen significant progress. Powered by the modern neural architectures and large-scale data, recent methods achieve high performance in depth estimation from a single image. Meanwhile, reconstructing and decomposing common scenes into individual 3D objects remains a hard challenge due to the large variety of objects, frequent occlusions and complex object relations. Notably, beyond shape and pose estimation of individual objects, applications in robotics and animation require physically-plausible scene reconstruction where objects obey physical principles of non-penetration and realistic contacts. In this work we advance object-level scene reconstruction along two directions. First, we introduce MessyKitchens , a new dataset with real-world scenes featuring cluttered environments and providing high-fidelity object-level ground truth in terms of 3D object shapes, poses and accurate object contacts. Second, we build on the recent SAM 3D approach for single-object reconstruction and extend it with Multi-Object Decoder (MOD) for joint object-level scene reconstruction. To validate our contributions, we demonstrate MessyKitchens to significantly improve previous datasets in registration accuracy and inter-object penetration. We also compare our multi-object reconstruction approach on three datasets and demonstrate consistent and significant improvements of MOD over the state of the art. Our new benchmark, code and pre-trained models will become publicly available on our project website: https://messykitchens.github.io/ . † † footnotetext: * Equal contribution. Figure 1 : MessyKitchens benchmark. Images of real scenes and corresponding high-fidelity object-level 3D scenes reconstructions composed of accurate object scans. 1 Introduction Accurate 3D scene reconstruction plays a pivotal role for many applications in digital arts and content creation, industrial inspection, surgery, heritage preservation, navigation as well as robot learning and simulation. While some tasks, e.g., navigation [ 9 , 47 , 2 , 42 ] , may only require free space estimation and scene-level surface reconstruction, other tasks, e.g., robotic manipulation [ 19 , 36 , 1 , 38 , 51 , 6 , 56 ] and animation [ 43 , 55 , 5 ] often rely on detailed reconstruction of individual objects. Object-level scene reconstruction is a difficult task challenged by the large variety of object shapes and frequent occlusions. Moreover, beyond object shape and pose estimation, animation and simulation tasks require physically-plausible scene reconstruction with correct estimation of object contacts, deformations and other parameters. Recent methods for 3D scene reconstruction have evolved from classic geometry-based formulations to learning-based approaches. The latter methods rely on the learned inductive biases and enable accurate shape predictions from a single image. In particular, several recent methods such as DepthAnything [ 59 ] , VGGT [ 52 ] and Gen3C [ 45 ] significantly advance results for monocular depth estimation. In comparison, object-level scene reconstruction has received relatively less attention. Among existing methods, MIDI [ 27 ] and PartCrafter [ 37 ] present impressive results, but focus on synthetic scenes, while the recent SAM 3D [ 11 ] enables the estimation of the shape and pose of single objects in real images. Besides new methods, the progress in object-level scene reconstruction also requires realistic and high-fidelity benchmarks for training and evaluation. While several benchmarks exist [ 3 , 19 , 32 ] , their 3D ground truth often suffers from the limited registration accuracy and inter-object penetrations. To address these limitations, we advance object-level scene reconstruction on two fronts. First , we introduce MessyKitchens , a new dataset with 100 real-world scenes featuring cluttered environments along with high-fidelity object-level 3D ground truth. As shown in Fig. 1 , our dataset contains contact-rich scenes composed of 130 varying kitchen objects that have been scanned and registered with high precision. Moreover, MessyKitchens provides accurate object contacts, and hence enables evaluation of geometrically-precise and physically-realistic scene reconstruction. In addition, we also provide a large-scale synthetic training set MessyKitchens-train composed of 1.8k contact-rich scenes and 10.8k rendered images. Second , alongside the benchmark, we propose a method for joint object-level scene reconstruction. Our approach builds upon the recent SAM 3D framework for single-object reconstruction and extends it with a Multi-Object Decoder (MOD) that jointly predicts the geometry and poses of multiple objects in a scene. By reconstructing objects simultaneously, MOD captures contextual relationships and enforces more physically-plausible configurations. In summary, we propose the following contributions: • We introduce MessyKitchens, a new benchmark with cluttered real scenes and high-fidelity 3D object-level ground truth, including accurate object shapes, poses and contacts along with precise scene-level object registration. • We propose Multi-Object Decoder (MOD), a method that extends SAM 3D to the joint modeling and reconstruction of multiple objects in a scene. • Through extensive experiments we demonstrate MOD to outperform state-of-the-art object-level scene reconstruction in MessyKitchens, GraspNet-1B [ 19 ] and HouseCat6D [ 32 ] . We also demonstrate significantly improved 3D accuracy of the MessyKitchens benchmark compared to other recent datasets. 2 Related Works Existing benchmarks for 3D scenes. Early datasets such as LINEMOD [ 25 ] , T-LESS [ 26 ] , YCB-M [ 22 ] , and YCB-Video [ 57 ] established standardized evaluation protocols for 3D pose estimation and object-level reconstruction, significantly advancing the field. However, they were captured in controlled laboratory settings and typically contain a limited number of objects and scenes with restricted category diversity. Datasets such as MP6D [ 10 ] and T-LESS [ 26 ] focus largely on industrial objects and lack representation of everyday kitchen categories, limiting their realism for domestic environments. Although synthetic datasets such as Falling Things [ 49 ] and ZeroGrasp-11B [ 29 ] scale up object count substantially, they remain purely simulation-based and do not reflect the challenges of real-world scene acquisition and annotation. More recent datasets, including GraspNet-1B [ 19 ] , HouseCat6D [ 32 ] , PhoCaL [ 54 ] , Omni6DPose [ 63 ] , KITchen [ 61 ] , PACE [ 60 ] , and GraspClutter6D [ 3 ] , increase object diversity and incorporate kitchen and transparent objects. Nevertheless, they often face limitations in annotation accuracy, scene complexity, or scalability, as their capture pipelines rely on controlled environments and specialized hardware, such as robotic manipulators or motion capture systems. In contrast, MessyKitchens is collected using a scalable pipeline that does not require specialized setup, enabling scene acquisition across diverse locations. Alternatives provide limited contact quality, which is one of the core focuses of our work. Moreover, our dataset achieves higher object-to-scene registration accuracy while remaining comparable in object diversity and scale. Object-centric 3D scene reconstruction. Object-centric 3D scene reconstruction has traditionally evolved from feed-forward and retrieval-based strategies. Feed-forward methods typically leverage encoder-decoder architectures to regress scene properties, such as geometry, instance labels, and poses, directly from images [ 39 , 62 , 40 , 13 , 65 , 58 , 53 , 21 ] , while retrieval-based approaches, such as DiffCAD [ 20 ] , align high-quality 3D assets from existing databases to input [ 23 , 30 , 34 , 35 ] . However, these methods are often limited by the scarcity of supervised 3D data and a heavy dependence on database diversity, which hinders their ability to generalize to complex out-of-distribution scenes [ 8 , 15 , 16 ] . Recently, compositional generative paradigms

Read the complete text