Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding
Yuchen Rao *, Stefan Ainetter *, Sinisa Stekovic, Vincent Lepetit, Friedrich Fraundorfer, Project page: stefan-ainetter.github.io/SCANnotatepp · 2025
Read Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding by Yuchen Rao *, Stefan Ainetter *, Sinisa Stekovic, Vincent Lepetit, Friedrich Fraundorfer, Project page: stefan-ainetter.github.io/SCANnotatepp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13580.txt.
Introduction
Yuchen Rao and co-authors show that machine-made labels can teach 3D scene models better than hand-drawn ones. Their 2025 paper starts from a pipeline that matches objects in scanned rooms to catalog CAD models and records their full poses. That pipeline had annotated an older dataset; here it is run on the newer ScanNet++ scans, which lacked such labels.
Models trained on the automatic labels beat models trained on manual labels at two tasks. One task fills in missing parts of point clouds. The other finds and aligns the right CAD model from a single view. The authors read this as proof that automatic annotation can raise quality while cutting cost at the same time. They release the new annotations, called SCANnotate++, with their trained models.
Catalog record
| CREATOR | Yuchen Rao *, Stefan Ainetter *, Sinisa Stekovic, Vincent Lepetit, Friedrich Fraundorfer, Project page: stefan-ainetter.github.io/SCANnotatepp |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding Yuchen Rao *, Stefan Ainetter *, Sinisa Stekovic, Vincent Lepetit, Friedrich Fraundorfer, Project page: stefan-ainetter.github.io/SCANnotatepp Abstract † † ∗ The first two authors contributed equally to this work. High-level 3D scene understanding is essential in many applications. However, the challenges of generating accurate 3D annotations make development of deep learning models difficult. We turn to recent advancements in automatic retrieval of synthetic CAD models, and show that data generated by such methods can be used as high-quality ground truth for training supervised deep learning models. More exactly, we employ a pipeline akin to the one previously used to automatically annotate objects in ScanNet scenes with their 9D poses and CAD models. This time, we apply it to the recent ScanNet++ v1 dataset, which previously lacked such annotations. Our findings demonstrate that it is not only possible to train deep learning models on these automatically-obtained annotations but that the resulting models outperform those trained on manually annotated data. We validate this on two distinct tasks: point cloud completion and single-view CAD model retrieval and alignment. Our results underscore the potential of automatic 3D annotations to enhance model performance while significantly reducing annotation costs. To support future research in 3D scene understanding, we release our annotations, which we call SCANnotate++, along with our trained models. Task 1: Supervised point cloud completion Task 2: Supervised CAD model retrieval and alignment Examples of our annotations for ScanNet++ Figure 1 : We provide high-quality shape and pose annotations for objects in RGB-D scans for the ScanNet++ dataset. Because our pipeline is automatic, we can annotate almost all objects in the scene even when they are only partially visible, and the annotations are of consistently high quality. We demonstrate that we can use these annotations for supervised learning including point cloud completion (Task 1) and CAD model retrieval and alignment (Task 2), and that the model performance can be increased either by using the annotations as additional training data, or by using them to fine tune a pre-trained model for a previously unseen dataset. 1 Introduction 3D scene understanding requires precise object recognition, reconstruction, and alignment. A critical building block in such technologies is the generation of high-quality 3D data, enabling use-cases of artificial intelligence applications for perceiving, interpreting, and interacting with virtual environments. A variety of multi-view datasets have been developed for 3D scene understanding, including SceneNN [ 17 ] , ScanNet [ 13 ] , ScanNet++ [ 42 ] , Matterport3D [ 7 ] , and ARKitScenes [ 5 ] . However, most of these datasets lack detailed annotations for object shapes, with the exception of ScanNet, for which the Scan2CAD dataset [ 3 ] provides 3D alignments of CAD models to target objects. The lack of available annotations can be explained by the challenge of manually annotating 3D shapes which can be difficult and time consuming for the human annotator, and is therefore prone to human inconsistencies and errors. An alternative approach involves creating synthetic datasets [ 43 , 32 ] , which is computationally expensive and requires significant resources. Additionally, methods trained in synthetic environments are likely to fail when applied on real (physical) world scenes, revealing as the well known issue of synthetic-to-real domain gap in deep learning. In this work, we examine whether automatic approaches for recovering 3D geometry of objects can be used to train deep learning models. In particular, we examine annotations obtained by HOC-Search [ 2 ] that jointly performs discrete and continuous search to automatically retrieve CAD models from a large database of synthetic shapes, e.g . ShapeNet [ 8 ] , and aligns them with objects in 3D scenes. The method assumes that a rough estimate of the object’s 3D pose is given for the target object, either through manual annotations or using deep neural networks [ 39 ] , and then performs optimization in image space of the target scene to create an accurate alignment between a retrieved CAD model and the target object. As [ 2 ] further demonstrates for the ScanNet dataset [ 13 ] , the quality of such annotations is comparable and often better than the manual annotations from the Scan2CAD dataset [ 3 ] . This was used by the authors of [ 3 ] to create the SCANnotate dataset, which contains CAD model alignments for 18617 objects for 1513 scenes of the ScanNet dataset. Here, we first apply HOC-Search on the ScanNet++ v1 dataset [ 42 ] to automatically obtain additional CAD model alignments for 5290 objects from 280 scans. We call this new dataset SCANnotate++. Then, as evaluation of the annotation quality, we show that deep learning approaches can be trained on such automatically annotated data. We consider two tasks that are critical for 3D scene understanding: First, we tackle point cloud completion and introduce a pipeline with two stages. We train the ShapeGF point cloud auto-encoder network [ 6 ] on complete point clouds from the ShapeNet dataset [ 8 ] . Then, we train a second encoder on real world partial point clouds with the objective to output the same embeddings as the first encoder, hence enabling point cloud completion. As our results for the ScanNet dataset [ 13 ] demonstrate, our model trained using automatically generated annotations performs better than the model trained on manually annotated training data from Scan2CAD [ 3 ] . Second, we consider CAD model retrieval and alignment from single RGB images. We show that ROCA [ 16 ] trained on the automatically generated SCANnotate dataset performs better than the original model trained on manual annotations from the Scan2CAD dataset, which shows the general usefulness of automatically generated CAD model annotations. Furthermore, we show that it is now possible to train and test ROCA on the ScanNet++ dataset by using our proposed SCANnotate++ dataset, and we are able to present compelling results. In summary, this paper provides insights into how pipelines for automatic annotations can be beneficial when training deep learning models for 3D scene understanding. We adapt HOC-Search to automatically obtain high-quality 3D object annotations for indoor scenes. As the result, we introduce SCANnotate++, a new dataset that provides CAD model alignments for objects in the ScanNet++ dataset. In addition, we showcase two successful use-cases for training deep learning methods on such automatically obtained annotations: point cloud completion, and single-view CAD model retrieval and alignment. We will make our dataset and models available, to foster future developments in 3D scene understanding. 2 Related Work Dataset CAD Model Annotations # 3D Scans # Annotated Objects Annotation Method Pix3D [ 36 ] Pix3D [ 36 ] ✗ 10k Manual ARKit Scenes [ 5 ] LASA [ 24 ] 920 11k Manual ScanNet [ 13 ] Scan2CAD [ 3 ] 1513 14k Manual ScanNet [ 13 ] SCANnotate [ 1 , 2 ] 1513 18k Automatic ScanNet++ v1 [ 42 ] SCANnotate++ (ours) 280 (*) 5k Automatic Table 1 : Comparison of the most popular real RGB-D datasets for indoor scene understanding where CAD model and object pose annotations are available. (*) At the time of annotating the dataset, only ScanNet++ v1 was publicly available. Hence, the current version of SCANnotate++